Seqera Platform Feature Requests

Hi as the Seqera folks are aware we've had a lot of issues lately migrating to scripted "Infrastructure as Code" deployments with Seqera Platform, and I think the root of a lot of the problems we have hit is that when you use things like tw / seqerakit / Seqera Terraform Provider for deploying Compute Environments, there are a LOT of settings, most all of them independently toggle-able, yet if you want things like Wave + Fusion to work its critical that you use the correct combinations of settings with the correct combinations of values for those settings. This has lead to us making many rounds of thinking we deployed a working CE with Seqera based on small testing, only to find with larger scale testing (when the users start running pipelines) that something is wrong and things break. I think the real solution to this is for Seqera to host some clear and repeatable and re-usable examples of full configurations for various use cases. Similar to how we have the "Nextflow Patterns" webpage that documents various end-to-end scripted methods for Nextflow, I feel that we really need something analogous for the Seqera Platform scripted deployments, because its too easy to miss settings that you did not realize were required for your CE to work. I think some of the following would be good examples to start with, described in both tw and seqerakit and Seqera Terraform Provider; deploying a Compute Environment with EBS AutoScale the above but with Wave included the above but with Wave + Fusion included launching a test pipeline on the CE just created above doing all of the above with both TowerForge created CE's and with Manually Added CE's I think some examples like this are already in the seqerakit GitHub repo, not sure if there are equivalents already available for tw-cli . Ultimately it will be needed for the Terraform Provider as well, and they will all benefit from being available in one place so that users can see everything at once.

We have S3 buckets that other accounts can write to. We want to optionally add acl=bucket-owner-full-control when uploading files via Data Explorer in Seqera Platform to ensure that the bucket owner owns and has full control over new objects uploaded .

This is a follow up to the ticket here Context The situation being that we need to have certain sets of Nextflow configs that get applied to all pipeline runs in order to ensure correct settings for things like AWS client, AWS tagging, process walltime and resource limits, etc. get applied for users on our infrastructure to avoid common issues. The problem with current methods of managing Nextflow configs on Seqera Platform is that it seems you can only really do one of the following; attach a Nextflow config to a Compute Environment - does not work if the CE is Shared attach a Nextflow config to a pipeline in LaunchPad - does not work if you want to have the same configs in all pipelines, you end up with dupilcated and independent config code that needs to be updated constantly for every single pipeline individually attach a Nextflow config to an individual pipeline Run at launch time - does not work since every user now must enter the correct config every single time they need to run a pipeline embed the config into the pipeline codebase (e.g. in the nextflow.config in the git repo for the pipeline - does not work when you want to have a single source of shared configs for all pipelines As per the other ticket, currently we are handling this by hosting a "global" nextflow.config file on a S3 bucket and then using this in our Pre-Run Script attached to every Compute Environment in Seqera Platform; ```bash THIS INCLUDES ORG-SPECIFIC NEXTFLOW CONFIGS ; DO NOT REMOVE THIS aws s3 cp s3: //team-bucket/configs/seqera/seqera.config ./seqera.config echo $NXF_CONFIG_BASE64 | base64 -d > /tmp/decoded_config echo "" >> /tmp/decoded_config cat ./seqera.config >> /tmp/decoded_config export NXF_CONFIG_BASE64=$(cat /tmp/decoded_config | base64 | tr -d '\n') ``` This works but has a run into problems when users need to modify settings for their specific pipeline. It also has issues of requiring an external S3 bucket configuration to support this. And since we are utilizing the Pre-Run Script like this for all runs, it means that its more precarious for users to modify the Pre-Run Script block themselves. It also causes conflicts when users need to modify settings in their pipeline for things like Nextflow process or executor scopes that would conflict with what we have in our global config here. Proposal Ultimately I would like to get rid of this methodology and replace it with something that worked more smoothly directly in Seqera Platform. The Nextflow configs should themselves be hosted directly in Seqera Platform as independent, versioned, managed resources, and not solely as text-fields glued to the Compute Environment or Pipeline or Run. This could be implemented by creating a new entry in each Seqera Workspace for "Profiles" which could simply be Nextflow config profiles ( https://www.nextflow.io/docs/latest/config.html#config-profiles ) that are available to apply to pipeline Run's at launch time. This would enable some of the following features; Seqera Nextflow Profiles could be Shared across Workspaces / Orgs or kept Private in a specific Workspace a "default" Profile could be configured for all pipeline runs in a Seqera Platform Organization - important to make sure that this behavior is happening server-side and not client-side to avoid inconsistent behavior between the Platform UI vs. tw / seqerakit / terraform Seqera Nextflow Profiles would be addititive just like with regular Nextflow profiles so that we can have a shared Organization-wide "base" profile configured by default, and users can append their own custom profiles Profiles managed in Seqera should be version-controlled, somehow, so that its possible to track changes and roll back if there are issues by having the config Profiles in Seqera, all users would be able to see the Shared profiles in order to become familiar with the settings needed for their pipeline to run instead of every new pipeline developer having to write their nextflow.config from scratch and constantly missing critical config settings that they did not know they needed for their pipeline to function in Seqera Platform on our infrastructure

The existing sharing mechanism in Seqera Platform allows users to share pipelines across an organization. However, this mechanism: Exclusively supports the sharing of pipelines but does not yet extend to datasets, actions, or compute environments. It follows an all-or-nothing approach, granting access to shared pipelines to all workspaces, even when such access may not be required. A more flexible sharing model might allow users to share: Compute environments Pipelines Datasets/data sources Additionally, users could have the flexibility to define the sharing in a more granular way. For instance, they can specify which workspaces should have access to the shared resources, providing greater control over resource distribution. This enhancement would aim to make resource sharing in Seqera Platform more versatile and user-centric, allowing for a more tailored approach to collaboration and resource accessibility.

When working with bucket objects in clinical (GxP) environments, it would be helpful to add support for the following: Versioning : enable vs. disable versioning for bucket objects Object locking : (Write-Once-Read-Many or WORM) model for storing objects (GxP): - Retention periods: set period for locking an object version so it can't be overwritten or deleted - Legal hold: no expiration date for locking individual object versions

Users may struggle to understand raw bucket paths in Data Explorer (e.g. which project, dataset type, analysis run) leading to potential confusion around their data. We should add: Labels for cloud bucket entries / data-links that include project name, dataset type (raw/QC/processed), analysis or workflow descriptor. Show those labels alongside the path in Data Explorer Labels should be editable by workspace maintainers/admins. Search + filter should work on labels. Benefits: Easier navigation for non-technical users. Less support/handholding from technical staff. Reduced mis-use or mis-selection of wrong data.

At the moment if you want to fix a Nextflow version, you have to use a pre-run script to export the NXF_VER environment version. This is fine, but it's not intuitive for what is a fairly central part of the experience of running Nextflow on Seqera Platform. Suggestion is to have a new field in the UI to set the Nextflow version. This could potentially be when creating a new Compute Environment, ideally with a secondary field which could be set when launching a workflow (either to set new or to override the default in the CE). The field should default to using the latest stable release of Nextflow, but give the option of using the latest edge release instead, or a specified exact release version.

Enable users to use pipeline secrets in the head node, for example in the post-run script so to enable automation and integration with 3rd party tools and APIs

When I share a workflow with someone via URL but have to resume it later for various reasons, it becomes hard to find the final succeeded run for my colleague. While I have the option to re-share the URL over and over again or my colleague could go ahead and search for the run name in the Run's page. The latter would require my colleague to know, that I probably resubmitted the run. Neither are particularly seamless. It would be useful, if submissions belonging to a same run where easier to connect to make it clear immediately that my failed run was resubmitted somewhere and where to find it.

Better real-time logging and monitoring capabilities would help to debug workflow run tasks (specifically in the compute environment) more effectively. The current system does not provide CE logs in real-time, which makes it difficult to diagnose issues promptly.