Pipeline Versioning - Create, track, and launch multiple pipeline configurations
planned
Anton Tsyganov-Bodounov
This project will introduce versioning capabilities to Seqera Platform, allowing users to create, save, and reference different versions of pipelines based on their configuration and parameters.
Planned Features
Automatic Version Tracking
- Automatic version creation whenever a pipeline is created, modified, or launched with edited restricted parameters
- Checksum-based tracking for pipeline integrity and provenance
Version Management
- Users with pipeline edit capabilities can assign custom labels to pipeline versions
- Ability to set any version as the default for launch
- Option to save completed workflow runs as new versions
Restricted Parameter Control
- Users with pipeline edit capabilities can modify restricted parameters at the pipeline level
- Locked configurations for users with launch-only permissions to ensure controlled pipeline execution
- Support for custom nextflow_schema.json files to define editable parameters
Version Selection at Launch
- Default version shown in launch form for all users
- Users with pipeline edit capabilities can select and launch any version
- Launch-only users can select from available labelled versions
Commit ID Tracking
- Store commit IDs alongside revision information for deterministic pipeline execution
- Option to pin to specific commits or use latest branch updates
Target Users
This functionality is intended for bioinformaticians who customise platform pipelines for themselves or their users, applicable to all customers including Enterprise requiring pipeline execution control.
Anton Tsyganov-Bodounov
planned
Anton Tsyganov-Bodounov
Merged in a post:
Customization of pipelines' launch interfaces
C
Concrete Hornet
We would like to build custom launch interfaces for nf-core pipelines in our Seqera Platform instance. By that I mean a restricted, minimal interface that only allows to set a minimal set of parameters while the rest is either frozen to a predefined default, or derived from these parameters.
For instance, for the nf-core/demultiplex pipeline, the only thing we'd like launch users to be able to set is the flowcell ID, the filename of the samplesheet and to which project this flowcell belongs. The input and output path shall be derived from the flowcell and project ID, and other options (e.g.
demultiplexer
) shall not be editable by the user. For instance, to derived the outdir from the project_id
, we'd envisage something like params.outdir = "s3://some-company-namespace-${params.project_id}/demultiplex"
.To achieve this, for now, we built a custom Nextflow pipeline that wraps the nf-core pipeline with a custom schema. Example. However, ideally, the same could be achieved directly in Seqera Platform. For that, I believe the following would be necessary:
- support for custom schemas outside the pipeline repo, see also
- support for custom nextflow.configoutside the pipeline repo
- some sort of scripting to manipulate params and possibly input samplesheets (i.e. generate params and samplesheets from whatever is specified in the custom schema).
Anton Tsyganov-Bodounov
Merged in a post:
Pipeline details: versions support
B
Brass Wildcat
In the platform, each pipeline is linked to a specific repository version and a predefined set of parameters. This feature will enable associating multiple configurations and/or versions to a single pipeline to boost flexibility and simplify user interaction.
Rob Newman
Merged in a post:
Allow custom schemas from a second source
Rike Hanssen
At the moment custom schemas can be set in the pipeline launch form, but they have to be in the same repository as the pipeline code.
When adding community pipelines (like nf-core) that custom schema would need to be available in the same repository.
To make this process easier, it would be helpful to be able to add a schema from another location like a raw github URL or s3 path.
Rob Newman
Merged in a post:
Improved support of schemas of multi-revision workflows
M
Mighty Bird
Dear Seqera-Team,
As of Version 23.2.0, the Seqera Platform has poor support for launching multi-revision pipelines and the UI facilitates the misconfiguration of parameters.
Therefore, I wanted to point out a potential UI/UX improvement.A user launching a run via Seqera Platform’s browser interface is first presented with a form for entering pipeline parameters. Similar to nf-core launch, this form is based on the
nextflow_schema.json
file. Only if all mandatory parameters were provided, it is possible to proceed to the Launch settings. Here, a different workflow revision, e.g. an older version, may be selected. Unfortunately, it is rather the rule than the exception that parameters and schema change between workflow revisions. Hence, the current UI does not allow configuring launches of older revisions properly.
For example,
nf-core/taxprofiler
uses the parameter kraken2_save_readclassification
up to revision and 1.0.1 and kraken2_save_readclassifications
for newer releases. Even if a user is aware of this change, it is not possible to provide the correct parameter kraken2_save_readclassification
when launching older revisions, since it will be validated against the most recent schema. When providing kraken2_save_readclassifications
instead, it is ignored for older revisions and the pipeline’s default is used. Moreover, it happens silently, so a user can easily fail to notice that the workflow is not executed as planned.To no longer promote misconfiguration via the UI, I suggest putting the revision choice first and thereafter displaying the parameter form, based on the actual
nextflow_schema.json
of that particular revision: bash git --git-dir="</path/to/pipeline.git>" show <revision>:nextflow_schema.json
If the schema is for technical reasons only read when adding or updating pipelines, and looping through all revisions is not feasible, it should be possible to disable the parameter form in favour of a general text area to enter all parameters at once.
Thanks a lot for consideration.
Ticket attachments:
- SchemaRender.png
- RevisionSelector.png