Platform needs the ability to apply configs globally to all pipelines
acknowledged
C
Charcoal Mandrill
There was a past Feature Request for this here https://seqera.canny.io/feature-requests/p/ability-for-seqera-platform-to-apply-custom-config-files-to-all-selected-pipelin which is now marked as "Completed", however it appears that this feature was not actually implemented.
As described previously, we need to be able to control pipeline executions across the entire Seqera Platform instance with a global set of configs that would be applied to all pipeline runs.
The solution for this which has been implemented only ties configs to a single Compute Environment. This is not adequate because the configs do not work if you are manually adding CE's across workspaces, and they get lost and need to be re-created when you re-build the CE's (which seems likely to happen every time a new vuln is found in the default base Linux AMI's). And we have encountered a large number of other problems related to this exact situation of being forced to attempt to glue our required global configs to CE's.
Ultimately what is needed are configs that are truly globally accessible and managed and applicable throughout the entire Platform instance. And they need to take effect regardless of if a pipeline is launched from the Platform front-end, from
tw
, or from the API endpoints.
Edmund Miller
Charcoal Mandrill In my mind, it would slot in right next to where platform spikes in platform or tower-Forge configs.
I'm also thinking you could use a repo like nf-core configs and "Git-ops" the whole thing kinda like this feature in Tailscale: https://tailscale.com/kb/1204/gitops-acls
So "on-push" a GitHub action would deploy your configs across Platform instances, orgs, workspaces.
C
Charcoal Mandrill
Edmund Miller
> In my mind, it would slot in right next to where platform spikes in platform or tower-Forge configs.
This is what I had previously requested several times now I think. You can see on the "Run" page, under "Configuration", there are usually two or more config files listed, one of which is always
/nextflow.config
which I am assuming is a "global" Nextflow config that Seqera Platform is supplying to every single pipeline run that contains various configurations (such as the AWS configs which get automatically populated in each run regardless of the AWS settings in the codebase's own repo)In the past I had put in several Support Requests trying to figure out how we could utilize this such that our required internal infrastructure configs could be embedded in every pipeline run handled by Seqera Platform. I think the conclusion was that this was not possible for some reason. I dont really remember why, but this is pretty much the exact feature we need and have been requesting for a long time now.
Having to take manual or automated scripted actions just for the sake of correctly configuring the Seqera Platform's configs feels like an anti-feature. This should really just be part of the Platform. Some kind of Organization-level Nextflow config or global Platform-wide config that can be included in all Pipeline runs on the Platform
Another thing to keep in mind; it seems like it is still not possible to edit a Compute Environment after its been created. So any proposed solution that includes "updating" various configs across Platform to include the desired Nextflow config does not work because the Compute Environment (not updateable) has been given the "global configs" in this current implementation. So I dont think that is an option either.
C
Charcoal Mandrill
Edmund Miller also a sidenote, that TailScale web page is blocked for me.
Rob Newman
Thanks for the new feature request Charcoal Mandrill and apologies that the previous completed request didn't resolve your larger issue. We'll review this with our internal SciDev team and provide an update.
Rob Newman
acknowledged