Seqera Pipelines enterprise Pipeline Import should add pipeline containers to our ECR
acknowledged
F
Flamingo pink Python
One of the new exciting features announced at the recent Nextflow Summit was the ability to directly import public Nextflow Pipelines in your Enterprise Seqera Platform ("Tower") instance. https://seqera.io/pipelines/
We are very interested in being able to use this feature. However, we have some considerations that we fear may block or impair our ability to use this. Our company security policies mandate that all containers used on our AWS accounts must be held in our AWS ECR so that they can be made available for security scans, among other things.
This puts us in the situation where, we cannot just add a public pipeline to our Enterprise Seqera Platform instances as-is, we must first introspect the pipeline, dig up all the containers used by it, pull them from their public locations (e.g. Quay.io), and then push them into our ECR. From there, further steps may be required (though these tend to be automated). The biggest hurdle is this step, of extracting all the pipeline's containers and pushing them into our ECR. Then when we add the pipeline to the Platform Launchpad, it will likely require that some supplemental Nextflow Config will be needed in order to change the
docker.registry
, etc., parameters to that of our ECR instead. Note that this also requires that the pipeline itself not have the Registry embedded in the container URI. As we are in the middle of developing this SOP ourselves, it occurs to me that the Seqera Platform instance (Tower) in the position where it could, theoretically, just do all of this itself, automatically. Upon importing a pipeline, especially from https://seqera.io/pipelines/ , the Enterprise instance should just automatically extract the list of containers and push them into our ECR and then include supplemental configs for that pipeline to only use our ECR, not the public containers.
Is a feature like this available? If not, would it be possible to have something like this developed? Otherwise we are going to have to develop it ourselves and it likely wont be as elegant or robust as what the Platform could have performed or kicked-off itself as part of the pipeline import process.
F
Flamingo pink Python
as a follow-up, I think there are actually two separate components here;
- getting copies of all containers into our ECR
- updating the pipeline in the Launchpad with configs to use the ECR
the latter seems more trivial and possibly also optional based on requirements, and also seems like it might already be possible to implement with
tw
or similar?So I think the more difficult part to resolve is the former, automated import or mirroring of all containers used in a pipeline to our ECR.
I am also wondering if something like an automated "Actions" function could be used for this, or some kind of webhook that triggers a Lambda, or other type of automated action, which could just use
nextflow inspect
followed by docker push
. However I was looking at the Seqera Platform Actions menu and it did not seem clear if something like this was possible yet? Also open to other potential methods to automate the process too.F
Fellow Bee
Wave can mirror all pipeline containers in your desired registry e.g.
wave.enabled = true
wave.strategy = 'container'
wave.freeze = true
wave.build.repository = '00000000.dkr.ecr.eu-west-1.amazonaws.com/my/mirror'
F
Flamingo pink Python
Fellow Bee Thanks this sounds promising, however, based on the docs here https://www.nextflow.io/docs/stable/wave.html#push-to-a-private-repository it sounds like the
wave.build.repository
would contain only the final container after it has been updated with new layers by Wave. I think we would also want to have the original source container mirrored in the ECR as-is as well. Not sure if that is included in this process?F
Flamingo pink Python
an update, we finally got this to work, thanks
seems like we will end up looking into other methods to get the original containers mirrored in ECR separately
Rob Newman
Flamingo pink Python: Based on Paolo's comment and your responses, is this feature request still valid as-is or can it be marked as either resolved, or if not, modified/simplified to what is still missing? Thanks.
Rob Newman
acknowledged