Support S3 conditional writes (If-None-Match) for publish operations
acknowledged
Jon Manning
Add support for AWS S3 conditional writes on publish operations in Nextflow and Fusion, so bucket policies requiring If-None-Match: * can be used to enforce object immutability at the storage layer.
Why?
Strongest guarantee that published outputs are never overwritten, since enforcement lives in S3 itself and also defends against non-Nextflow writers. The publish overwrite: false option is the current workaround but is set in pipeline code (per-process on the classic publishDir directive, or per-output on the new workflow output block), can't be enforced globally at the run or platform level, and only protects against Nextflow.
Current gap
Neither Nextflow nor Fusion sends If-None-Match on S3 writes today, so a bucket policy requiring the header would reject every write, including workDir staging, making the policy unusable as-is.
M
Michael Tansini
marked this post as
acknowledged