When running Nextflow pipelines with AWS Batch in Seqera, it seems that Nextflow saves the pipeline log file (nextflow.log) and the console stdout / stderr log files to a location under the root dir of the running Docker container

This makes it extremely awkward and difficult to get a copy of these files while the pipeline is running if for some reason they cannot be downloaded from Seqera platform (the "Download log files" buttons in the Seqera Run page are all greyed out).

I end up having to identify the EC2 running the Nextflow head job, SSM into it, identify and enter the running Docker container running the Nextflow head process, copy the log files from inside the container to a location on the host that has hopefully been volume bind mounted by the Docker container, then on the EC2 host upload those files to some location in S3, then finally on my laptop download the files from S3

The process would be a lot easier if the logs were saved somewhere on the host EC2 directly, perferably in a location that is automatically getting synced to S3 while the Nextflow head process is running. So that if Nextflow itself is stalled, the logs would still be in S3 at least.

Would be great if Seqera Platform could ensure this was configured when launching the Nextflow pipelines!