Intelligent Compute should have configs to prefer dedicated EC2's for some tasks
acknowledged
C
Charcoal Mandrill
Related to the Feature Request about NVMe cache in the EC2 here https://feedback.seqera.io/feature-requests/p/nvme-free-disk-space-should-be-a-consumable-resource-for-intelligent-compute-sch and related to some past discussions, just wanted to get a dedicated Request for this as well.
In many cases we are favoring "stability" of Nextflow task execution moreso than "optimization". To this end, I think a lot of headaches could be avoided if we were able to more easily force Nextflow jobs in AWS Batch to be scheduled on dedicated EC2 instances, instead of attempting to pack tons of jobs on fewer EC2's. This especially becomes a concern when things like NVMe starts getting heavily utilized and there are concerns that multiple jobs e.g. using Fusion heavily for large data could end up colliding and potentially cause degraded performance or failures if the NVMe fills up from multiple jobs running at once, or for other reasons.
So overall, we would be interested to have the option to configure Intelligent Compute in a way that favors giving jobs their own dedicated EC2 instances, along with taking any other potential measures, to attempt to ensure stability of task execution.
Thanks
C
Charcoal Mandrill
so I was thinking about this and how it could lead to "inefficient" usage of the EC2 for example if you submit a task and in your Nextflow config, the task has
cpus = 4 ; memory = 16.GB
set, and for whatever reason the task ends up on an EC2 with more CPU or Memory than you asked for (e.g. 8 CPUs and 32GB RAM). It would be nice the Nextflow task be able to "scale up" to match the available resources. Typically we want the
task.cpus
to match 100% of the available vCPU's requested, but task.memory
often needs to to leave 10-20% host memory available to avoid crashing. So in this example, it would be helpful if the Nextflow task could somehow detect that it has been over-provisioned resources and scale the task up to cpus = 8 ; memory = 32.GB * 0.80
. Not sure how hard it would be to implement something like this, but it would be a cool feature. I suspect the implemention could utilize something like env vars that Intelligent Compute could inject into the task environment, combined with Nextflow config settings that could optionally detect those vars and utilize their values in the .command.sh somehow. Or maybe there are other easier ways, not sure.
Regardless, having the option to give Nextflow tasks their own dedicated EC2 then allowing the tasks to automatically scale up their own resource usages to match whatever they get provisioned seems it could be a useful feature even if it kinda goes against the ideas of optimizing the EC2 size to match the task.
Rob Newman
updated the status to
acknowledged