Improve Google Batch array task creation speed
acknowledged
Rob Newman
When using Google Batch with array tasks, creating an array of ~1,000 tasks takes 6-8 minutes despite the head job having 16 cores with low CPU utilization. Array creation appears limited by serial task packing. This bottleneck prevents scheduling from outrunning task completion on fast sub-second tasks, significantly increasing total benchmark runtime. Parallelizing array creation or using batch submission APIs would reduce this overhead.
Rob Newman
updated the status to
acknowledged