Fusion race condition on GCS when parallel array tasks write to shared files
acknowledged
Rob Newman
When many fast array tasks run in parallel on Google Cloud Storage, concurrent writes to shared files (
.command.begin
, .command.out
, .exitcode
) trigger GCS per-object I/O rate limits (HTTP 429), surfaced as Fusion error 174. This can terminate an entire pipeline. The proposed mitigation for separate per-task file paths may not be fully working. S3 does not exhibit this behavior due to different concurrency handling.Rob Newman
updated the status to
acknowledged