Are there any plans to support Batch with Prefect? I think it could be a great solution for our pipelines.
d
Devin McCabe
03/13/2024, 1:47 PM
Also interested in this. Right now, if you want to have separate cpu/gpu/memory/disk resources applied to individual tasks, you'd be better off not using Prefect. Unfortunately, other options like Flyte and Metaflow aren't nearly as mature. A GCP Batch task worker (possibly combined with a local Dask executor on a VM to handle scheduling) would be ideal. It's a shame it doesn't exist yet, since the need for hetergeneous task resources is only going to increase with more and more ML appearing in pipelines.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.