Are there any plans to support Batch with Prefect?...
# prefect-gcp
m
Are there any plans to support Batch with Prefect? I think it could be a great solution for our pipelines.
d
Also interested in this. Right now, if you want to have separate cpu/gpu/memory/disk resources applied to individual tasks, you'd be better off not using Prefect. Unfortunately, other options like Flyte and Metaflow aren't nearly as mature. A GCP Batch task worker (possibly combined with a local Dask executor on a VM to handle scheduling) would be ideal. It's a shame it doesn't exist yet, since the need for hetergeneous task resources is only going to increase with more and more ML appearing in pipelines.