hi! I'm just reading about prefect and have a basic question. Can you fan out kubernetes pods in a dynamic dag with prefect? E.g. given an

env_var

with 10 parameters, fan out to 10 pods when an initial task has finished. Basically I mainly want to use prefect as a plain orchestrator of pods like airflow pod operator. I see in the docs it documents a similar use case with Dask, but couldn't see an example which did similar with kubernetes pods.

Hi @Eamon Keane do you plan on having those 10 pods be tasks in your flow? This is generally accomplished by using Dask as you mentioned and in the case of running your flow on Kubernetes you could use something like the DaskKubernetesEnvironment which could be configured to create 10 workers (as pods) to run your work in parallel. If these pods aren’t particularly tasks in your flow but are just things you want to orchestrate in Kubernetes you could use some tasks in the task library like the CreateNamespacedPod task or make your own!

thanks josh - these would indeed be tasks in the flow.... would

createNamespacedPod

work in this case? This is basically what happens with kuberentes pod operator in airflow where the status of the pods is monitored (via kubernetes api) and results reported back to task db.