Dear Prefect-Community,
My Senior send me here to politely ask if it is possible to map prefect-task to individual kubernetes pods instead of paralell python-tasks. :)
@Kevin Kho appreciate the fast response!!
Are you able to provide an example? π
k
Kevin Kho
01/12/2022, 4:28 PM
It would be:
Copy code
run = RunNamespacedJob(..)
with Flow(..) as flow:
run.map(body=...)
Where
body
will be a list of dicts.
r
Rouven
01/12/2022, 4:30 PM
Ah! Thank you @Kevin Kho. Keep up the good work and profile picture! π
marvin 1
π 1
k
Kevin Kho
01/12/2022, 4:33 PM
If your tasks are not k8s jobs, then you would use dask_kubernetes to make a Dask cluster that creates multiple pods, and then you would map over that cluster.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.