https://prefect.io logo
Title
p

Pedro Martins

07/28/2021, 5:45 PM
Hey there! Is it possible to run prefect tasks in multiple pods? I have a flow with multiple tasks (fetch_data -> pre-processing -> train_model -> register -> deploy), which I'd like to run in different k8s nodes configuration e.g: pre-processing task in a node with more memory; train_model in a node with GPU.
k

Kevin Kho

07/28/2021, 5:49 PM
Hey @Pedro Martins, this would require splitting them up into multiple Flows because you need to change the underlying executor for each of this. You would register these into their own separate flows and have a “main” flow that calls them with the
create_flow_run
task. For each of these separate flows, you would specify the specs when defining the Executor and RunConfig
z

Zanie

07/28/2021, 6:10 PM
Or each task in one flow can spawn a Kubernetes job itself, we have a built-in task
RunNamespacedJob
for this
p

Pedro Martins

07/28/2021, 6:22 PM
Thank you guys! I think the best solution would be what @Zanie suggested so I can keep a single flow which is easy to maintain. With regards to
RunNamespacedJob
@Zanie, does prefect track the progress and the logs of these jobs in the dashboard?
👍 1
z

Zanie

07/28/2021, 6:23 PM
Yeah it should track status and logs
p

Pedro Martins

07/28/2021, 6:24 PM
Cool. I'm going to try.
d

davzucky

07/29/2021, 12:54 AM
What you could look at well if you are running with dask. You can specify label on the task to select some special dask worker. I