https://prefect.io logo
Title
h

Haotian Li

02/08/2023, 9:09 AM
Hi Prefect Team, first a thank you to the wonderful job you have done creating this platform! I have a question about heterogeneous flow run: 1. Is there any way for a user to create a flow that runs on different type of hardware in the queue? I need to create a flow that partially run on CPU kubernetes queue and partially on GPU enabled nodes. There doesn't seem to be a way to do this and each flow can only run on pure CPU or all with GPU. 2. If not, what's the best practice for doing this? Should I break a flow into separate subflows and deploy them separately and run each step by hand? Or can I trigger this automatically by having a task run other deployed flow in a different queue. 3. Is there any plan to add per task tagging system for agents so when an agent pulls from a queue it only pulls tasks with a certain tag
t

Tim-Oliver

02/08/2023, 3:35 PM
I approach this by deploying different flows and compose them in a meta-flow. https://docs.prefect.io/concepts/flows/#composing-flows
h

Haotian Li

02/08/2023, 4:11 PM
Doesn't the metaflow need to be submitted to an agent queue as well? Also the doc page doesn't contain any information about metaflow
t

Tim-Oliver

02/08/2023, 4:15 PM
Sorry, in my mind meta-flow is the flow calling the other flows. But the meta-flow itself is also just a normal flow. And you are right, it needs to run via an agent as well.
h

Haotian Li

02/08/2023, 4:17 PM
So it still can't be used in different hardware environment.
t

Tim-Oliver

02/08/2023, 4:24 PM
But you can have a gpu-queue and a cpu-queue with agents listening to one or the other based on where they are running. The agent on the GPU machine only polls work from the gpu-queue
The tricky part will be moving data between the two. You would need some storage to which both have access
I think I was wrong here. I think my understanding of how this works is not correct.
h

Haotian Li

02/08/2023, 4:46 PM
Ah, thanks for taking the time to answer this. I think this is the solution of using a flow to trigger another flow
t

Tim-Oliver

02/08/2023, 5:04 PM
I am currently stuck on a very similar problem, where I want to submit the same flow multiple times from a meta-flow. Let me know when you figured it out.
r

redsquare

02/08/2023, 5:23 PM
can you not just have two deployments that have different nodeSelector/Affinity rules to target the relevant node
h

Haotian Li

02/09/2023, 5:49 AM
@redsquare Yes, but still you need to run one flow on a singular node type.
@Tim-Oliver Sure thing