If i was to setup prefect infra with workers / work pools on EKS with k8s, and i had a workflow where I wanted to have certain steps run on a cheaper cpu based worker (e.g. for data processing), and other steps run on a gpu based worker (e.g. for model training), how would i go about that? • Infra side would i create two work pools - one cpu based pool and one gpu based pool? • prefect client side would i make the different steps flows and use the flow of flows pattern, as i think you can specify different work pools for different flows to run on right? (Specifically how would you do that though?)

hi @Tom Matthews - i think you're on the right track here > one cpu based pool and one gpu based pool? • I would say you could have 2 work pools, one of each kind of execution environment ◦ define the job template however you want, the work pool should start already have sane defaults according to its infra (k8s in your case), but you can hardcode things into the template, add blocks / variables / etc into the template (check out the Advanced tab in the UI on your work pool to see what i mean visually) • write your flows ◦ use

run_deployment

as needed within your parent flow, the flows you invoke with this will be treated / tracked as a subflow of the calling parent (however you can also put this call in a task, if you wanted to leverage the

.map

/

.submit

interface of tasks with deployments) - here's a run_deployment example where we distribute work over child flows • create deployments for each of these flows that point at the appropriate work pool ◦ in your prefect.yaml, you can set your work pool like this (or make things DRY with yaml defs) and then when this deployment runs, the worker will use that work pools job template when submitting to the infrastructure ◦ you can override work pool config (like cpu, mem req etc) on a per deployment basis like this (this examples happens to override env on the work pool, for just this deployment)

Hey Nate sorry for my delayed reply, thanks so much for your response this is very helpful!

Hey Nate. If I could ask a quick clarifying question about this. So essentially, if a subflow is kicked off, for training as an example, the parent flow would have to continue running for the duration of training, correct? At least if there are more flows/tasks to follow, like evaluation/deployment. If training is long, this just seems like wasted compute time in the parent flow. Is there an alternative to this? Or would it make more sense in this case to just finish out the rest of the flow inside the subflow, e.g. training/evaluation/deployment, so the parent flow could just terminate. I guess as an overarching question, do you know how people usually organize an end-to-end workflow like that? Thanks

hey @Andrew - a couple points: > the parent flow would have to continue running for the duration of training, correct? not necessarily, you can say

run_deployment(..., timeout=0)

to trigger the deployment and return immediately ("fire and forget") if you want to avoid blocking the parent flow. one consequence being that if your parent flow needs the result of that deployment you triggered, you'll have to otherwise poll the API to see when it finished and you'll have to

persist_result=True

(with something like

result_storage=S3.load("my-s3-bucket")

) on that deployment you triggered so you can fetch that result in the parent via the api ( e.g.

(await client.read_flow_run(...)).state.result()

). but lots of times, you can just use events to react to the completion of whatever you triggered with

run_deployment

so that you don't have worry about polling logic (instead of gathering child results back in the parent) secondly, we have some work in flight now that will make this easier, basically you'll be able to

.submit

native tasks / subflows from the parent flow as backgrounds processes, without blocking the main process - more to come on that soon!

Sounds good, thanks for the info!

secondly, we have some work in flight now that will make this easier, basically you'll be able to

.submit

native tasks / subflows from the parent flow as backgrounds processes, without blocking the main process - more to come on that soon!

Really looking forward to this, i'm fighting with asyncio in running flows concurrently at the moment. Any idea when this might be released?

the (approximate) plan is to enable subflows to run in this manner in the next couple weeks, tasks hopefully by the end of the year

Great! I guess with tasks you don't need to worry as much as if they're IO bound you can just make them synchronous and just use the ConcurrentTaskRunner and submit them?

I guess with tasks you don't need to worry as much as if they're IO bound you can just make them synchronous and just use the ConcurrentTaskRunner and submit them?

with tasks, we can already sort of "background" them with

.submit

(how exactly you decide to do that, yeah, depends on the details of your use case) but not so much with subflows (they're part of the main process historically). so the plan is to allow you to submit stuff (tasks, subflows etc) as a sort of sidecar process to avoid blocking the main process, we'll share details on how that'll look later on 🙂