@Marvin As someone new to Prefect, I have a general question about the tool’s capabilities. My team manages ML workflows that require right-sizing infrastructure at different stages of processing. In Prefect, what would be the best way to manage this? Is there the ability to configure tasks to run with different infrastructure configurations / worker pools? Or the ability to orchestrate flows with different infrastructure configurations one after another? What is the reccomended approach for this type of use case. Thank you for the help 🤖

In Prefect 3.x, managing infrastructure configurations for different stages of your ML workflows can be efficiently handled using work pools. Here’s a concise overview of how you can achieve this: 1. Work Pools: Prefect uses work pools to manage the distribution and execution of flows across different infrastructures. Each work pool is optimized for specific use cases, allowing you to configure tasks to run with different infrastructure setups. By switching a deployment’s work pool, you can easily change the worker that will execute your runs, facilitating the promotion of runs through environments or debugging locally. Learn more about work pools. 2. Automatic Infrastructure Provisioning: Prefect simplifies the process of setting up infrastructure by offering automatic provisioning. This is done using the

prefect work-pool create

CLI command with the

--provision-infra

flag, which provisions necessary serverless cloud resources. This feature requires the relevant cloud CLI library installed and authenticated with your cloud provider. Explore infrastructure examples. 3. Concurrency and Managed Execution: Prefect’s managed execution work pools have limitations based on your account tier, affecting the number of concurrent flow runs and managed execution work pools you can have. This is crucial for planning and right-sizing your infrastructure. Read more on managed infrastructure. Given your preference for an imperative programming style, Prefect's flexibility in defining tasks and flows should align well with your coding approach. Remember that in Prefect 3.x, agents have been replaced by workers, and work pools replace the infrastructure blocks from earlier versions. If you have more specific questions or need further assistance, feel free to ask!

Hmm, perhaps I’m misunderstanding, but that advice doesn’t appear to address the requirement. If work pools are specified by deployment (which is essentially a deployed flow configuration), wouldn’t that mean the infrastructure for each task in a deployment will be shared? Or is there an option to change the work pool at the task level?

hey @Kyle Clark! happy to take over for marvin here I would say marvin did roughly address the high level concepts involved, except it missed key enabling features, which is either run_deployment or deployment triggers so say you have • flow A on k8s • flow B on ECS • flow C in a subprocess you write a deploy each of these things, each assigned to their own work pool (i.e. infra config) then you have a choice for how to chain these together • write a "parent" flow that imperatively chains your deployments together like

@flow
def orchestrate_all_the_things(params: ...):
   flow_run_a = run_deployment("flow-a/deployment-a", parameters=dict(...)

   flow_run_b = run_deployment("flow-b/deployment-b", parameters=dict(...))

   flow_run_c = run_deployment("flow-c/deployment-c", parameters=dict(...))

key thing is that

run_deployment

triggers the work remotely in whatever infra is implied by each deployment's work pool and you can optionally add normal python for conditionals and stuff like that • alternatively, you can declaratively chain these deployments with event triggers. for example ◦ flow B should run when flow A produces a

flow-run.Completed

event ◦ flow C should run when flow B produces a

flow-run.Completed

AND flow A also emits

flow-run.Completed

so in the latter world, you put a

trigger

on a deployment other resources you mind find useful • template repo w boilerplate for deploying • getting started yt series

as for the right-sizing part, if your compute sequence needs to determine compute resource allocations at runtime, your

run_deployment()

calls can also accept a

job_variables

dict where you can specify those per the specification of your work pool type. So if your second deployment was determined to need 4GiB of memory just before it's kicked off, you could do something like

ecs_mem_allocation = determine_dataset_size() # returns 4096, for example
flow_run_b = run_deployment("flow-b/deployment-b", parameters=dict(...), job_variables={"memory": ecs_mem_allocation))

Excellent, thank you both for the speedy advice! This is very helpful