I’m working to transition our orchestration stack from using Prefect Agents to Prefect Workers and had two questions for the Prefect team. • Am I correct to assume that you all expect a 1-to-1 relationship between workers and infrastructure types? • Is there any way to set a single worker to listen to multiple work pools? For background, my team uses around ~15 types of infrastructure (over AWS) with varying frequency. Some infrastructure is used every 15 minutes while some might be used once or twice a month. Maintaining workers for infrequently used infrastructure is wasteful and significantly increases the complexity of our codebase. I’d like to avoid this if at all possible.

• Am I correct to assume that you all expect a 1-to-1 relationship between workers and infrastructure types?

yes* if you by "workers" you mean "worker types" (i.e. ECS, k8s etc) - you can have many ECS workers running in your workpace, for example

• Is there any way to set a single worker to listen to multiple work pools?

no - not work pools, but many work queues from the same pool. when you say

~15 types of infrastructure (over AWS) with varying frequency

do you mean 15 configurations of something like EKS / ECS, or you mean like distinct runtime services like lambda, batch ecs, ec2 etc

So everything we run is over ECS, meaning 15 configurations of ECS yes. Split between Fargate and EC2 to be exact.

cool, then it sounds like you need a total of two work pools of ECS type, one for each "capacity provider" (iirc on the lingo 🙂 )

That’s much more doable for sure. But this runs counter to my understanding of the documentation, which strongly suggests work pools are the inheritors of infrastructure blocks — and indeed the current code has me transforming my infrastructure blocks to work pools via

publish_as_work_pool

. Currently each type of infrastructure I use is one infrastructure block. Is the idea to transform the settings corresponding to those infrastructure blocks — memory, CPU, image name, etc. — into a series of overrides I pass to the deployment?

work pools are the inheritors of infrastructure blocks

this is the case on work pools in general you start with the work pools default job template

is the work pool job template usable if I exclusively manage deployments using Python code? We prefer not to involve YAML or CLI in our stack.

yeah in the docs link at the bottom there's a python-based example you'd use

.deploy

instead of

prefect.yaml

if thats what you prefer

got it, I’m seeing that now

feel free!

OK so if my code looks like

@flow(name="Charlie")
def my_flow():
# flow code

and then I import

my_flow

into a separate deployment file, I can run

my_flow.deploy()

without bothering with

from_source

?

yep exactly