It looks like you want to limit concurrency on a work-queue level based on specific deployment tags. You could attach multiple tags that require this limit on a single work-queue:

prefect work-queue create your_queue -t etl_jobs

this generates UUID and then you can set:

prefect work-queue set-concurrency-limit WORK_QUEUE_ID 5

and then you create an agent process for this work queue:

prefect agent start WORK_QUEUE_ID

Then you do exactly the same thing for

ml_jobs

tag and limit 2. I believe you understood and implemented everything correctly but what confused you is that you considered the agent as sort of "the thing that executes and runs everything". A more helpful mental model would be: • the EC2 instance is your agent • the result of

prefect agent start WORK_QUEUE_ID

is a single lightweight agent process • a work queue is a deployment filter mechanism to ensure that a specific agent process polls only for the relevant deployments and ensures that concurrency limits for those deployments are communicated to the agent • each lightweight agent process polls only for the relevant deployments (as defined by work-queue tags, flow runner types, or specific deployment IDs), and deploys only as many as it's allowed to (as specified by the work-queue concurrency limits)

The way this is currently implemented makes it easier to troubleshoot and is more adaptable to changes in your future workflow requirements. For example, this allows you to: • stop the agent polling for ML-specific work queue without affecting your ETL jobs • easily move the agent process to different infrastructure, e.g. to Kubernetes without affecting any scheduled runs - they remain queued in the work queue and can be picked up once your agent polling for this work queue ID was started again • inspect scheduled runs for ETL or ML jobs separately But curious to hear your feedback on those abstractions. Do you consider this confusing that the work queue and agent are two separate objects? Or do you see it being somehow negative to have multiple agent processes?