Jacob Blanco
03/30/2022, 9:18 AMAnna Geller
03/30/2022, 10:40 AMprefect work-queue create your_queue -t etl_jobs
this generates UUID and then you can set:
prefect work-queue set-concurrency-limit WORK_QUEUE_ID 5
and then you create an agent process for this work queue:
prefect agent start WORK_QUEUE_ID
Then you do exactly the same thing for ml_jobs
tag and limit 2.
I believe you understood and implemented everything correctly but what confused you is that you considered the agent as sort of "the thing that executes and runs everything". A more helpful mental model would be:
• the EC2 instance is your agent
• the result of prefect agent start WORK_QUEUE_ID
is a single lightweight agent process
• a work queue is a deployment filter mechanism to ensure that a specific agent process polls only for the relevant deployments and ensures that concurrency limits for those deployments are communicated to the agent
• each lightweight agent process polls only for the relevant deployments (as defined by work-queue tags, flow runner types, or specific deployment IDs), and deploys only as many as it's allowed to (as specified by the work-queue concurrency limits)Jacob Blanco
03/31/2022, 12:29 AMAnna Geller
03/31/2022, 12:34 AMJacob Blanco
03/31/2022, 12:36 AMAnna Geller
03/31/2022, 12:47 AM