Hi! We have a dbt flow that takes arguments for

--select

,

--exclude

and so on. It works very well, however I'm struggling scheduling "batches" of various components in prefect. Our environment is 1 deployment in 1 work pool with "default" work-queue. The work pool was created by a kubernetes helm chart. Now if I schedule 10 runs for Batch A

runs=10;
for i in {1..$runs};
    do prefect deployment run --id <deployment_id> --param commands='["dbt --no-use-colors ls"]';
done

it will create 10 flow-runs as intended, but all of these flow runs will start immediately. This is not what is desired and these 10 runs must run sequentially one after the next. I've solved this by adding concurrency limit of 1 in the work-pool. Not ideal, but it works! But now I have another batch (Batch B) of flows I'd like to run for the same deployment, but with different parameters

runs=50;
for i in {1..$runs};
    do prefect deployment run --id <deployment_id> --param commands='["dbt --no-use-colors ls -s something"]';
done

The flows within this batch needs to also run sequentially, but the batch itself needs to run at the same time as the previous batch. (See the attached illustration..) Naturally I'd use work-queues for this, but the Cloud UI tells me this is deprecated and should not be used.. what approach is recommended to accomplish this? If I schedule the two batches right now, the all flows from Batch A will need to complete before the flows from Batch B start 😞

Work queues aren't deprecated! We're just making a few changes to how we show their status and keep track of their health, but you should absolutely use them the way you've described 🙂

Ah I see! Reading the tooltip, I had somehow imagined the

work-queues

are going away.. I was wrong. Thank you @Kevin Grismore!