Hey all,
We're using Prefect to run our data pipelines. We have a setup where we have separate databases for each of our customers. We want to make sure that at any given time only one flow run occurs on a customer at a time (no two concurrent runs per customer at any point in time). However, we would like to run flows concurrently across the customer databases. Does Prefect support this?
SequentialTaskRunner
for each customer, but
ConcurrentTaskRunner
across all customers.
m
Mason Menges
03/07/2023, 3:35 PM
Would you mind sharing a bit more about how your deployments/flows are configured in this context, technically there you could split this out into different work-pools, one of each customer, with a concurrency limit of 1 but that may not be feasible depending on the number of customers you're working with
a
Anish Giri
03/08/2023, 8:26 AM
I haven't created a deployment yet, in terms of how my flows are configured - can you expand on what exactly you mean? I have one flow that runs a couple of tasks in it. I haven't specified any parameters in the flow. The same flow will run for each customer, just that the target database is different.