https://prefect.io logo
Title
d

David Wang

02/02/2023, 3:00 AM
Hi all. We have a lot of tenants that we want to run the same ELT flow for. The flow is stateful so it’s important that no two instances of the flow runs for the same tenant at the same time. I can foresee this potentially happening if I simply set an IntervalSchedule on the flow because sometimes the flow may take some time to complete, the next run might start before the previous run completes. My thought on accomplishing this is to generate (for each tenant) a work queue with a concurrency limit of 1 and a deployment (with different parameters) for each tenant with that work queue. All of these work queues would share the same prefix. Then I could have agents that could match on the prefix and run the work. The flows could then be concurrent across tenants but also retain the property that there won’t be concurrent execution for the same tenant. Unfortunately after writing the code for this, I found that the agent I started refused to concurrently run more than one flow at a time. Moreover, if I removed the concurrency limit on one of the queues, the agent started running flows from all the queues concurrently, even starting multiple late jobs from the same work queues (and violating the concurrency limits on those queues)! Have I run into a bug or do I just not understand the intended behavior of agents and work queues? EDIT: Fixed, see thread.
Okay, I think I found the issue. I created the work queues with the
tags
argument, not knowing it was a legacy feature that filtered on certain flows with the tag. This resulted in some strange behavior. Removing the
tags
from the work queue fixed it.