Hello! I wonder if there is a way to control flow...
# ask-community
m
Hello! I wonder if there is a way to control flows within deployments and check if the flow is not already running? Sometimes, flow runs longer than it should (because of locks on the database) and I want to ensure that the next scheduled flow will not start if there is one already running. Best regards, Mike
r
Hey ! It’s not within deployments but you can use concurrency limit to achieve this : https://docs.prefect.io/latest/guides/global-concurrency-limits/#global-concurrency-limits-and-rate-limits
m
@Robin Niel Thanks for the answer but I think that this is not what I am looking for.
r
It can ensure that no flows start before the one before has ended, is it not what you wanted ?
m
But as far I understand, those limits e.g. on pools will lock just the pool, which I don't want to do. We have dozens of work-pools where every pools has at least 20 flows to deal with. I can't set concurrency limits on them. Referring to concurrency manager, as far I understand this: the limit is within a single running flow, not the whole fleet of all scheduled flows of same deployment, am I right?
r
Yes you are right I see what you mean now. Indeed, the documentation i sent you is about limiting the execution within a flow. For example you could have an entry task that blocks the flow execution until the other one has finished but that would mean potentially having long running flows which might not be ideal I’m afraid the only way would be to use a specific work pool with a limit of 1 to achieve this. Are you limited in creating new work pools ?
m
No, I mean I can create any number of pools I want to have, but we use a infra schema were: • there are 5-6 work-pools • each work-pool has at least 5 queues • each pod is attached to each pool • and each work-queue has its own container