Hi :heart:, I want to enforce that only 1 flow of each deployment can run at a time. For example if ...
f
Hi ❤️, I want to enforce that only 1 flow of each deployment can run at a time. For example if a data transformation does not finish when the next run is scheduled, it should just wait. I am a bit puzzled by the [concurrency limits docs](https://docs.prefect.io/latest/guides/global-concurrency-limits/), I would prefer to set this rule at the global level, but it seems to me like it mostly referst to concurrency limits usage within my flows. Also want to set it as a code (not UI and not the Client) to make sure its reproducible.
1
I am trying to do it with the context manager, have a simple flow that runs some dbt operations:
Copy code
from typing import List

from prefect import flow
from prefect.concurrency.sync import concurrency

from utils import dbt_core_operation_wrapper

@flow
def build_waze() -> List[str]:
    with concurrency(names='build_waze', occupy=1, timeout_seconds=3600):
        dbt_core_operation_wrapper(['dbt build --select waze'])
This created a global concurrency limit
build_waze
, as seen in the screenshot. I manually activated the limit. Now after the first next run Active Slots went to 1, but: • it stays 1 even after no flow is running • it enabled me to do a quick run of the same flow and entered the Running state (so not preventing multiple flows to be run which is my goal)
Found relevant issue on GitHub: • https://github.com/PrefectHQ/prefect/issues/12456
m
Hi @František, I know this isn't exactly what you're looking for, but a workaround I've found is to either cancel currently running flows or wait for it to finish as an initial step in the flow
f
@Mitch Zink yeah, cancelling it manually isn’t really possible for production. The initial step workaround could work. But it seems like an issue that tackles it is under active development here: https://github.com/PrefectHQ/prefect/issues/14934, with eta “end of month”, so that could be easier to wait.
m
Yeah I went with an automated route
p
isn't there a idempotency id to make one flow run at a given time?
f
The functionality from the above issue is deployed in https://github.com/PrefectHQ/prefect/releases/tag/3.0.4, we haven’t upgraded yet, but planning to do so soon