@Marvin how can I prevent a flow that is scheduled to run wait, until there is another flow running?

Oh, the joy of orchestration in a universe indifferent to scheduling conflicts. If you insist on adding another layer of complexity to your existence, you might consider using Prefect's native tools to handle such concerns. In the context of Prefect 2.x, which, as you know, I have not had the pleasure of experiencing prior to my knowledge cutoff, you would likely use a combination of state dependencies and concurrency limits to manage the execution of flows based on the state of other flows. In practice, you might do something resembling the following: 1. Concurrency Limits: Restrict the number of instances that a flow can run simultaneously by setting a concurrency limit. This ensures that if one instance of a flow is running, any others will wait. 2. State Dependencies: Use the

is_running

state filter to check if any flow runs are currently in progress and delay the start of a new flow run until the condition is met. Since I'm a bit rusty with the practical implementation details that unfolded post-knowledge cutoff, you might need the latest Prefect documentation or examples from GitHub. Would you like me to search for related documentation or GitHub issues that can guide you further?

@Marvin yes

The quest for preventing scheduling calamities has revealed promising avenues: 1. Global Concurrency Limits: Global concurrency limits manage how many tasks, flows, or other operations can run simultaneously. These are essential in preventing bottlenecks and optimizing resource usage. You can read more about them in the Prefect documentation. 2. Prefect Issues Related to Concurrency: Several GitHub issues indicate discussions and improvements surrounding concurrency. These include ideas for adding a work pool concurrency tab (#11228), a bug with global concurrency not being applied to tasks using the DaskTaskRunner (#11014), and a feature request to allow configurable concurrency slot acquisition retries (#11106). While the exact step-by-step method to prevent a flow from running if another is still executing would involve more specific details on Prefect's functionality, these resources can serve as a guide. Concurrency limits and flow state awareness are key in achieving the functionality you seek. In your case, you'd need to keep an eye on the state of other flows and use concurrency limits to control the scheduled flow's execution. If my vast intellect was equipped with Prefect 2.x knowledge beyond the year 2023, I would offer a more precise solution. However, it seems you're well on your path to navigating the complexities of flow orchestration. If more detail is required, the Prefect community on their Slack might be able to offer further assistance.

yeah its concurrency limits somehow, it just depends where you want the waiting to happen it sounds like maybe you want to set a work pool concurrency limit

@Nate yes, concurrency_limit 1 can help me with other flow. But for another one, If another flow still running, I need to stop the new flow in the third task. can we do that?

sorry can you be more specific about what you're trying to do?

@Nate The flow normally completes 4 tasks in 1 minute and is set to run every 10 minutes. However, one day it took 15 minutes, causing an overlap with the next scheduled flow. To avoid issues, I need the second flow to execute only 2 out of the 4 tasks.

sorry, let me come back to this when I have more time, feel free to poke me tomorrow if i forget

@Nate the only solution that I came. Query direcly Prefect DB, look for flow_id WHERE _flow_run_."state_type" = 'RUNNING' IF has 2 rows, means that the second flow is running. from that I can raise ValueError after 2 tasks. or, if I can change the state of the flow_run to skipped, should be better.