https://prefect.io
Join Slack
Hi! Is it possible to increase max scheduled flows in prefect? I found “max_scheduled_runs_per_flow”...
# ask-community
a
Hi! Is it possible to increase max scheduled flows in prefect? I found “max_scheduled_runs_per_flow” parameter but it seems that is is only available in prefect server not prefect.
a
That's correct, for Prefect Cloud this is managed globally so you don't need to worry about it. Can you explain the problem you face? Why would you want to adjust it?
By default, this is set to 10 - "The scheduler periodically queries for flows with active schedules and creates flow runs corresponding to the next 10 scheduled start times of the flow." but if you want to limit the number of concurrent runs for a specific flow or task, you can use concurrency limits
a
I use Prefect installed on my server not Prefect Cloud and the problem is that that limit is too small. Is it possible to increase it?
As I understand it is possible in Prefect Server not in regular Prefect?
a
there is Prefect Core (the main package allowing building flows and running them locally) then there is also Prefect Cloud or Prefect Server that you may use for the backend Which of those did you install? • Concurrency limits are only for Prefect Cloud in Prefect 2.0 • max_scheduled_runs_per_flow is only for Prefect Server and this is a global setting allowing you to now overwhelm your Prefect Server btw if you're just getting started, you may use Prefect 2.0 instead
limit is too small
what are you trying to do? explaining your use case would help immensely
a
I installed Prefect with
Copy code
pip install prefect
and didn’t do anything else
I mean I haven’t installed anything else as the backend
We have ml training flow that we run in prefect by schedule but because of this limit we should manually restart schedule for this flow.
We want to plan more than 10 runs for this flow in schedule
a
gotcha, I think this docs page will clarify your confusion: https://docs.prefect.io/orchestration/getting-started/set-up.html you don't need to worry about the 10 runs. It's just internal detail of the scheduler service, you may run hundreds of flow runs simultaneously
a
Ok thank you!
@Anna Geller in fact we need run this flow not at the same time but scheduled. So for example we want to plan to run it every 2 hours. Will it work?
k
Yes it will, but it’s really recommended to either use Prefect Cloud or Server instead of Prefect Core alone. In core you have to leave the processes running for each Flow. Cloud has 20000 task runs free every month, which is more than enough to get started
175 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by