Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Andrey Vinogradov
04/15/2022, 9:14 AMHi! Is it possible to increase max scheduled flows in prefect? I found “max_scheduled_runs_per_flow” parameter but it seems that is is only available in prefect server not prefect.
a
Anna Geller
04/15/2022, 10:20 AMThat's correct, for Prefect Cloud this is managed globally so you don't need to worry about it. Can you explain the problem you face? Why would you want to adjust it?
By default, this is set to 10 - "The scheduler periodically queries for flows with active schedules and creates flow runs corresponding to the next 10 scheduled start times of the flow."
but if you want to limit the number of concurrent runs for a specific flow or task, you can use concurrency limits
a
Andrey Vinogradov
04/15/2022, 10:26 AMI use Prefect installed on my server not Prefect Cloud and the problem is that that limit is too small. Is it possible to increase it?
As I understand it is possible in Prefect Server not in regular Prefect?
a
Anna Geller
04/15/2022, 10:29 AMthere is Prefect Core (the main package allowing building flows and running them locally)
then there is also Prefect Cloud or Prefect Server that you may use for the backend
Which of those did you install?
• Concurrency limits are only for Prefect Cloud in Prefect 2.0
• max_scheduled_runs_per_flow is only for Prefect Server and this is a global setting allowing you to now overwhelm your Prefect Server
btw if you're just getting started, you may use Prefect 2.0 instead
limit is too smallwhat are you trying to do? explaining your use case would help immensely
a
Andrey Vinogradov
04/15/2022, 10:33 AMI installed Prefect with
pip install prefectI mean I haven’t installed anything else as the backend
We have ml training flow that we run in prefect by schedule but because of this limit we should manually restart schedule for this flow.
We want to plan more than 10 runs for this flow in schedule
a
Anna Geller
04/15/2022, 10:48 AMgotcha, I think this docs page will clarify your confusion: https://docs.prefect.io/orchestration/getting-started/set-up.html
you don't need to worry about the 10 runs. It's just internal detail of the scheduler service, you may run hundreds of flow runs simultaneously
a
Andrey Vinogradov
04/15/2022, 10:49 AMOk thank you!
@Anna Geller in fact we need run this flow not at the same time but scheduled. So for example we want to plan to run it every 2 hours. Will it work?
k
Kevin Kho
04/15/2022, 1:32 PMYes it will, but it’s really recommended to either use Prefect Cloud or Server instead of Prefect Core alone. In core you have to leave the processes running for each Flow. Cloud has 20000 task runs free every month, which is more than enough to get started