https://prefect.io
Join Slack
Good morning! We are trying to run a single flow with many (about 30) different parameters at the sa...
# ask-community
b
Good morning! We are trying to run a single flow with many (about 30) different parameters at the same schedule: all Monday morning 9am. This morning exactly 10 runs were performed, the others were silently skipped. We run Prefect server with a single LocalAgent. It seems like we're running into some default 'max 10' setting. The agent displays as having 10 'submittable runs' -- maybe this can be increased but no hits for 
submittable
in the docs... Any pointers?
w
Looking at the code, there does seem to be a limit, but it looks to be 
100
not 10.. I don’t see the text ‘submittable’ anywhere in the client or server repos; can you paste an exact string?
b
But then again this flow (the only one to be executed by this agent) claims only 10 upcoming runs too:
w
Copy code
query AgentSubmittableRuns {
    flow_run (where: {state: {_eq: "Scheduled"}}) {
    id
    name
    state
    labels
    scheduled_start_time
    version
    flow_id
    }
}
…looks to be the query it runs to get the submittable jobs, which doesn’t impose a limit
What happens when you click on the 
Late
icon? Are the rest of them shown there by any chance?
b
Hey thanks for the help! No there are no 
late
flows
w
Weird; I don’t see anything under the hood that should be limiting you. Hopefully someone more knowledgable has an idea.
k
Hey @Bouke Krom, are you on server ot cloud?
b
server
k
What version are you on?
b
0.15.0
k
I can’t see why there would be any limit. When you say they were skipped, did the flow runs just disappear or did they fail? Is your local agent also capable of handling more than 10 concurrent flows?
b
Found it! The limit of max 10 into the future per flow is defined here: https://github.com/PrefectHQ/server/blob/93b08f97558ac3c620e5ab9e44b781daf23eb9dc/src/prefect_server/api/flows.py#L598 • For our use case I would like to increase the number (at least to 100) • For general use, I'd say that this is a bug. Scheduling more than 10 runs of the same flow at the same moment will fail silently. The scheduling happens here: https://github.com/PrefectHQ/server/blob/93b08f97558ac3c620e5ab9e44b781daf23eb9dc/src/prefect_server/services/towel/scheduler.py#L20 It queries 500 flows at a time so the limit of 10 per flow seems out of balance ◦ Increase 10 to 100? ◦ Even then make sure that flow runs that are skipped are noticed I'd be happy to contribute, but this probably involves a bit more than just PR'ing the hardcoded 
10
. What is the right course of action @Kevin Kho @Jeremiah?
k
Will ask the team about this
Great digging btw! Would you be willing to make a PR? It will probably look like • creating a new setting in 
config.toml
 (in the server repo), something like 
max_runs_to_schedule_per_flow
• replacing the hard coded 10 with this setting
The default of 10 would still be good (some people take down their infrastructure)
b
Thanks! Yes I can make a PR -- I would need to read up on how the 
config.toml
is used etc. Just to check though: • To me it would make more sense if there is a 'time to schedule into the future' (e.g. 1 week ahead or something). This complicates the fix but it could remove the difference in scheduling between 1 or several flows. Is there any appetite for that?
some people take down their infrastructure
Not sure what you mean. Would there be a downside to increasing the default of 10? Since I'm the first one running into this for 13 months it probably doesnt make a difference to anyone 🤔
k
Isn’t it the other way around? You’re the first person to run into it (in the last couple of months) so we don’t need to increase the default? 😅 Not 100% sure what you mean about scheduling int he future? We do have 
start_dates
attached to schedules
And what I meant was that people in the past have run into issues where running multiple concurrent flows kills their machines.
b
Yes, the inverse is also true 😄
Currently the code schedules the first 10 runs for every flow. This only protects against concurrent flow run problems if the runs are from the same flow. The other reason for limiting the number of flows is that schedules can plan ahead indefinitely, so you'd get endless lists. Introducing a parameter with a default is yet another way things can go wrong. Another approach could be to keep scheduling runs for every flow until they are too far into the future, e.g. plan every flow run for the coming week. This 'planning horizon' can be a setting but it only matters for the UI display. That way you're sure every run will be scheduled and there's no setting to get wrong.
z
@Bouke Krom for consistency, I would start with adding a default in 
config.toml
and replacing the hard coded setting. I'd be prefer to keep the default at 10 for consistency too. More advanced changes than that are very tricky because Prefect Cloud also uses some of Server's scheduling logic. For bigger changes, you might be better off writing your own scheduling service or using a custom Server version.
b
OK, clear! I can see how consistency is a key concern too, thanks for the reponse. I'll start working on a PR. Is it customary to make an issue first? (btw I see someone already tried to get the most minimal fix in: https://github.com/PrefectHQ/server/pull/225)
k
I don’t think you need to make an issue first, but it would probably be preferable. I asked Nicholas about this and he said there was just no support to push this through. The implementation with the 
config.toml
is probably the better approach anyway
b
@Marvin open "Make max number of scheduled runs per flow configurable"
k
oh marvin only works for authorized users. 😅. Will do it for you.
😁 1
@Marvin open “Make max number of scheduled runs per flow configurable” in server
m
b
Thankyou!
PR: https://github.com/PrefectHQ/server/pull/281 First time contributing code to Prefect, any feedback welcome
w
Maybe a second test that asserts that the default is 10? Looks good to me personally
k
Looks good! Just wondering, what happens here if someone wants to change the default in the future? Or I guess, does the default 10 really do anything? Whatever value in the 
config.toml
is more important right?
w
He moved that value-default up into the formal parameters of the method, so it should only take effect if nothing is passed along from the config
Now what would be worth verifying is that the config comes across as 
None
in that case, as opposed to something like 
''
👍 2
b
Thanks for reviewing!
Maybe a second test that asserts that the default is 10?
The first 'happy flow' tests for 10 scheduled runs (hardcoded), so that will fail if the default (in config) is changed
Nothing configured in the 
config.toml
Hmm then things will fail. An empty value is not accepted by the toml decoder. If you leave out the value altogether you'd get a 
BoxKeyError
on accessing the attribute from the config object. Configuration values are used everywhere without 
try/excepting
for that case so I assume failing is the way to go.
Btw what is the release cadence of prefect-server?
z
Right now release cadence is "as necessary" for prefect server, usually once a month or so We're working on a few changes in Prefect Core that we'd like to have in before the next server release, so I'd expect one in the next two weeks or so
b
Perfect!
z
@Bouke Krom fyi we just released a new version of Prefect Server https://github.com/PrefectHQ/server/releases/tag/2021.09.02
Thanks again for contributing!
j
Now that’s service!
b
Indeed! Thank you very much!
k
🦜
10 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by