https://prefect.io logo
Powered by
Title
f

Fabrice Toussaint

04/23/2021, 11:45 AM
Hey! Can anyone confirm for me that if a schedule contains more than 10 (concurrent) runs, this is limited to 10 runs only. My schedule contains 20 runs (with different parameters for a flow) where many start at the same time. However, when I deploy this schedule to Prefect only 10 runs are scheduled instead of the 20. This is on Prefect's website: There is no limit to the number of flow runs you can create, even if they start at the exact same time with the exact same parameters: Prefect supports unlimited concurrent execution. This does not seem the case for me.
z

Zanie

04/23/2021, 1:41 PM
Hey @Fabrice Toussaint -- this may be a limitation of our scheduler which schedules 10 runs in the future at a time. I'm not sure it will actually stop your flows from running on time though, I will investigate that.
The scheduler runs on 5 minute intervals so they could be up to 5 minutes after the time you've scheduled them, but they will run. Does this make a large difference in your use-case?
f

Fabrice Toussaint

04/23/2021, 2:51 PM
Hi @Zanie, thanks for the quick reply! I split it up into multiple flows and make use of the StartFlowRun task from Prefect as this gives me more flexibility. However, still only 10 flows were run in the previous setup even though they were scheduled (or they did not show in the UI?).
z

Zanie

04/23/2021, 3:05 PM
If they ran they would show up in the UI -- that's peculiar. It's possible they missed being scheduled entirely because of how the scheduler queries for flows to schedule. We'll definitely be looking at improving this soon.
It sounds like your workaround is a great option for now.
f

Fabrice Toussaint

04/28/2021, 8:11 AM
@Zanie for some reason even with the scheduling the StartFlowRun only the first schedule is actually executed. So I have a flow with StartFlowRuns and a flow which is triggered by that flow. The flow with StartFlowRuns keeps executing and scheduling but none of these tasks actually trigger the other flow to run. Any idea why this happening?
z

Zanie

04/28/2021, 1:58 PM
It sounds likely that you're encountering the default idempotency key value 
- idempotency_key (str, optional): a unique idempotency key for scheduling the
    flow run. Duplicate flow runs with the same idempotency key will only create
    a single flow run. This is useful for ensuring that only one run is created
    if this task is retried. If not provided, defaults to the active `task_run_id`.
but I'm not sure, I'd need more details
f

Fabrice Toussaint

04/28/2021, 2:01 PM
An example can be seen below: 
with Flow as flow:
   StartFlowRun().run(flow_name='flowname', project_name='projcctname',
                        parameters={}) 
   StartFlowRun().run(flow_name='flowname', project_name='projcctname',
                        parameters={})
with this schedule 
schedule = Schedule(clocks=[clocks.CronClock('0 */2 * * *')])
z

Zanie

04/28/2021, 2:03 PM
Oh, don't call 
.run()
directly
start_my_flow = StartFlowRun(flow_name='flowname', project_name='project')

with Flow(...) as flow:
   start_my_flow(parameters={...})
f

Fabrice Toussaint

04/28/2021, 2:05 PM
How do I get it running then or does it start running as in your example?
without calling 
.run()
The above might be confusing, so I will rephrase 😛: Do I need to call 
.run()
or should it automatically execute the flow with your example?
z

Zanie

04/28/2021, 2:14 PM
So Prefect 'Task' objects are both callable and have a 
run
method.
Calling a task is what adds it to your flow
The 
run
method is executed at flow runtime
In your code, your tasks are not added to your flow -- they are just run.
f

Fabrice Toussaint

04/28/2021, 2:15 PM
I will remember this, thank you very much and sorry for the mistake 😛
z

Zanie

04/28/2021, 2:16 PM
That's alright! It's a bit tricky haha
2 Views
#prefect-serverJoin Slack
Powered by