Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    Hey! Can anyone confirm for me that if a schedule contains more than 10 (concurrent) runs, this is limited to 10 runs only. My schedule contains 20 runs (with different parameters for a flow) where many start at the same time. However, when I deploy this schedule to Prefect only 10 runs are scheduled instead of the 20. This is on Prefect's website: There is no limit to the number of flow runs you can create, even if they start at the exact same time with the exact same parameters: Prefect supports unlimited concurrent execution. This does not seem the case for me.
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @Fabrice Toussaint -- this may be a limitation of our scheduler which schedules 10 runs in the future at a time. I'm not sure it will actually stop your flows from running on time though, I will investigate that.
    The scheduler runs on 5 minute intervals so they could be up to 5 minutes after the time you've scheduled them, but they will run. Does this make a large difference in your use-case?
    Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    Hi @Michael Adkins, thanks for the quick reply! I split it up into multiple flows and make use of the StartFlowRun task from Prefect as this gives me more flexibility. However, still only 10 flows were run in the previous setup even though they were scheduled (or they did not show in the UI?).
    Michael Adkins

    Michael Adkins

    1 year ago
    If they ran they would show up in the UI -- that's peculiar. It's possible they missed being scheduled entirely because of how the scheduler queries for flows to schedule. We'll definitely be looking at improving this soon.
    It sounds like your workaround is a great option for now.
    Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    @Michael Adkins for some reason even with the scheduling the StartFlowRun only the first schedule is actually executed. So I have a flow with StartFlowRuns and a flow which is triggered by that flow. The flow with StartFlowRuns keeps executing and scheduling but none of these tasks actually trigger the other flow to run. Any idea why this happening?
    Michael Adkins

    Michael Adkins

    1 year ago
    It sounds likely that you're encountering the default idempotency key value
    - idempotency_key (str, optional): a unique idempotency key for scheduling the
        flow run. Duplicate flow runs with the same idempotency key will only create
        a single flow run. This is useful for ensuring that only one run is created
        if this task is retried. If not provided, defaults to the active `task_run_id`.
    but I'm not sure, I'd need more details
    Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    An example can be seen below:
    with Flow as flow:
       StartFlowRun().run(flow_name='flowname', project_name='projcctname',
                            parameters={}) 
       StartFlowRun().run(flow_name='flowname', project_name='projcctname',
                            parameters={})
    with this schedule
    schedule = Schedule(clocks=[clocks.CronClock('0 */2 * * *')])
    Michael Adkins

    Michael Adkins

    1 year ago
    Oh, don't call
    .run()
    directly
    start_my_flow = StartFlowRun(flow_name='flowname', project_name='project')
    
    with Flow(...) as flow:
       start_my_flow(parameters={...})
    Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    How do I get it running then or does it start running as in your example?
    without calling
    .run()
    The above might be confusing, so I will rephrase 😛: Do I need to call
    .run()
    or should it automatically execute the flow with your example?
    Michael Adkins

    Michael Adkins

    1 year ago
    So Prefect 'Task' objects are both callable and have a
    run
    method.
    Calling a task is what adds it to your flow
    The
    run
    method is executed at flow runtime
    In your code, your tasks are not added to your flow -- they are just run.
    Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    I will remember this, thank you very much and sorry for the mistake 😛
    Michael Adkins

    Michael Adkins

    1 year ago
    That's alright! It's a bit tricky haha