Steve R
03/24/2022, 3:21 PMKevin Kho
create_flow_run
task, which triggers a new flow run from the backend.
I don’t even know if you can achieve concurrent flows using core. I think the thread gets occupied with one flow and might not kick off other ones. The Prefect agent than connects to the backend handles this for you.
I think this is a scenario where you may have to use Dask to orchestrate Prefect like you are suggesting…but then I’m wondering what Prefect gives you here?Steve R
03/24/2022, 3:28 PMConcurrent flow runs are not supported byflow.run()
from https://docs.prefect.io/core/concepts/flows.html#running-a-flow-on-scheduleis a convenient way to run a flow on schedule, but it does not support concurrent flow runs. It will wait for a run to completely finish, including things like tasks that require retries, before starting the next run. However, Prefect schedules never return start times in the past. This means that if a flow run is still running when another flow run is supposed to start, the second flow run won't happen at all. If you require concurrent runs in a local process, consider using the lower-levelflow.run()
classes directly.FlowRunner
Kevin Kho
flow.run()
Steve R
03/24/2022, 4:40 PMIf you require concurrent runs in a local process, consider using the lower-levelclasses directly.FlowRunner
Kevin Kho
Steve R
03/24/2022, 5:11 PMKevin Kho
Steve R
03/24/2022, 5:15 PMKevin Kho
Steve R
03/24/2022, 5:44 PMKevin Kho