Gopal
08/26/2019, 12:56 PMJeremiah
flow.run()
is indeed a blocking call (it’s really more of a convenience method for some of the low-level machinery), but when paired with Dask the actual execution will be asynchronous and remote.Gopal
08/27/2019, 12:53 AMJeremiah
prefect.schedules.IntervalSchedule
or prefect.schedules.CronSchedule
to your flow and simply call flow.run()
with a DaskExecutor
. The flow.run()
call will block, but the flows will run in the dask cluster. If you require fully asynchronous (each flow launched discretely, no blocking process) then Cron might do it.Gopal
08/27/2019, 2:24 AMJeremiah
Gopal
08/27/2019, 3:55 AMJeremiah
flow.run()
is always a blocking call because the FlowRunner
is doing state management. I’m tempted to say you could simply submit flow.run()
itself to your dask cluster (create a dask client and distributed.fire_and_forget(client.submit(flow.run))
) but cc @Chris White in case that creates an issue with dask worker clients…Gopal
08/28/2019, 12:43 AMChris White
Gopal
08/30/2019, 2:46 AM