james.lamb
05/20/2020, 8:54 PMDaskExecutor
to speed it up. It's clear from the documentation how to do this if you already have a Dask cluster up and running and just want to use it as an executor for a flow run.
For my use case, I'd like to start the Dask cluster at the beginning of a flow run and stop it at the end of a flow run. Right now I'm running this flow in a standalone Python process (just a script with flow code that ends in flow.run()
), not using an agent talking to Prefect Cloud.
What is the recommended way to get the behavior I want, where the Dask cluster gets started when the flow run starts and stopped when it stops, without using Prefect Cloud?
• Somehow use Dask Cloud Provider Environment or Dask Kubernetes Environment without Prefect Cloud
• Extend DaskExecutor by overriding its setup and teardown to start / stop the cluster
• something else that I'm missing
• it's not possible, use Prefect Cloud
Thanks very much!nicholas
05/20/2020, 9:06 PM## Create your Dask cluster
flow.run()
## Tear down your Dask cluster
james.lamb
05/20/2020, 9:07 PMIntervalSchedule
.
That means that the ## Tear Down your Dask cluster
code would never be reached, right?nicholas
05/20/2020, 9:08 PMjames.lamb
05/20/2020, 9:13 PMDaskExecutor
with these hooks would be a bad idea?nicholas
05/20/2020, 9:19 PMjames.lamb
05/20/2020, 9:22 PMDaskExecutor
, I'll definitely share what I learned and how it might inform the implementation for that issue.
Thanks very much for your help!nicholas
05/20/2020, 9:23 PMjames.lamb
05/20/2020, 9:27 PM