james.lamb05/20/2020, 8:54 PM
to speed it up. It's clear from the documentation how to do this if you already have a Dask cluster up and running and just want to use it as an executor for a flow run. For my use case, I'd like to start the Dask cluster at the beginning of a flow run and stop it at the end of a flow run. Right now I'm running this flow in a standalone Python process (just a script with flow code that ends in
), not using an agent talking to Prefect Cloud. What is the recommended way to get the behavior I want, where the Dask cluster gets started when the flow run starts and stopped when it stops, without using Prefect Cloud? • Somehow use Dask Cloud Provider Environment or Dask Kubernetes Environment without Prefect Cloud • Extend DaskExecutor by overriding its setup and teardown to start / stop the cluster • something else that I'm missing • it's not possible, use Prefect Cloud Thanks very much!
## Create your Dask cluster flow.run() ## Tear down your Dask cluster
james.lamb05/20/2020, 9:07 PM
. That means that the
code would never be reached, right?
## Tear Down your Dask cluster
james.lamb05/20/2020, 9:13 PM
with these hooks would be a bad idea?
james.lamb05/20/2020, 9:22 PM
, I'll definitely share what I learned and how it might inform the implementation for that issue. Thanks very much for your help!
james.lamb05/20/2020, 9:27 PM