https://prefect.io logo
Channels
Powered by
w

wiretrack

01/03/2021, 12:05 AM
Hey all, happy new year everyone. Hope everyone stays safe and healthy! I’m currently experimenting with an ETL architecture, and I read in the docs that Dask cluster was the preferred way of running a flow. But I couldn’t find examples of using the Executer within the functional API, only with the imperative API (
flow.run(executor=Dask
). I did however started a Dask Cluster locally, and set the environment variables, as per the docs, but my tasks are still syncronous, and not running in parallel, and I can’t see anything in the dask workers logs. is there anything I should be doing to enjoy the speed and benefits of the parallel and distributed computation?
For anybody who comes across this in the future, I fixed by setting the 
executer
and the 
environment
like so:
Copy code
python
```executor = DaskExecutor()
environment = LocalEnvironment(executor)

with Flow('Distributed Flow', environment=environment) as flow:
z

Zanie

01/05/2021, 4:31 PM
Heads up that we’d recommend using the new 
RunConfigs
over environments. https://docs.prefect.io/orchestration/flow_config/run_configs.html
Powered by