Hi!
I am trying to run a flow with mapped tasks with a DaskExecutor on my machine, I have a couple of queries :
1. will the overall run time benefit if I run the same flow on Prefect Cloud?
2. Does prefect cloud manage the spawn of a Dask cluster resource ?
k
Kevin Kho
09/10/2021, 12:34 AM
Hey @Abhas P , local will be faster than Prefect cloud because each task needs to update its state with the Prefect API at least 3 times. These cause an overhead.
Some DaskExecutors allow you to shut down after usage so it’s a setting when you do:
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.