Hi Prefect team, when using DaskTaskRunner in a flow, is there a recommended pattern for using Dask features such as DataFrame and DaskBag? I’d like to be able to run tasks in the flow-instantiated cluster, then also take advantage of multi-node computation. I see that the
DaskTaskRunner
class has a Dask client under
self._client
, but it does not look like that’s intended to be used by the flow or tasks. Any tips are appricated!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.