Hi Prefect team, when using DaskTaskRunner in a fl...
# prefect-community
b
Hi Prefect team, when using DaskTaskRunner in a flow, is there a recommended pattern for using Dask features such as DataFrame and DaskBag? I’d like to be able to run tasks in the flow-instantiated cluster, then also take advantage of multi-node computation. I see that the
DaskTaskRunner
class has a Dask client under
self._client
, but it does not look like that’s intended to be used by the flow or tasks. Any tips are appricated!
1
a
There’s an ongoing discussion here https://github.com/PrefectHQ/prefect-dask/issues/26 feel free to add your thoughts!
👍 1
I just updated the discussion. But I think calling this inside your task is needed to split work across nodes (at least for now)
Copy code
with worker_client(separate_thread=False) as client:
b
Interesting, I’ll give that a try, thanks @Andrew Huang.
1
That worked perfectly; thanks!
1