https://prefect.io logo
Title
b

badasstronaut

09/26/2022, 9:13 PM
Hi Prefect team, when using DaskTaskRunner in a flow, is there a recommended pattern for using Dask features such as DataFrame and DaskBag? I’d like to be able to run tasks in the flow-instantiated cluster, then also take advantage of multi-node computation. I see that the
DaskTaskRunner
class has a Dask client under
self._client
, but it does not look like that’s intended to be used by the flow or tasks. Any tips are appricated!
1
a

Andrew Huang

09/26/2022, 10:08 PM
There’s an ongoing discussion here https://github.com/PrefectHQ/prefect-dask/issues/26 feel free to add your thoughts!
👍 1
I just updated the discussion. But I think calling this inside your task is needed to split work across nodes (at least for now)
with worker_client(separate_thread=False) as client:
b

badasstronaut

09/26/2022, 10:30 PM
Interesting, I’ll give that a try, thanks @Andrew Huang.
1
That worked perfectly; thanks!
1