Hi all, I've got kind of a dumb question. If I already have a bunch of dask.delayed functions, and I'm going to pull them into a prefect, what's the best way to do that? Should I call compute on them within the prefect Task?

Hello @Hugo Shi, Depending on your use case, it may be prudent to write these functions into prefect tasks, construct a Prefect flow context, and execute your flow using a parallelized task executor like the DaskExecutor. This may not account for all scenarios using dask.delayed, but it should be enough to get your parallelism running initially.

I can definitely do that - I'm more asking what the best practices are for folks that already have dask delayed (for example if I'm already doing EDA with a bunch of dask delayed functions), is the suggestion to re-cast them as prefect tasks? I can definitely keep my raw funcs separate, and turn them into delayed or prefect tasks, depending on my use case

I'm not a heavy dask user, but I'm definitely interested in what others have been able to accomplish here. 👂

My thoughts are that prefect Tasks should be higher level synchronous things, and so if I"m using dask delayed, probably i should do a bunch of them in the dask, and then call compute on the results at the end. Someone telling me that I'm spot on or way off would be appreciated =)

Thoughts @Jim Crist-Harif?

We're still working out patterns for users to make use of dask from inside of prefect. Right now I'd say that you're probably fine to use any dask stuff inside a task, provided that you don't return dask objects from the task. Prefect handles return objects in special ways (caching the object somewhere, serializing it between workers, etc...), which can do weird things with dask collections.

@Jim Crist-Harif thanks!

Thanks for the question, this is also something I was thinking/wondering about today!

@Marvin archive “Should I return Dask delayed objects from Prefect tasks?”