Hello, my question became somewhat lengthy so I've placed it in the attached snippet. The tl;dr is: Can I

yield

part of a task result for use downstream? If not, what is the best alternative? Or am I just thinking about this all wrong?

Hi @Michael Wooley -- you can't yield data from a task right now (and I do not think Dask will make it easy to implement). I think your best option is writing the data to files then passing a reference to the files to your mapped transform as you mentioned. If you pass the data itself instead of a pointer then you will almost certainly run into memory constraints.

Okay, just wanted to confirm. Thanks @Zanie!