Hey folks, I'm trying to do following but can't figure it out. Any kind of pointers will be appreciated. I want to build a pipeline where it loads a file larger than RAM my local machine have. I can do that with Dask alone, but not sure how to do that in prefect task. In other words, is it possible that the first task of a flow load a chunk of dataframe and once it processed, it will load next chunk and so on? Thanks.

I am still very new to this, but it seems like you want to use

Task.map

and be sure start your flow with the Dask executor.

Hey @Rutvik Patel, this is how you would work with Dask Dataframes in a Prefect task

Thanks, @Kevin Kho. It was helpful. 🙂

No problem!