Rob Hardwick

03/02/2023, 2:57 PM
Hi, can anyone tell me if prefect 1 keeps task results in memory for the duration of the flow? Especially with DaskExecutor? We have a flow that handles some large outputs/inputs at and early stage in the flow, but they have no utility for the end result of the flow. It seems like they might be held in memory throughout the entire duration of the flow. I notice there is an option
in prefect 2, but does anybody know if there is a way to disable memory caching in prefect 1?

Christopher Boyd

03/02/2023, 3:19 PM
With DaskExecutor specifically, the dask scheduler manages memory
I believe there was an option to be more aggressive with memory reclamation, I’ll have to see if I can find it again
:gratitude-thank-you: 1

Rob Hardwick

03/02/2023, 3:49 PM
Thanks Christopher will look into this