Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, can anyone tell me if prefect 1 keeps task results in memory for the duration of the flow? Especially with DaskExecutor?

We have a flow that handles some large outputs/inputs at and early stage in the flow, but they have no utility for the end result of the flow. It seems like they might be held in memory throughout the entire duration of the flow.

I notice there is an option `cache_result_in_memory=False`  in prefect 2, but does anybody know if there is a way to disable memory caching in prefect 1?

With DaskExecutor specifically, the dask scheduler manages memory

I believe there was an option to be more aggressive with memory reclamation, I’ll have to see if I can find it again

<https://distributed.dask.org/en/stable/memory.html>
<https://distributed.dask.org/en/stable/worker-memory.html#automatically-trim-memory>