Hello, I am using prefect to run some machine learning workload. Each time my run fails with an OOM error related to prefect. It is related to caching results, I am not using output caching so i suspect it is related to the automatic input caching (I am using a 500Gib disk) , can anyone help me with this?

Hey @Oussama Louati, do you have any results configured? Are you returning large objects from tasks (like a model or dataframe)

Hello, Yes i do, we are preprocessing a big dataset, each task in the pipeline will do some preprocessing or transformation on a dataframe and pass it to the next step etc etc.

What agent are you using? Local?

Yes

Yeah you can turn off checkpointing here

export PREFECT__FLOWS__CHECKPOINTING="false"

, will do right ?

Yes. Then you might want to delete some of the results that have been written in

.prefect/results

if you don’t need them anymore

Many thanks, i will test and i hope it works

Try passing

checkpoint = False

to the tasks like this

Is my assumption wrong. I though that perfect is using everything the temp storage to pass really of task to another task. This is a behavior we saw on our side This is why we had to create a workflow to clear the result folder

Prefect persists the output of tasks so that you can start the task from the same spot when you retry. You can turn off checkpointing. Some people explicit use the same filename so it just gets overwritten each time the flow is run

Perfect need to that as well when you use dask to retrieve dependency tasks values as it is not using dask memory. ?

When running it’s memory but when restarting a failed flow it’s from the persisted file

Ok. I though with dask executor it needed the result store thank you for confirming