Hi! I am running into an issue where the agent is running out of diskspace. It looks like the problem is that the PREFECT_LOCAL_STORAGE_PATH is filled up with persistant Results when running the flows. Some of my tasks are returning DataFrame’s which a significant amount of data which i assume creates the issue. So therefore, i am asking here on advice on how to proceed. I have thought along the following ideas: • Are there any ways i can limit (by stored bytes, or in time) how persistent the Results should be? • Otherwise, i guess the other option is to rewrite the flow so not the full dataframe is return. (For example dump the data frame to a temporary file which can be read by the next task.) • I guess another option might be to create a clean-up task in the end of the flow that delelets the stored results. This seems to me be the ugliest solution. What would you recommend? Or is there anything that i am missing?

I have a similar problem and have thought about writing not only a @task to clean up the files, but also a flow to delete all of the folders prefect creates when running a flow. Let me know if you come up with a better solution

Feels good to know that someone is running into the same issues 😉 Yeah, i think i will for now implement a clean-up task.

thanks for asking; this is an ongoing work in progress and we plan to keep you updated about the progress via release notes and #CKNSX5WG3 channel I'll share with the team that this is important and urgent for you

Thanks! @Anna Geller

As Anna mentioned, an improvement is in the works. In the meantime you can set local storage to a tmp directory.