Hi!
I am running into an issue where the agent is running out of diskspace.
It looks like the problem is that the PREFECT_LOCAL_STORAGE_PATH is filled up with persistant Results when running the flows.
Some of my tasks are returning DataFrame’s which a significant amount of data which i assume creates the issue.
So therefore, i am asking here on advice on how to proceed.
I have thought along the following ideas:
• Are there any ways i can limit (by stored bytes, or in time) how persistent the Results should be?
• Otherwise, i guess the other option is to rewrite the flow so not the full dataframe is return. (For example dump the data frame to a temporary file which can be read by the next task.)
• I guess another option might be to create a clean-up task in the end of the flow that delelets the stored results. This seems to me be the ugliest solution.
What would you recommend? Or is there anything that i am missing?