Hi Community! Are there guidelines to free up the previous tasks' memory after they are finished? We are running memory intensive pipelines and run into out of memory error. The output of each task is only needed in the next task, but not in the subsequent tasks. Is there room for memory optimization?

You can persist the data and then just pass the location downstream to be loaded in. Are you doing that already?

When you say persist the data, are you referring to storing the data into disk and load it back in the next task?

yeah like

<http://df.to|df.to>_parquet

so that it’s not held in memory because Prefect holds the mapped results and then you pass the location string instead of the df.

Cool. I can do that. But just to understand Prefect a bit better, there is no easy way to clear the memory of task outputs? So python gc.collect has no effect on task outputs?

Lastly, you can always simply delete Python objects if you no longer need them:

del df

Cool cool. Something like del df and gc.collect() should work then. Thanks!

Looks like SO Python folks also recommend just that 😄

del my_object
gc.collect()

https://stackoverflow.com/a/1316793/9509388

You can if you do it inside the task yep