Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all, a bit lost on where to start with this one. I'm noticing huge memory consumption that does not get released when running flows, sub-flows and tasks. I've searched around a bit and understand there are some things I can look into with Python garbage collector. Would like to ask for any tips on other things to dig into as I try to resolve this. At the moment most of my Prefect operations involve downloading data files, some up to 3-4 GB and saving that new version to disk.
Scant details here, but looking for high level tips. Thanks!

I recently added

`@flow(cache_result_in_memory=False)`

and it's made a massive difference

image.png

<@U06CB578ETH> I tried that on the flow and two sub-flows I have.
Here is the output on memory usage.
I'll need to keep digging as the memory is not yet being released at the end of the flow/sub-flows.