Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Has anyone come across issues with Scheduled Flows not releasing the memory?

We have an ETL setup to run every day that will just consume additional memory on each run. Are there any easy gotchas that can cause this behaviour before I begin digging into this?

The first thing that comes to my mind is caching; if you have any tasks with the `cache_for` kwarg set, the cache lives entirely in-memory for Core-only workflows

Nothing set for that yet (something I am excited to use soon though)

Untitled

Okay found my issue, it’s the fact that two consecutive flow “results” are stored in memory before being released. I suspect this is so Prefect can compare Flow1 to Flow2 but it did mean I needed to plan for double the memory requirements of one run.

This snippet sort of demonstrates what I am suggesting. I don’t know if this behaviour is possible to stop as I don’t need the results of the last Extraction in the next one, just a thought