Has anyone come across issues with Scheduled Flows not releasing the memory?
We have an ETL setup to run every day that will just consume additional memory on each run. Are there any easy gotchas that can cause this behaviour before I begin digging into this?
02/09/2020, 9:54 PM
The first thing that comes to my mind is caching; if you have any tasks with the
kwarg set, the cache lives entirely in-memory for Core-only workflows
02/09/2020, 9:58 PM
Nothing set for that yet (something I am excited to use soon though)
Okay found my issue, it’s the fact that two consecutive flow “results” are stored in memory before being released. I suspect this is so Prefect can compare Flow1 to Flow2 but it did mean I needed to plan for double the memory requirements of one run.
This snippet sort of demonstrates what I am suggesting. I don’t know if this behaviour is possible to stop as I don’t need the results of the last Extraction in the next one, just a thought