Yesterday I left about 1000 pipelines running every 1h with a concurrency limit of 250 and when I went back to see the result, I realized that the process consumed almost 100Gb of ram. I was very confused, because none of the pipelines store values locally, they just query an external database and send the data to another API. But even without storing any value in cache it consumed all the cache storage power of my machine. I imagine that it must be the Prefect's own sqlite database that must be consuming so much resource. Is there a way to limit this or make Prefect stop using cache memory?

You can use cgroups, but at that rate, it might just be easier to use a docker agent

@Christopher Boyd, I just guessed that it could be the prefect database (sqlite), but in fact I don't know what is consuming these resources. I reserved a specific machine to run the prefect container, it only runs this container, nothing more. running Prefect in docker container:

OS: Debian 11

server:

OS: Linux Centos 8 
RAM: 128GB
CPUS: 64
Threads/core: 2

flows:

Schedule: 1/1h
Scheduled flows: 1300
Tasks Concurrency: 1
Flow Concurrency (pool): 250

Without knowing what’s actually using the memory, it’s hard to evaluate . You can try the following for starters to sort processes by memory consumption . ps -o pid,user,%mem,command ax | sort -b -k3 -r It’s also still not clear if you are running both server and flow runs simultaneously on this same system ? Lastly , is there an actual issue with the memory being consumed ? Is this causing a problem ? Linux natively uses what it has available to it, and will actively swap out as it needs to.

I know that it is possible to limit the consumption of the container, but it would be preferable to understand what is consuming so much resource.

I would start with

top

and

ps

1 process per flow