@Marvin I'm running an AWS ECS worker and finding I can only run <50 flows simultaneously before running out of my 2 GB of RAM on the worker node. Why does it take so much memory to start a flow run?

The memory usage of your AWS ECS worker could be influenced by several factors. Here are a few possibilities: 1. Push Work Pools: Prefect Cloud can submit flow runs for execution to serverless computing infrastructure without running a worker. This includes AWS ECS Tasks. If you're using this feature, it could be contributing to the memory usage. More Info 2. Database Overload: There could be a memory problem related to the Prefect database (

~/.prefect/prefect.db

). Reducing the number of logs to avoid overloading the database or setting concurrency limits on some tasks could help manage memory usage. More Info 3. Flow Configuration: The way your flows are configured could also impact memory usage. For instance, if your flow involves a large number of tasks or if the tasks themselves are memory-intensive, this could contribute to high memory usage. Remember that each flow run is a separate process, and each process has its own memory overhead. If you're running many flows simultaneously, the cumulative memory overhead could be significant. You might want to consider optimizing your flows or scaling your worker nodes to handle more memory.