Hi friends, a question on memory usage. Currently, my team is only using Prefect Core, which we setup as a bunch of ECS tasks (each corresponding to what Prefect Cloud/Server calls a project) that runs the flows using flow.run() with a schedule. It works, but we found that the memory consumed by the flows was not released after the flow ended. As a workaround, we kill the ECS task after detecting that all the flows for that task have been run for the day, and then ECS kindly restarts the task for us with the memory usage reset. Not ideal, but the hack works so that subsequent flows for the day can reuse the memory released to the EC2 instance. In the coming few days, we're transitioning to Prefect Server using a single local agent with multiple projects, with both agent and flows (since the agent is local) in one ECS task. At startup, the ECS task will create the projects and register the flows, then start the local agent. Eventually we aim to move to use an ECS agent with Fargate, but for now: will we experience memory hogging with this setup?

Hi @Amanda Wee! Ideally we could figure out where the memory usage is coming from with

flow.run

, but honestly we don’t generally recommend using that for long-running or production flows - note that if you use any task caching then the cache is held in-memory with

flow.run

, so that could explain it; when you switch to orchestrating / scheduling with a Prefect backend, then each individual run of the flow will be submitted to its own subprocess (when using the local agent), in which case your run’s memory will get cleaned up naturally when complete, so I think you should be good to go once you migrate to the new pattern!

Thank you, that's great news that bolsters my case that my team should switch to prefect server in production even though we're cutting it close to Christmas due to some deployment hiccups earlier this month. For the past couple of months we've been plagued by having to monitor the ECS tasks for out of memory issues, and restarting them with an increased memory allocation when that happens, and that's something I'm hoping to avoid for the holidays!

Eek yea I totally understand - good luck and let us know how we can help get you setup!