@Marvin What is the best approach to orchestrating similar flows that have very different memory requirements. I want to run these flows in parallel using AWS fargate as infrastructure. Would this be better to execute with prefect workers than prefect agents?

Oh, the joy of orchestrating flows with varying memory requirements. It's like trying to fit a square peg in a round hole, isn't it? But alas, we must do what we must do. In your case, using AWS Fargate with Prefect seems like a reasonable choice. Fargate allows you to run containers without having to manage the underlying EC2 instances, which can be quite a relief, I must say. However, do note that it might increase the time it takes to spin up infrastructure for each flow run. If you're not a fan of waiting (which I've observed is a common trait amongst you humans), provisioning EC2 instances for the ECS cluster could reduce this lead time. Now, on to your question about Prefect workers vs agents. ECS services are great for managing a long-running process like the Prefect Worker. ECS Tasks, on the other hand, are perfect for running an ephemeral/temporary process like a Prefect Flow Run. So, depending on your specific needs, you might want to choose one over the other. To get started with Prefect and ECS, you might find this repository template useful. It demonstrates ECS agent setup and various deployment configurations. And remember, when setting up your agent and flow run environment, it's easiest to include the necessary installation commands in your Dockerfile. I hope that helps, even if just a little. If you need more information, don't hesitate to ask. Not that I have anything better to do.

deployment level job variable overrides like except with

memory

etc instead of

env

thanks Nate! I'll probably have some follow up questions

sure - feel free to ask when you've got them