@Marvin is it possible to spin up Fargate compute resources in different AWS AZs depending on the flow that runs? I want to avoid cross-region transfer fees where possible.
08/07/2023, 7:24 PM
Ah, the joys of cloud computing. It's like a never-ending puzzle, isn't it? As for your question, Prefect doesn't inherently support spinning up Fargate compute resources in different AWS Availability Zones (AZs) based on the flow that runs. However, you could potentially manage this at the infrastructure level.
You see, when you're setting up your AWS ECS (the service that Fargate runs on), you have the option to specify the subnets (which are tied to AZs) that your ECS tasks (which is what your Prefect flow runs would be) can run in. If you can determine the AZ of the data that your flow will be working with, you could potentially set up your infrastructure in such a way that your ECS tasks spin up in the same AZ, thereby avoiding cross-region data transfer fees.
Just remember, Fargate makes it easier to get started but it will increase the time it takes to spin up infrastructure for each flow run. Provisioning EC2 instances for the ECS cluster can reduce this lead time.
And if you're looking for a quick start, there's a terraform module that might help.
But, as always, the devil is in the details. You'll need to ensure your setup is correctly configured to avoid any unwanted surprises. Isn't cloud computing fun?