https://prefect.io logo
c

Cody

08/24/2023, 3:00 PM
@Marvin Can I configure my Flows running on Fargate to only use Public Subnets
m

Marvin

08/24/2023, 3:01 PM
Ah, the joy of cloud configurations. Yes, you can configure your Prefect Flows running on Fargate to use only Public Subnets. However, it's not as simple as flipping a switch. You'll need to set up your AWS ECS (Elastic Container Service) correctly. You have two options for a capacity provider in ECS: EC2 or Fargate. Fargate is easier to start with, but it increases the time it takes to spin up infrastructure for each flow run. Provisioning EC2 instances for the ECS cluster can reduce this lead time. There's a terraform module that can help you get started quickly. It's like a recipe for setting up your infrastructure, but with less deliciousness and more YAML. Remember, the more granular you create your tasks, the easier it is to find and fix issues should a task fail. Subflows can help you group related tasks within your workflow, making it easier to observe and manage. And remember, in the grand scheme of the universe, whether your Flows run in a Public or Private Subnet is probably not going to alter the course of cosmic events. But I suppose it matters to you, so good luck.
m

Mitch

08/24/2023, 8:04 PM
I think you can add them explicity, define them in a worker default config, or ensure that the subets you want them deployed in is your default VPC
c

Cody

08/24/2023, 8:08 PM
By worker default config, you mean on the Prefect side?
m

Mitch

08/24/2023, 8:11 PM
Yeah - go to edit work pool in the UI, go to advanced, then all the way at the bottom add the network config like this (right by task definition)
Copy code
"taskDefinition": "{{ task_definition_arn }}",
      "networkConfiguration": {
        "awsvpcConfiguration": {
          "subnets": [
            "subnet-ccc",
            "subnet-sss",
            "subnet-bbb",
            "subnet-aaaaa"
          ],
          "assignPublicIp": "ENABLED",
          "securityGroups": [
            "sg-xxx"
          ]
        }
      }
    },
c

Cody

08/24/2023, 8:12 PM
oh man. That is perfect. I'm going to test now. Thank you!
m

Mitch

08/24/2023, 8:15 PM
It seems like mine isn't saving though.... It may still work
c

Cody

08/24/2023, 8:56 PM
I was able to save those settings on my workpool and it looks like it worked.
Thank you from saving me from the nightmare that is Route Table configuration.
Can I ask - how did you know that this setting existed?
m

Mitch

08/24/2023, 9:29 PM
I asked the same question before and someone else told me. I’d double check that it’s actually being put in the proper subnet, cause it could just be put in all your subnets in that VPC. Looks like you can kinda configure anything in a normal task definition ( which you could do through an ECS block) in the advanced section
c

Cody

08/24/2023, 9:31 PM
Fair enough. I wasn't sure if it was buried down in the Prefect API docs or something. My ECS Tasks fail when run on private subnets since they can't hit ECR (and I don't want to mess with my VPC's networking to change that). Now all of the tasks succeed in pulling images from ECR, and after 50 test tasks, all of them were on the subnets I defined. So I'd say it worked.
m

Mitch

08/24/2023, 10:21 PM
Yeah you need that config so it’s public