Hi all. I’ve been facing an issue where my scheduled flows on Prefect cloud crash with no logs and this state message:

Flow run could not be submitted to infrastructure

. My Prefect is using ECS as infra basically following this (I’m new to using Terraform & ECS btw). We’ve been using this set up since the beginning of the year but the crashes started happening out of nowhere ~16 hours ago. Any advice welcome.

Thanks Sarhan, I’ll look into this.

Hey @Will Raphaelson and the wider community as well (any help welcome), Adding more context to this issue, it started because of us making changes to our network ACLs for the default VPC and that impacted the default security group’s outbound rules. What we’re trying to do now is get ECS to use the security group that’s generated when all other resources are created (when terraform is applied based on this Prefect Recipe). The agent task that’s started when the ECS cluster is created uses this security group without issue but any new tasks created from a flow-run triggered from a deployment in prefect cloud uses the default security group which we don’t want to have outbound rules for and so it’s unable to pull the docker container from DockerHub. Is there a way to amend the security group that gets associated to the new tasks? What variables could be set with terraform for this? CC: @Aifred Ong

hey Sarhan - i’m curious to see how you’ve configured your

ECSTask

for the job itself in the flow code (so not the agent), but have you taken a look at the

prefect-aws

docs, specifically around running an ECS task with a specific VPC and/or security group ID? https://prefecthq.github.io/prefect-aws/ecs/#prefect_aws.ecs.ECSTask-attributes

ECSTask(
    command=["echo", "hello world"],
    vpc_id="vpc-01abcdf123456789a",
    task_customizations=[
        {
            "op": "add",
            "path": "/networkConfiguration/awsvpcConfiguration/securityGroups",
            "value": ["sg-d72e9599956a084f5"],
        },
    ],
)

Thanks Edward, it looks like this was what we have been searching for. we missed this snippet 🙂

Thanks for pointing this out Edward. We weren’t doing customisation of the security group as part of the ecs-task block. Will give it a try 👍

awesome, enjoy!

Hi, I have just stumbled across this myself. 🙂 It looks like the issue raised in https://github.com/PrefectHQ/prefect-aws/issues/298 is related to this. Needing to supply security group IDs at a task level is inconvenient; ideally, with the upcoming terraform provider (?), one should be able to pass the Security Group through to the work pool that is provisioned by the provider rather than specifying it at a task/flow level.

Hi @Craig de Gouveia, I initially thought it was a prefect task/flow setting as well but the ECSTask definition shared by Edward, specifically just the

task_customizations

piece is set in the ECS Task block. Once that’s set, any flow run using the block as infra will use your desired security group over the default security group. Hope you’re able to solve your issue! 🙂