https://prefect.io logo
s

Sarhan

07/11/2023, 12:27 AM
Hi all. I’ve been facing an issue where my scheduled flows on Prefect cloud crash with no logs and this state message:
Flow run could not be submitted to infrastructure
. My Prefect is using ECS as infra basically following this

Prefect tutorial

(I’m new to using Terraform & ECS btw). We’ve been using this set up since the beginning of the year but the crashes started happening out of nowhere ~16 hours ago. Any advice welcome.
Looking into the ECS task, I’m seeing this message
w

Will Raphaelson

07/11/2023, 1:33 AM
Thanks Sarhan, I’ll look into this.
👍 1
s

Sarhan

07/17/2023, 8:57 AM
Hey @Will Raphaelson and the wider community as well (any help welcome), Adding more context to this issue, it started because of us making changes to our network ACLs for the default VPC and that impacted the default security group’s outbound rules. What we’re trying to do now is get ECS to use the security group that’s generated when all other resources are created (when terraform is applied based on this Prefect Recipe). The agent task that’s started when the ECS cluster is created uses this security group without issue but any new tasks created from a flow-run triggered from a deployment in prefect cloud uses the default security group which we don’t want to have outbound rules for and so it’s unable to pull the docker container from DockerHub. Is there a way to amend the security group that gets associated to the new tasks? What variables could be set with terraform for this? CC: @Aifred Ong
👀 1
e

Edward Park

07/17/2023, 4:32 PM
hey Sarhan - i’m curious to see how you’ve configured your
ECSTask
for the job itself in the flow code (so not the agent), but have you taken a look at the
prefect-aws
docs, specifically around running an ECS task with a specific VPC and/or security group ID? https://prefecthq.github.io/prefect-aws/ecs/#prefect_aws.ecs.ECSTask-attributes
Copy code
ECSTask(
    command=["echo", "hello world"],
    vpc_id="vpc-01abcdf123456789a",
    task_customizations=[
        {
            "op": "add",
            "path": "/networkConfiguration/awsvpcConfiguration/securityGroups",
            "value": ["sg-d72e9599956a084f5"],
        },
    ],
)
👀 1
a

Aifred Ong

07/18/2023, 1:01 AM
Thanks Edward, it looks like this was what we have been searching for. we missed this snippet 🙂
s

Sarhan

07/18/2023, 1:19 AM
Thanks for pointing this out Edward. We weren’t doing customisation of the security group as part of the ecs-task block. Will give it a try 👍
We’ve managed to get it working again. Thanks again Edward
e

Edward Park

07/18/2023, 4:16 PM
awesome, enjoy!
c

Craig de Gouveia

07/28/2023, 10:43 AM
Hi, I have just stumbled across this myself. 🙂 It looks like the issue raised in https://github.com/PrefectHQ/prefect-aws/issues/298 is related to this. Needing to supply security group IDs at a task level is inconvenient; ideally, with the upcoming terraform provider (?), one should be able to pass the Security Group through to the work pool that is provisioned by the provider rather than specifying it at a task/flow level.
s

Sarhan

07/30/2023, 2:24 PM
Hi @Craig de Gouveia, I initially thought it was a prefect task/flow setting as well but the ECSTask definition shared by Edward, specifically just the
task_customizations
piece is set in the ECS Task block. Once that’s set, any flow run using the block as infra will use your desired security group over the default security group. Hope you’re able to solve your issue! 🙂
❤️ 1
3 Views