https://prefect.io
Join Slack
Did some digging. Does prefect override my selected VPC subnets when running the task? <https://gith...
# prefect-aws
e
Did some digging. Does prefect override my selected VPC subnets when running the task? https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/ecs.py#L924
✅ 1
Cc @Zanie is that correct?
z
e
I have a custom task definition arn being used. Do I still need to define those?
z
I think most of the network config needs to happen at runtime not task definition time?
e
You are correct. But that’s already defined in the running service, no?
I was under the impression that if I set a static task definition arn it won’t run another task? Which does seem to happen. But for some reason when I run a flow, it can’t read secrets. The agent does start up correctly and is able to read tasks. Or am I thinking about this all wrong?
z
Like the agent is running in a task too?
I’m not really sure what you’re asking
e
Same 😂
z
Networking is the worst 😄
e
I used this for my starter https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent And added datadog as the logging mechanism (with the api key as a secret). When I run the agent it works fine saying that it’s waiting for flow runs. But when a flow actually runs it can’t access the secret
z
Perhaps @Emil Christensen knows what’s up
🙌 1
e
Hey @Ethan Veres 👋 The recipe specifically allows the agent’s execution role to access only the API key secret. You may need to attach a permission to allow reading your new secret to the role that your task uses.
e
Thank you @Zanie @Emil Christensen! What I failed to realize is that the ECS agent spawns additional ECS tasks, with configuration from the block!
🙌 1
✅ 2
j
Are you looking for https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/ecs.py#L62-L74
Thank you!! I just changed the VPC to our company's main VPC after following dataflowops template and specifying the subnet here allowed my container to access ECR again! Solved:
Copy code
prefect_aws.ecs.TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "<https://api.ecr>.<region>.<http://amazonaws.com/|amazonaws.com/>"
37 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by