Hi all, I am trying to run a Prefect agent on ECS and have been following this tutorial. I used the below configurations (along with exporting the runner token, aws creds, etc to environment variables). However I am continuously receiving a

ValueError: Failed to infer default networkConfiguration, please explicitly configure using --run-task-kwargs

error. I have tried all teh different combinations I can think of and been through the docs (both prefect and ECS) but cant find a solution that works. Has anyone else been able to solve this?

export networkConfiguration="{'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['<EC2 instance subnet>'], 'securityGroups': ['<EC2 instance security group>']}}"
prefect agent ecs start --token $RUNNER_TOKEN_FARGATE_AGENT \
 --task-role-arn=arn:aws:iam::<AWS ACCOUNT ID>:role/ECSTaskS3Role \
 --log-level INFO --label prod --label s3-flow-storage \
 --name prefect-prod

Hey @Alex Welch From the docs it seems that if you want to provide

networkConfiguration

you have to provide it in a YAML file. Something like this

prefect agent ecs start --run-task-kwargs /path/to/options.yaml

i noticed that, but i didnt see an example of what the yaml needed to look like

This might helpm you https://github.com/PrefectHQ/prefect/blob/18fcb5dfce7ba3ecf9db3c6a6f0efa21d9403d88/tests/agent/test_ecs_agent.py#L244 Maybe someone from Prefect Team can point you to the

kwargs.yaml

, I’m not able to find it 😅

thanks, ill give it a try

Would you mind sharing the YAML content? I’m not sure this is related to newtorkConfiguration, though

Can you share your logs? Maybe setting logs to debug (if not already on debug level) would help in troubleshooting

I suggest you restart the agent with the following options:

--show-flow-logs --verbose

This way you can see the flow run logs in the agent console

Error: no such option: --show-flow-logs

Try setting log level to debug

so i have log level set to debug\

😅

well thats amazing

Well, we don’t know how we did it….but we did it!

Failed to load and execute Flow's environment: UnpicklingError("invalid load key, '{'.")

This error indicates that the image used to run the flow was older than prefect 0.14.3, while the version of prefect used to register the flow was >= 0.14.3. In 0.14.3 we changed the serialization format to include some metadata like version numbers, so in the future if this happens you'll get a nice error message. We've never made guarantees that running a flow using a version of prefect older than the version the flow was registered with would work, so this wasn't really a breaking change, but the error that happens if you've accidentally been doing this is a bit cryptic.

Hey @Jim Crist-Harif 🙂 Thanks a lot for the explanation of the error, very useful!

@Jim Crist-Harif thank you for the explanation. I might recommend either adding some verbiage to the pickling error message that recommends checking your prefect versions or putting something in a trouble shooting page in the documents

i'm also just starting to play around with the ECS agent and am running into some of the same difficulties 😎

I think that is a great point

I confess that I'm not the best at AWS stuff, so there might be a better way. We tried to make this as simple as possible - with zero configuration specified, the ECS agent attempts to infer the proper VPC configuration for you. For most deployments you'll need

assignPublicIp

configured, since otherwise you won't have public internet access (so pulling an image from dockerhub wouldn't work). Users that would want to override this probably also know what to do since this requires extra AWS effort on their end.

Users that would want to override this probably also know what to do since this requires extra AWS effort on their end

In my case, all the networking had been handled by another team and I didn’t have any control over it. To make things more complicated was that I did not have access to the

default vpc

and so I had to go the route i ended up at. I think this may be true for many analytics-engineer type roles. The infra/network teams would handle all of that stuff to ensure security.

hi all, this was very helpful - thank you! i had the same problem where i encountered this trying to run on a cluster in my company's aws account (though no problems in my own account), so likely it was specific to the networking setup by our infra/devops team, where similar to @Alex Welch i have no control over.