Alex Welch
02/16/2021, 2:17 PMValueError: Failed to infer default networkConfiguration, please explicitly configure using --run-task-kwargs
error. I have tried all teh different combinations I can think of and been through the docs (both prefect and ECS) but cant find a solution that works. Has anyone else been able to solve this?
export networkConfiguration="{'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['<EC2 instance subnet>'], 'securityGroups': ['<EC2 instance security group>']}}"
prefect agent ecs start --token $RUNNER_TOKEN_FARGATE_AGENT \
--task-role-arn=arn:aws:iam::<AWS ACCOUNT ID>:role/ECSTaskS3Role \
--log-level INFO --label prod --label s3-flow-storage \
--name prefect-prod
--log-level DEBUG
it looks like nothing is being passed to the environment variables. I’m thinking this is the problem?networkConfiguration
as an --env
I get Error: Got unexpected extra arguments
ale
02/16/2021, 2:30 PMnetworkConfiguration
you have to provide it in a YAML file.
Something like this
prefect agent ecs start --run-task-kwargs /path/to/options.yaml
prefect agent ecs start --run-task-kwargs <s3://bucket/path/to/options.yaml>
If you want to provide the YAML file from S3Alex Welch
02/16/2021, 3:18 PMale
02/16/2021, 3:25 PMkwargs.yaml
, I’m not able to find it 😅Alex Welch
02/16/2021, 3:38 PMFailed to load and execute Flow's environment: UnpicklingError("invalid load key, '{'.")
when i run my flowale
02/16/2021, 3:45 PMAlex Welch
02/16/2021, 3:48 PMale
02/16/2021, 3:58 PMAlex Welch
02/16/2021, 3:59 PMale
02/16/2021, 4:02 PM--show-flow-logs --verbose
This way you can see the flow run logs in the agent consoleAlex Welch
02/16/2021, 4:04 PMale
02/16/2021, 4:06 PMprefect.utilities.debug.is_serializable
on your flow
https://docs.prefect.io/api/latest/utilities/debug.html#functionsAlex Welch
02/16/2021, 4:09 PMale
02/16/2021, 4:13 PMAlex Welch
02/16/2021, 4:13 PMale
02/16/2021, 4:14 PMJim Crist-Harif
02/16/2021, 5:00 PMThis error indicates that the image used to run the flow was older than prefect 0.14.3, while the version of prefect used to register the flow was >= 0.14.3. In 0.14.3 we changed the serialization format to include some metadata like version numbers, so in the future if this happens you'll get a nice error message. We've never made guarantees that running a flow using a version of prefect older than the version the flow was registered with would work, so this wasn't really a breaking change, but the error that happens if you've accidentally been doing this is a bit cryptic.Failed to load and execute Flow's environment: UnpicklingError("invalid load key, '{'.")
networkConfiguration
thing, specifying that via a yaml file provided to --run-task-kwargs
is the correct way to fix it. Is there something we could do to make this clearer? The error message you got specifically calls out using --run-task-kwargs
to fix it, and the behavior of this flag is documented in both the CLI help and the docs: https://docs.prefect.io/orchestration/agents/ecs.html#custom-runtime-options. Happy to make any docs updates needed to help future users who may run into this issue.ale
02/17/2021, 8:46 AMAlex Welch
02/18/2021, 4:27 PMnetworkConfiguration
I would recommend some documents around how to best use and structure --run-task-kwargs
options.yml
could or should look like? and again - I’ve seen the networkConfiguration
issue mentioned in other threads here. Maybe it makes sense to dump it in a Troubleshooting section of the documentation?Sean Talia
02/25/2021, 7:07 PMassignPublicIp= 'ENABLED'
on the vpc configuration?assignPublicIp
key set to ENABLED?Alex Welch
02/25/2021, 10:02 PMJim Crist-Harif
02/25/2021, 11:18 PMassignPublicIp
configured, since otherwise you won't have public internet access (so pulling an image from dockerhub wouldn't work). Users that would want to override this probably also know what to do since this requires extra AWS effort on their end.Alex Welch
02/26/2021, 4:32 AMUsers that would want to override this probably also know what to do since this requires extra AWS effort on their end
In my case, all the networking had been handled by another team and I didn’t have any control over it. To make things more complicated was that I did not have access to the default vpc
and so I had to go the route i ended up at. I think this may be true for many analytics-engineer type roles. The infra/network teams would handle all of that stuff to ensure security.felice
02/27/2021, 2:17 AM