https://prefect.io logo
Powered by
Title
a

Adam Roderick

03/08/2022, 3:59 PM
Have you seen this issue when launching jobs from the ECS Agent? I'm troubleshooting and not sure where to start with it. botocore doesn't like something about the parameters. Prefect version is 1.0
[2022-03-08 15:28:44,036] ERROR - agent | Exception encountered while deploying flow run a42f0aa4-2ac9-4d1e-a532-ffc4cfe5cb36
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 388, in _deploy_flow_run
deployment_info = self.deploy_flow(flow_run)
File "/usr/local/lib/python3.7/site-packages/prefect/agent/ecs/agent.py", line 322, in deploy_flow
resp = self.ecs_client.run_task(taskDefinition=taskdef_arn, **kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 692, in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 740, in _convert_to_request_dict
api_params, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 360, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter overrides.containerOverrides[0].environment[3].value, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
k

Kevin Kho

03/08/2022, 4:00 PM
I haven’t but we can take a look at the 
containerOverrides
part of your task definition?
Do you have an empty env variable? Maybe just give an empty string?
a

Adam Roderick

03/08/2022, 4:03 PM
Might be something to do with this, from the docs? It looks new to me 
The required dependencies for the ECS Agent aren't installed by default.
I don't have any containerOverrides
k

Kevin Kho

03/08/2022, 4:07 PM
Can I see your ECSRun then? You seem to have botocore though
a

Adam Roderick

03/08/2022, 4:07 PM
could be an empty env var, looking now. That would be in the agent or the flow run container?
flow.run_config = ECSRun(
        task_definition=task_definition,
        env={
            "DATATEER_ENV": environment,
            "AWS_DEFAULT_REGION": aws_region,
            "S3_PREFECT_RESULTS_BUCKET": prefect_results_bucket,
        },
        run_task_kwargs={
            "networkConfiguration": {
                "awsvpcConfiguration": {
                    "subnets": [
                        operations.get_parameter("/pipeline/subnet_1"),
                        operations.get_parameter("/pipeline/subnet_2"),
                    ],
                    "securityGroups": [operations.get_parameter("/pipeline/security_group")],
                    "assignPublicIp": "DISABLED",
                }
            }
        },
    )
k

Kevin Kho

03/08/2022, 4:08 PM
Can be either because agent settings and RunConfig settings get merged and then registered as an ECS Task together
I think it is an empty env var somewhere
👀 1
a

Adam Roderick

03/09/2022, 12:49 AM
@Kevin Kho yes, empty env var was the case. Thank you for taking a look with me
👍 1
10 Views
#prefect-communityJoin Slack
Powered by