https://prefect.io logo
Join Slack
Powered by
I am trying to run a flow via an ECS agent. The ag...
# ask-community
k
I am trying to run a flow via an ECS agent. The agent is up and running, but when I trigger a flow I get the following error:
Copy code
Parameter validation failed:
Unknown parameter in input: "null", must be one of: capacityProviderStrategy, cluster, count, enableECSManagedTags, enableExecuteCommand, group, launchType, networkConfiguration, overrides, placementConstraints, placementStrategy, platformVersion, propagateTags, referenceId, startedBy, tags, taskDefinition
This is coming from the boto3 call triggered here: https://github.com/PrefectHQ/prefect/blob/6b59d989dec33aad8c62ea2476fee519c32f5c63/src/prefect/agent/ecs/agent.py#L320 My agent is being started like so (with prefecthq/prefect:0.14.13-python3.8, maybe this is old?):
Copy code
prefect agent ecs start --run-task-kwargs s3://$bucket/run_task_kwargs.yml -a <https://my-api:4200/graphql>
My run task kwargs yaml has 
cluster
, 
launchType
and 
networkConfiguration
. I also attempted to provide values in the run config via the UI, but it did not seem to change much (added CPU, memory, task and exec role). Any help is much appreciated!
k
Hey @Kyle McChesney, could you share the 
.yml
, just anonymize sensitive info?
k
Copy code
cluster: test
launchType: FARGATE
networkConfiguration:
  awsvpcConfiguration:
    subnets:
      - subnet-######
      - subnet-######
      - subnet-######
      - subnet-######
    securityGroups:
      - sg-#######
    assignPublicIp: DISABLED
k
I am not seeing anything apparent. Will ask the team members. It doesn’t seem to be a version issue as all of those CLI flags are available. The NULL makes me link the 
yml
is not being passed right. It would error out though. Could you check the output if you do this?
Copy code
import yaml
from prefect.utilities.filesystems import read_bytes_from_path

yaml.safe_load(read_bytes_from_path(run_task_kwargs_path))
Where 
run_task_kwargs_path
is 
s3://$bucket/run_task_kwargs.yml
k
I can certainly run that, I guess the only question is where should I run that? should I try to exec into the agent container? Also for the record, starting up the agent was originally failing without this yml file (it was getting permission errors calling DescribeVpcs here: https://github.com/PrefectHQ/prefect/blob/6b59d989dec33aad8c62ea2476fee519c32f5c63/src/prefect/agent/ecs/agent.py#L236. Providing the yaml file fixed it, so I think it’s being read at least initially
k
On the machine that the agent is running. Ah ok will look a bit more.
k
let me try and get ecs exec working
k
Could you also try running the Flow with debug logs so we can try to get more info?
Are you doing anything with the 
ECSRun
around 
run_task_kwargs
?
k
not that I know of, my flow code is pretty vanilla
Copy code
import math 

from prefect import Flow, task, Parameter
from prefect.executors import LocalDaskExecutor


@task
def data(range_amount):
    return range(range_amount)


@task
def compute_sqaure_root(item):
    return math.sqrt(item)


@task
def sum_roots(items):
    return sum(items)


with Flow('ecs_test', executor=LocalDaskExecutor()) as flow:
    range_amount = Parameter('range_amount', default=100)
    items = data(range_amount)
    roots = compute_sqaure_root.map(items)
    summed = sum_roots(roots)
ecs exec is going to take a bit to get setup
k
You didn’t use 
ECSRun()
? Could you try 
flow.run_config = ECSRun()
? Import with 
from prefect.run_configs.ecs import ECSRun
k
I just selected ECSRun via the UI when I submitted the flow
i will add that to the code
k
Oh I wonder if that is causing issues since the 
run_task_kwargs
is probably 
null
and getting misinterpreted as a string when converted from JS to Python.
k
setting it in the code seemed to work, I am getting a runTask permission denied (which is not super unexpected). Thanks a bunch @Kevin Kho!
k
Yeah that’s likely just the IAM Role. I will get a bug filed with the UI team
I am typing up the issue, just wanted to ask. You need to supply the run_task_kwargs through the CLI to reproduce right? This didn’t happen without the run_task_kwargs?
k
its hard to say for sure
basically I could not even start the ecs agent without the run_task_kwargs on the CLI, since the lack of it caused the agent to run that “infer network config” function, and thus triggered a describe vpcs call that failed for IAM reasons. I would need to revert those changes, and then get the “infer network config” stuff to work before being able to tell if this particular part worked without the kwargs
Additionally, our AWS account has a pretty involved network toplogy, so I am pretty dubious that the default network config that it tries to infer would work
k
Gotcha. Thanks for the info!
For posterity and you’re welcome to chime in too
✅ 1
3 Views
Open in Slack
PreviousNext
Powered by