Hey wave I am new to Prefect Cloud and have hit a task defin Prefect Community #prefect-community

Hey :wave: , I am new to Prefect Cloud and have h...

Kieran

01/18/2021, 2:40 PM

Hey 👋 , I am new to Prefect Cloud and have hit a task definition snag which I can't seem to solve. We have an ECS Agent deployed on EC2 with the correct API tokens which shows up in our Cloud UI. We have a ECS cluster which we declare when launching the agent but leave the launch type as default (Fargate). Our example Flow is below:

Copy code

default_client = docker.from_env()
FLOW_NAME = "hello-flow"
flow_schedule = CronSchedule("0 8 * * *")
flow_storage = Docker(
    base_url=default_client.api.base_url,
    tls_config=docker.TLSConfig(default_client.api.cert),
    registry_url="<http://_________.dkr.ecr.eu-west-2.amazonaws.com/xxxxx/prefect|_________.dkr.ecr.eu-west-2.amazonaws.com/xxxxx/prefect>"
)
flow_run_config = ECSRun(
    cpu="512",
    memory="512",
    run_task_kwargs={"requiresCompatibilities": ["FARGATE"], "compatibilities": ["FARGATE"]}
)

with Flow(
    name=FLOW_NAME,
    schedule=flow_schedule,
    storage=flow_storage,
    run_config=flow_run_config
    ) as flow:
    say_hello()

if is_serializable(flow):
    flow.register(project_name="Test", registry_url=flow_storage)
else:
    raise TypeError("Flow did not serialise.")

We are getting the following error from our task logs:

Copy code

An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.

In an attempt to resolve this issue I tried adding

run_task_kwargs

as above but with no luck. Does anyone have any pointers? (I can see from the ECS task definition panel that the
Compatibilities
is set to EC2 and
Requires compatibilities
is blank and from this thread that could be the cause...)

Kieran

01/18/2021, 2:56 PM

Also, deploying the agent as so returns the same error:

Copy code

prefect agent ecs start --cluster prefect-cluster --launch-type EC2

josh

01/19/2021, 2:52 PM

Hmm @Kieran are you deploying to the

default

cluster or do you have another one created? I feel like I’ve seen this when the cluster the agent is trying to deploy to isn’t permissioned properly to use Fargate 🤔

Kieran

01/19/2021, 3:02 PM

Hey @josh we have provisioned a cluster for prefect (called prefect-cluster). Our best guess at the minute is that permissions (IAM roles) are off slightly. What is confusing is that memory and CPU limits can be set within ESCRun() and are correctly bring provided to the task definition but we seem unable to pass compatibilities kwargs to declare the task as a FARGATE launch type. Other threads in here seem to suggest that the error is a slight red herring and we need to declare the task and execution roles in a custom task definition yaml file to overcome this issue... (Or possibly/likely I'm just confused with the deployment details)

josh

01/19/2021, 3:05 PM

Yeah I think it could be IAM related as well, I saw someone internal to Prefect run into an issue like this last week but everything worked fine when they went back to using the default cluster. As for the compatibilities kwargs, yes currently you would need to define a custom task definition for the ECSRun. In implementing this run config we chose to only expose the bare minimum and are adding new ones as user requests come in! If you think this is something that would be helpful for users to expose as a direct kwarg feel free to open an issue for the suggestion 🙂

🙌 1

Sagun Garg

01/21/2021, 7:54 AM

@josh @Kieran apologies for jumping in between your thread, I am also getting the same challenge in creating ECSRun agent. I am getting an error

Copy code

RROR - agent | Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`

ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`

--run-task-kwargs

I have already implemented a task_definition.yaml as below as suggested by another community member in another thread but still no relief

Copy code

networkMode: awsvpc
cpu: 1024
memory: 2048
containerDefinitions:
  - name: flow
taskRoleArn: <ecs_task_role_arn>
executionRoleArn: <insert_ecsTaskExecutionRole_arn>

Kieran

01/25/2021, 9:46 PM

For anyone who reads this thread with the same issue as me. A customer task_definition file which declares FARGATE as the required compatible launch type has fixed my issue.

✅ 2

Jay Sundaram

03/13/2021, 9:11 PM

Hi @Kieran, Can you share the minimum contents of the task_definition file? Thank you in advance.

Kieran

03/16/2021, 7:06 PM

Hi @Jay Sundaram We use the folloiwing:

Copy code

networkMode: awsvpc
cpu: 256
memory: 512
containerDefinitions:
  - name: flow
requiresCompatibilities:
  - FARGATE
executionRoleArn: arn:aws:iam::xxxxxxxxxx:role/tf-iam-role-prefect-ecs-tasks <--- change to whatever you need

🙏 1

Open in Slack

Previous Next