Hi guys, I seem to have a problem with getting an ECSTask on EC2 to work. I got to the point when a simple hello world container ran without problems, but when I tried to execute the same flow again, it failed - several times. Meanwhile, the same ECSTask with the same parameters succeeds with Fargate. I am using

g5.4xlarge

EC2s in an an ASG with an amazon ecs-optimized linux gpu AMI. Here is how I am calling ECSTask:

ECSTask(
    command=["echo", "hello world"],
    image="<http://xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/my_repo:my_image|xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/my_repo:my_image>",
    launch_type="EC2",
    cluster="my-cluster",
    stream_output=True,
    execution_role_arn="arn:aws:iam::xxxxxxxxxxxx:role/prefect-ecs-task-role",
).run()

If anyone could help me figure out what I could try to get this working, I would be really grateful.

What doesn’t work? Do you have some errors that you are seeing, or what the failures are? Is it an ECS failure, or a Prefect Flow failure?

Hi @Christopher Boyd, thanks for your response - and sorry that I was not clear with my post. The error I am seeing is

prefect_aws.ecs.EssentialContainerExited: Essential container in task exited

, so I am guessing that it is an ECS failure reported by Prefect. The task goes to a

PENDING

status, but then abruptly stops. I also attached a screenshot of the full traceback. The issue is also weird, as on Fargate the same Docker image runs without problems, and one time on EC2 it also ran with the same config - since then I did not get it to work though. Thanks for your help!

Do you have cloudwatch for your ecs tasks? I might check there first as it might give more detail to why the container is having issues

The AWS logs show hello world getting printed out (which is then not streamed to the CLI). Would you have an idea why this task is getting marked as failed while on Fargate it does not?

log prints need to be marked in the flow / task with

@flow(log_prints=True)

to send them to the prefect logger by default. Beyond that, I don’t see any logs or indication to point to a failure

That I am aware of, but why do I get an

essential container in task exited

error from

prefect_aws

if seemingly everything is running fine?