https://prefect.io logo
Title
p

Peter Nagy

02/25/2023, 6:26 PM
Hi guys, I seem to have a problem with getting an ECSTask on EC2 to work. I got to the point when a simple hello world container ran without problems, but when I tried to execute the same flow again, it failed - several times. Meanwhile, the same ECSTask with the same parameters succeeds with Fargate. I am using
g5.4xlarge
EC2s in an an ASG with an amazon ecs-optimized linux gpu AMI. Here is how I am calling ECSTask:
ECSTask(
    command=["echo", "hello world"],
    image="<http://xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/my_repo:my_image|xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/my_repo:my_image>",
    launch_type="EC2",
    cluster="my-cluster",
    stream_output=True,
    execution_role_arn="arn:aws:iam::xxxxxxxxxxxx:role/prefect-ecs-task-role",
).run()
If anyone could help me figure out what I could try to get this working, I would be really grateful.
c

Christopher Boyd

02/27/2023, 8:08 PM
What doesn’t work? Do you have some errors that you are seeing, or what the failures are? Is it an ECS failure, or a Prefect Flow failure?
p

Peter Nagy

02/28/2023, 6:46 AM
Hi @Christopher Boyd, thanks for your response - and sorry that I was not clear with my post. The error I am seeing is
prefect_aws.ecs.EssentialContainerExited: Essential container in task exited
, so I am guessing that it is an ECS failure reported by Prefect. The task goes to a
PENDING
status, but then abruptly stops. I also attached a screenshot of the full traceback. The issue is also weird, as on Fargate the same Docker image runs without problems, and one time on EC2 it also ran with the same config - since then I did not get it to work though. Thanks for your help!
c

Christopher Boyd

02/28/2023, 1:29 PM
Do you have cloudwatch for your ecs tasks? I might check there first as it might give more detail to why the container is having issues
These are some common reasons an ECS task goes pending before dying: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-tasks-stuck-pending-state/ That’s a pretty hefty sized box to run it on, but without knowing your dataset or what’s being loaded in, it’s hard to say if it’s a memory / cpu issue, etc.
p

Peter Nagy

02/28/2023, 4:41 PM
The AWS logs show hello world getting printed out (which is then not streamed to the CLI). Would you have an idea why this task is getting marked as failed while on Fargate it does not?
c

Christopher Boyd

02/28/2023, 5:34 PM
log prints need to be marked in the flow / task with
@flow(log_prints=True)
to send them to the prefect logger by default. Beyond that, I don’t see any logs or indication to point to a failure
p

Peter Nagy

03/01/2023, 5:56 AM
That I am aware of, but why do I get an
essential container in task exited
error from
prefect_aws
if seemingly everything is running fine?