https://prefect.io logo
Join Slack
Powered by
Hi guys, I seem to have a problem with getting an ...
# ask-community
p
Hi guys, I seem to have a problem with getting an ECSTask on EC2 to work. I got to the point when a simple hello world container ran without problems, but when I tried to execute the same flow again, it failed - several times. Meanwhile, the same ECSTask with the same parameters succeeds with Fargate. I am using 
g5.4xlarge
EC2s in an an ASG with an amazon ecs-optimized linux gpu AMI. Here is how I am calling ECSTask:
Copy code
ECSTask(
    command=["echo", "hello world"],
    image="<http://xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/my_repo:my_image|xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/my_repo:my_image>",
    launch_type="EC2",
    cluster="my-cluster",
    stream_output=True,
    execution_role_arn="arn:aws:iam::xxxxxxxxxxxx:role/prefect-ecs-task-role",
).run()
If anyone could help me figure out what I could try to get this working, I would be really grateful.
c
What doesn’t work? Do you have some errors that you are seeing, or what the failures are? Is it an ECS failure, or a Prefect Flow failure?
p
Hi @Christopher Boyd, thanks for your response - and sorry that I was not clear with my post. The error I am seeing is 
prefect_aws.ecs.EssentialContainerExited: Essential container in task exited
, so I am guessing that it is an ECS failure reported by Prefect. The task goes to a 
PENDING
status, but then abruptly stops. I also attached a screenshot of the full traceback. The issue is also weird, as on Fargate the same Docker image runs without problems, and one time on EC2 it also ran with the same config - since then I did not get it to work though. Thanks for your help!
c
Do you have cloudwatch for your ecs tasks? I might check there first as it might give more detail to why the container is having issues
These are some common reasons an ECS task goes pending before dying: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-tasks-stuck-pending-state/ That’s a pretty hefty sized box to run it on, but without knowing your dataset or what’s being loaded in, it’s hard to say if it’s a memory / cpu issue, etc.
p
The AWS logs show hello world getting printed out (which is then not streamed to the CLI). Would you have an idea why this task is getting marked as failed while on Fargate it does not?
c
log prints need to be marked in the flow / task with 
@flow(log_prints=True)
to send them to the prefect logger by default. Beyond that, I don’t see any logs or indication to point to a failure
p
That I am aware of, but why do I get an 
essential container in task exited
error from 
prefect_aws
if seemingly everything is running fine?
2 Views
Open in Slack
PreviousNext
Powered by