I’m new to Prefect and have been attempting to run a flow using ECS. Unfortunately, my containers keep stopping immediately and the only info I see says 

Essential container in task exited

 . I’ve been trying to debug this for a bit and am not sure how to proceed. Any advice?

ECS makes it a pain to debug. If the container did indeed start, there should be task logs you can check. If there are no logs, from what I’ve seen it’s IAM roles a lot of the time

Thanks @Kevin Kho! It’s just a “hello world” type example so permissions required should be minimal. Is it the

execution_role_arn

that I should make sure has the right permissions or something else?

Yeah and if you have to pull an image from ECR. Actually there is this new updated ECS walkthrough that may help

Beautiful! I have a read a lot of docs and medium posts but not this one yet. Let me give it a shot

Important part is the detailed permissions

I gave this a shot and am seeing logs now, but other errors

That sounds good

OK so I’m still struggling with this and seeing the

Essential container in task exited

. Here’s the flow I’m trying to run:

import os
import time

import prefect
from prefect.storage import Docker
from prefect import task, Flow, Parameter
from prefect.run_configs import ECSRun

@task
def say_hello(name):
    time.sleep(10)

    greeting = os.environ.get('GREETING')
    logger = prefect.context.get('logger')
    <http://logger.info|logger.info>(f'{greeting}, {name}!')

with Flow('hello-flow-ecs') as flow:
    people = Parameter('people', default=['Arthur', 'Ford', 'Marvin'])
    say_hello.map(people)

flow.storage = Docker(
    registry_url='<http://134777600073.dkr.ecr.us-east-1.amazonaws.com/prefect/|134777600073.dkr.ecr.us-east-1.amazonaws.com/prefect/>', 
    image_name='hello-flow-ecs', 
    image_tag='latest',
    dockerfile='Dockerfile'
)

flow.run_config = ECSRun(
    cpu='.25 vCPU',
    memory='0.5 GB',
    task_role_arn='arn:aws:iam::134777600073:role/prefectTaskRole',
    execution_role_arn='arn:aws:iam::134777600073:role/ecsTaskExecutionRole',
    run_task_kwargs=dict(cluster='prefectEcsCluster', launchType='FARGATE'),
    env={'GREETING': 'Hello'},
    image='prefect/hello-flow-ecs:latest'
)

flow.register(
    project_name='Prefect Tutorial',
    labels=['prod'],
    add_default_labels=False
)

Let me know if you have any other ideas re: debugging or if anything looks obviously wrong!

I believe the logic looks fine. Is that image for ECSRun right? I am not 100% but does it have to be the full registry and image? Because with that syntax, I think it will be pulled from DockerHub

Hm, tried changing to that and still seeing the same error…

Hard to say cuz ECS doesn’t give good logs either when the container fails to start. I think I would maybe try using the base prefect image

prefecthq/prefect

and then storing the flow in something easy to access like Github. Then you could test with this ECSRun and Github Storage that are definitely publicly pullable to verify it’s not something else?

Thanks again for bearing with me — can you show me how I should modify my code to do that? I’m at the edges of my depth. Current version of the script is:

import os
import time

import prefect
from prefect.storage import Docker
from prefect import task, Flow, Parameter
from prefect.run_configs import ECSRun

@task
def say_hello(name):
    time.sleep(10)

    greeting = os.environ.get('GREETING')
    logger = prefect.context.get('logger')
    <http://logger.info|logger.info>(f'{greeting}, {name}!')

with Flow('hello-flow-ecs') as flow:
    people = Parameter('people', default=['Arthur', 'Ford', 'Marvin'])
    say_hello.map(people)

flow.storage = Docker(
    registry_url='<http://134777600073.dkr.ecr.us-east-1.amazonaws.com/prefect/|134777600073.dkr.ecr.us-east-1.amazonaws.com/prefect/>', 
    image_name='hello-flow-ecs', 
    image_tag='latest',
    dockerfile='Dockerfile'
)

flow.run_config = ECSRun(
    cpu='.25 vCPU',
    memory='0.5 GB',
    task_role_arn='arn:aws:iam::134777600073:role/prefectTaskRole',
    execution_role_arn='arn:aws:iam::134777600073:role/prefectECSAgentTaskExecutionRole',
    run_task_kwargs=dict(cluster='prefectEcsCluster', launchType='FARGATE'),
    env={'GREETING': 'Hello'},
    image='<http://134777600073.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-flow-ecs:latest|134777600073.dkr.ecr.us-east-1.amazonaws.com/prefect/hello-flow-ecs:latest>'
)

flow.register(
    project_name='Prefect Tutorial',
    labels=['prod'],
    add_default_labels=False
)

The image is the easier part with

image="prefecthq/prefect:latest-python3.x"

where x is your major python version

OK cool

Github storage will look like this: https://github.com/kvnkho/demos/blob/main/prefect/git_storage.py

OK cool, will try this

You can give me a link to the flow once you have it on Github too so I can check it. You might need Prefect installed with the Github extra

pip install prefect[github]

How’s this look? https://github.com/kingfink/prefect-demo/blob/master/hello_flow_ecs_github.py

Looks good to me I think

I had followed these instructions https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df

Oh that’s good. That’s progress haha.

🤦‍♂️ haha yes it is! Thank you for bearing with me

Of course! ECS can be a pain

Progress!!!

Task 'say_hello': Exception encountered during task execution!

Just to make sure it’s not auth, can you try stopping the agent on the ECS service (or leave it but use new labels for this test). Start your agent from the command line on your local:

prefect agent ecs start --cluster arn:aws:ecs:us-east-2:12345678:cluster/test-cluster --label ecs_test_c --label ecs_test_d

But before you do that, try running this to authenticate:

aws ecr get-login-password | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$<http://AWS_REGION.amazonaws.com|AWS_REGION.amazonaws.com>

And then run the flow with the ecs agent running on your local machine

How do I do the last part?

Trigger from the UI as how you ran the flow that worked, we just want it to be picked up by the agent running on your machine that we know for sure is logged in to pull from ECR

👍 so I should run the flow I ran from GitHub w/ those labels? Or the original flow?

Original. This time we’ll try pulling from ECR

OK cool

Actually you must be logged in to be able to push there…so this test may not be useful. Don’t worry about that

get-login-password

. But let’s still test running the flow with an agent on your machine that we know how sure is authenticated

Sounds good

Yeah to your cluster name that you made in ECS

OK

And your arn and region. The same one you had earlier

Cool, got those two!

Yeah like that and then register your Flows with those labels so it doesnt go to the old agent

Yup got that. Giving it a shot

With logs?

No logs in Prefect

Task logs on ECS

Not there either

for posterity: Kevin resolved that via DM. The solution was to use base image that contains prefect (specifically: changing the container image to the official Prefect image).

Hi everyone, I am running in a similar issue. I am using Prefect to run containers on ECS fargate using legacy java-based pipelines. My jobs are shipped into containers, they are running fine on ECS but prefect does not seem to catch when the containers are stopping (the Java pipelines are running correctly). When I look down to the Fargate task the container stopped after the job is done and the only information that I have is:

Essential container in task exited

Any idea how to solve this?

Hi @Benjamin Bonhomme, even though you get the same error, I think your use case seems a bit different. It’s hard for me to give any recommendations right away, since it would require looking into your infrastructure. If you’re interested, I could put you in touch with a team who specializes in infrastructure issues like this and provides more in-depth guidance. Otherwise, could you share a bit more about what storage do you use, how do you build your flows?

So with Tim we added more logs to ECS, and that gave more info that the image being used was not cross platform. It was being built on a M1 Mac and not working on Linux. Not sure if this is your issue because that error message is a vague catch all though. If you could add logs with something like this:

task_definition = yaml.safe_load(
      """
      cpu: 1024
      memory: 2048
      containerDefinitions:
      - name: flow
        logConfiguration:
          logDriver: awslogs
          options:
            awslogs-group: /ecs/prefect-agent
            awslogs-region: us-east-1
            awslogs-stream-prefix: ecs
            awslogs-create-group: 'true'
      """
  )
ECSRun( ..., task_definition=task_definition)

we might be able to see if it’s low hanging fruit. Otherwise this might be networking/IAM related which will be a lot more involved

Hi guys, Thank you for your response. @Anna Geller, the infrastructure looks good to me in a way that it runs fine, Prefect just doesn't catch when the container exits. Having a chat will surely help 🙂 @Kevin Kho, I am not sure if you are referring to my message but the image build is done on a Mac Intel proc.

I think if you add the logs like above to the task definition, maybe we can get more insight. I was more of explaining how we addressed Tim’s issue

@Benjamin Bonhomme if your infrastructure and agent setup look good, then we can probably resolve it together if you would give us a bit more information. Could you set the logging level to debug to get more info + show us how did you find out that the container exists on ECS and Prefect doesn’t catch that?

ECSRun(..., env="PREFECT__LOGGING__LEVEL":"DEBUG")

Alright, thank you guys for your help by the way, I wan't sure I should expose the case/logs here. Prefect Agent runs on ECS Fargate, log level: DEBUG, logs listening to the incoming flows:

[2021-10-18 15:42:29,173] DEBUG - agent | Found 1 ready flow run(s): {'afb8c55f-c1fc-4cbc-926f-2d003bd538cc'}

[2021-10-18 15:42:29,174] DEBUG - agent | Retrieving metadata for 1 flow run(s)...

[2021-10-18 15:42:29,333] DEBUG - agent | Submitting flow run afb8c55f-c1fc-4cbc-926f-2d003bd538cc for deployment...

[2021-10-18 15:42:29,333] DEBUG - agent | Sleeping flow run poller for 0.25 seconds...

[2021-10-18 15:42:29,334] INFO - agent | Deploying flow run afb8c55f-c1fc-4cbc-926f-2d003bd538cc to execution environment...

[2021-10-18 15:42:29,335] DEBUG - agent | Updating flow run afb8c55f-c1fc-4cbc-926f-2d003bd538cc state from Scheduled -> Submitted...

[2021-10-18 15:42:29,540] DEBUG - agent | Registering new task definition for flow f35b13ff-12db-48d5-901d-fd9de5365e8e

[2021-10-18 15:42:29,585] DEBUG - agent | Querying for ready flow runs...

[2021-10-18 15:42:29,638] DEBUG - agent | Registered task definition arn:aws:ecs:eu-west-1:XXX:task-definition/prefect-ecs-slack-notify:23 for flow f35b13ff-12db-48d5-901d-fd9de5365e8e

[2021-10-18 15:42:29,998] DEBUG - agent | No ready flow runs found.

[2021-10-18 15:42:29,998] DEBUG - agent | Sleeping flow run poller for 0.5 seconds...

[2021-10-18 15:42:30,289] DEBUG - agent | Deregistering task definition arn:aws:ecs:eu-west-1:XXX:task-definition/prefect-ecs-slack-notify:23

[2021-10-18 15:42:30,344] DEBUG - agent | Started task 'arn:aws:ecs:eu-west-1:XXX:task/etl-jobs-ecs-cluster/YYY' for flow run 'afb8c55f-c1fc-4cbc-926f-2d003bd538cc'

[2021-10-18 15:42:30,345] INFO - agent | Completed deployment of flow run afb8c55f-c1fc-4cbc-926f-2d003bd538cc

[2021-10-18 15:42:30,498] DEBUG - agent | Querying for ready flow runs...

[2021-10-18 15:42:30,650] DEBUG - agent | No ready flow runs found.

[2021-10-18 15:42:30,650] DEBUG - agent | Sleeping flow run poller for 1.0 seconds...

Flow logs (log level on DEBUG too): Application logs of my Java application (Slack Webhook saying Hi, I receive the message on the channel without problem) I just replaced my AWS account_id with XXX and the container_id with YYY. @Anna Geller, I can check the task on AWS with its ID logged by prefect, the container runs with the entrypoint I defined (the one from the PREFECT__CONTEXT__IMAGE defined in the task), does its job (in this case notifying me on slack), exits and then nothing happens as Prefect doesn't catch anything. Lazarus will try to run the task after a while. Not sure it brings you a lot of light. If anything comes through your mind or you need some more information, I am here! 🙂

Are there logs on the AWS ECS task? (separate from Prefect logs)

In the logs from the AWS ECS task there is only the logs I print in the console from my Java pipeline.

@Benjamin Bonhomme can you share with us your Flow code? I think there must be something wrong with the Flow definition. I looked into how AWS ECS handles that and "ECS Essential container in task exited" basically means that the task finished, without knowing yet, whether this was successful or not. Since you run your Java application in your container, there could be some configuration needed so that we can infer in Prefect whether your task was successful or not. And we can't do that unless we see your Flow structure 🙂

@Anna Geller, alright it makes sense. The flow itself is structured in the most simplistic way to run a container:

from prefect.run_configs import ECSRun
from prefect import Flow
import yaml


FLOW_NAME = "ecs_slack_notify"
IMAGE_NAME = "<http://XXX.dkr.ecr.eu-west-1.amazonaws.com/slack_notify_from_prefect:0.1|XXX.dkr.ecr.eu-west-1.amazonaws.com/slack_notify_from_prefect:0.1>"
EXECUTION_ROLE_ARN = "arn:aws:iam::XXX:role/PREFECT_ECS_AGENT_TASK_EXECUTION_ROLE"
TASK_ROLE_ARN = "arn:aws:iam::XXX:role/PREFECT_TASK_ROLE"

TASK_DEFINITION = yaml.safe_load(
"""
cpu: 256
memory: 512
containerDefinitions:
- name: flow
  logConfiguration:
    logDriver: awslogs
    options:
      awslogs-group: /ecs/prefect-agent
      awslogs-region: eu-west-1
      awslogs-stream-prefix: ecs
      awslogs-create-group: 'true'
networkMode: awsvpc
"""
  )

RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'etl-jobs-ecs-cluster'},
                    labels=["prod"],
                    task_role_arn=TASK_ROLE_ARN,
                    execution_role_arn=EXECUTION_ROLE_ARN,
                    image=IMAGE_NAME,
                    memory=512,
                    cpu=256,
                    env={"PREFECT__LOGGING__LEVEL":"DEBUG"},
                    task_definition=TASK_DEFINITION)

flow = Flow(FLOW_NAME, run_config=RUN_CONFIG)

if __name__ == "__main__":
    flow.register("etl-jobs")

@Benjamin Bonhomme I see what is the problem. Your flow has no tasks. Is there some way you could trigger your Java pipeline from a ShellTask? This way, you could add this task to the Flow so that Prefect can govern its execution.

Thank you @Anna Geller. As the Java pipelines are packaged in docker containers, do you have in mind what kind of shell command I could pass to trigger the container execution within an ECS fargate cluster? As I understand the shell command will not be executed in the context of the container running the pipelines

@Benjamin Bonhomme another solution that comes to my mind would be to have this Java application as one ECS task, and then you could have a Flow that triggers this boto3 run_task. But not sure how beneficial Prefect would be for you in this case. Let me ask the team for other ideas.

@Anna Geller, so I tried putting the same command used as the entrypoint of the container. The thing is that the Flow is defined with the run_config including the container I want to run, once the Flow is instanciated, a container is created and runs the entrypoint, the ShellTask is supposed to start after but since there is no signal back to prefect once the container starts the ShellTask never runs eventually. The question about in which context the Shell runs remains.

Hey @Benjamin Bonhomme, so you can do:

RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'etl-jobs-ecs-cluster'},
                    labels=["prod"],
                    task_role_arn=TASK_ROLE_ARN,
                    execution_role_arn=EXECUTION_ROLE_ARN,
                    image=IMAGE_NAME,
                    memory=512,
                    cpu=256,
                    env={"PREFECT__LOGGING__LEVEL":"DEBUG"},
                    task_definition=TASK_DEFINITION)

with Flow(FLOW_NAME, run_config=RUN_CONFIG) as flow:
     task(lambda x: True)(0)

just to put a task there because the issue is that there is no task right now inside the Flow. I think the Docker entrypoint will still run here so you will achieve running it, but if you rely on the ENTRYPOINT to execute it, it’s not a Prefect task and you won’t get errors or logging around the Java program execution.

@Kevin Kho, @Anna Geller, thank you for the tips, it is actually very helpful! I removed the entrypoint from my images and I am trying to run the same script through the a ShellTask. An exit code 127 is returned on execution of the command I can't figure out what is the cause of it since the script is executable (chmod +x in my Dockerfile) and the command I pass in is as such: "/bin/sh -c /abs_path/script.sh" with the absolute path

@Benjamin Bonhomme got it. I have great news. I sat down this morning and drafted an example ECSRunTask that I may contribute to Prefect task library once this is more polished. But it’s working already for a very simple use case: https://gist.github.com/4637b4098071ff5af87b19d83cebd8d4 Can you try running your flow this way? • you would need to adjust the ECSRunTask call to include your subnets and other resources • you won’t see your Java application logs in Prefect Cloud, but at least this should work more reliably because it’s just triggering an ECS task definition to run

Hi guys, I have been trying to get my head around using boto to orchestrate the tasks. I don't find it a really convenient workflow since the task needs to be defined on AWS in the first place it doesn't really fit my needs (quite a long process to replicate to all our jobs). I really like the ShellTask option since it makes prefect shining and fit a sane developement workflow. I am still stuck with the non 0 exit from the command.

To get the

ShellTask

to work, you would need to build your own container that has Java I think. Could you elaborate a bit how you went about it? Did you create a container, remove the entrypoint, supply that to ECSRun, and then call the shell task in the Flow? Could you show me the updated code

@Benjamin Bonhomme in addition to what Kevin said, what may work well for you would be a multi-stage build for a Docker image. You would have your Java app packaged in the first step, then in the second part you would use Prefect base image and COPY the java binaries into the Prefect image. This would make it easier to use

ShellTask

and get stdout and stderr from the Java app into Prefect. Here is an example https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds

@Anna Geller, @Kevin Kho, The base image I use to build the pipeline is actually the prefect one (

FROM prefecthq/prefect:latest-python3.8

) Afterward, I install the Java runtime necessary to run the Java Jobs (

openjdk-11-jre-headless

) As for the Java part of it I just COPY the .zip file and unzip it in the working directory, containing the shell script and the Jar files necessary for the run. So both Prefect & a Java runtime are embedded in the container. The images have no Entrypoint note: the .sh file is copied at / Here is the code:

from prefect.run_configs import ECSRun
from prefect import Flow
from prefect.storage import Docker
from prefect.tasks.shell import ShellTask
import yaml


FLOW_NAME = "ecs_slack_notify"
IMAGE_NAME = "<http://XXX.dkr.ecr.eu-west-1.amazonaws.com/slack_notify_from_prefect:0.1|XXX.dkr.ecr.eu-west-1.amazonaws.com/slack_notify_from_prefect:0.1>"
EXECUTION_ROLE_ARN = "arn:aws:iam::XXX:role/PREFECT_ECS_AGENT_TASK_EXECUTION_ROLE"
TASK_ROLE_ARN = "arn:aws:iam::XXX:role/PREFECT_TASK_ROLE"

TASK_DEFINITION = yaml.safe_load(
"""
cpu: 256
memory: 512
containerDefinitions:
- name: flow
  logConfiguration:
    logDriver: awslogs
    options:
      awslogs-group: /ecs/prefect-agent
      awslogs-region: eu-west-1
      awslogs-stream-prefix: ecs
      awslogs-create-group: 'true'
networkMode: awsvpc
"""
  )

RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'etl-jobs-ecs-cluster'},
                    labels=["prod"],
                    task_role_arn=TASK_ROLE_ARN,
                    execution_role_arn=EXECUTION_ROLE_ARN,
                    image=IMAGE_NAME,
                    memory=512,
                    cpu=256,
                    env={"PREFECT__LOGGING__LEVEL":"DEBUG"},
                    task_definition=TASK_DEFINITION)


task = ShellTask()

with Flow(FLOW_NAME, run_config=RUN_CONFIG) as flow:
    print("Flow starts")
    run_container = task(command="/bin/sh -c slack_notify_from_prefect_run.sh")

flow.storage = Docker(
    registry_url="<http://XXX.dkr.ecr.eu-west-1.amazonaws.com|XXX.dkr.ecr.eu-west-1.amazonaws.com>",
    image_name="flows_storage",
    image_tag="latest",
    dockerfile="Dockerfile-flow-storage"
)


flow.storage.build()


if __name__ == "__main__":
    flow.register("talend-jobs")

thanks @Benjamin Bonhomme, does it work now?

No @Anna Geller, I am still getting the exit code 127 even though the .sh exists in the referred path and is executable. Hence my questions about the context of the ShellTask command

@Benjamin Bonhomme ShellTask would execute the command in your working directory. There is helper_script that allows you e.g. to execute some command beforehand like changing a directory or so. You could specify a working directory in your Dockerfile:

WORKDIR /your/path

by default, the Docker storage uses /opt/prefect/flows to store flows. Overall, what may make it all a bit more difficult is that you use Docker storage. Perhaps if you would use S3 Storage as a flow storage instead, this would be easier? because you have no dependencies in your flow at all, and your custom image that you supply to ECSRun covers all dependencies. So I think changing the storage to S3 could make things easier. This is how you could use it:

from prefect.storage import S3

FLOW_NAME = "s3_storage_demo"
STORAGE = S3(
    bucket="prefect-datasets",
    key=f"flows/{FLOW_NAME}.py",
    stored_as_script=True,
    # if you add local_script_path, Prefect will upload the local Flow script to S3 during registration
    local_script_path=f"{FLOW_NAME}.py",
)

Hi @Anna Geller, hi @Kevin Kho, The ShellTask works like a charm to execute the desired scripts within the container. I am getting the the logs and things I redirect to the stdout perfectly! As for the storage, S3 is indeed a lot easier, I find the Docker storage quite confusing. Now that I got a generic Docker Image to run pretty much anything I want, I will explore the sub Flows possibilities to create a sound execution pipeline. Thank you so much for your help guys! I just discovered Prefect and I really appreciate it! I am looking forward to contribute to it in the closest future! 🙂 Cheers, Benjamin