https://prefect.io logo
Channels
Powered by
j

jack

11/04/2021, 1:47 AM
When running flows on Amazon ECS, must the container that runs the flow be based on the PrefectHQ/prefect docker image? I've been attempting to use ECSRun() with S3 storage and a docker image based on amazonlinux:2, but the ECS log shows this error each time: 
/bin/sh: prefect: command not found
The Dockerfile used to build the image based on amazonlinux:2 runs 
pip install prefect[aws]
, and when I run the image locally 
prefect
is in the path.
k

Kevin Kho

11/04/2021, 1:49 AM
Hey @jack, not necessarily, but it would just need prefect installed. Could you show me your Dockerfile? Or maybe some simplified version of it?
j

jack

11/04/2021, 2:22 AM
Here is the simplified Dockerfile:
Copy code
FROM amazonlinux:2.0.20211005.0

WORKDIR /project-name

# Add files required for provisioning
ADD requirements.txt /project-name/
ADD bin /project-name/bin/


# Install things
RUN yum update -y
RUN amazon-linux-extras enable python3.8
RUN yum install -y mlocate python38
RUN updatedb
RUN python3.8 -m pip install --upgrade pip
RUN pip3.8 install -r requirements.txt

# Add remaining required files
ADD dispatcher.py /project-name/
ADD models /project-name/models/

CMD ["python3.8", "dispatcher.py"]
In my Task Definition 
command
is set to 
["python3.8", "dispatcher.py"]
but the ECS task instead appears to run 
["/bin/sh","-c","prefect execute flow-run"]
k

Kevin Kho

11/04/2021, 2:53 AM
So just to explain what happens, Prefect adds commands on top of your container to run the Flow. The agent will pull the flow from storage, and then run it on top of the RunConfig with 
prefect execute flow-run
. So this command is used internally to run it on the image. This is working as intended.
If you are thinking that the wrong version of Python is being pulled, that is my suspicion. I was trying to build your container locally to dig more, but I am having issues with 
gcc
. How did you install it? This is what I’m trying:
Copy code
FROM amazonlinux:2.0.20211005.0

# Install things
RUN yum update -y
RUN amazon-linux-extras enable python3.8
RUN yum install -y mlocate python38
RUN yum install gcc
RUN python3.8 -m pip install --upgrade pip
RUN pip3.8 install prefect
What is the content of 
dispatcher.py
? You shouldn’t need it to run a flow on ECS
I guess this is running as intended. Do you have logging enabled on this ECS task defnition?
j

jack

11/04/2021, 3:00 AM
If I remove 
RUN yum install gcc
from the Dockerfile you pasted,it builds on my machine.
k

Kevin Kho

11/04/2021, 3:03 AM
Oh what? I get
Copy code
#9 7.622     unable to execute 'gcc': No such file or directory
#9 7.622     C compiler or Python headers are not installed on this system. Try to run:
#9 7.622     sudo yum install gcc python3-devel
#9 7.622     error: command 'gcc' failed with exit status 1
And I tried installing 
python3-devel
and 
gcc
and it’s not working for me. What is your Docker version? (though I think this shouldn’t matter)
👀 1
j

jack

11/04/2021, 3:04 AM
Copy code
jd /tmp/td docker --version
Docker version 20.10.8, build 3967b7d
Say my directory looks like this:
Copy code
dispatcher.py
models/
    model_1.py
    model_2.py
The flow is defined in dispatcher.py. The flow's tasks use models.model_1 and models.model_2. I assumed that dispatcher.py and the models directory needed to live on the image that runs the flow. If that is not the case, then where would this code live?
k

Kevin Kho

11/04/2021, 3:11 AM
Ah ok so for this setup, yes you do need Docker to put them all together. And you are right that in this case, the Flow code needs to live in the image (though if this were all one file, you could store it on Github then run on top of the image.). What I meant with my comment earlier was that you shouldn’t need the
Copy code
CMD ["python3.8", "dispatcher.py"]
as part of the Dockerfile because Prefect goes in and runs the flow code for you by adding that command to the container that you are seeing.
j

jack

11/04/2021, 3:12 AM
Ok, I'll remove the 
CMD
statement then.
k

Kevin Kho

11/04/2021, 3:13 AM
But that’s not our issue here I think. It’s really the 
prefect
command is not found. But if you exec into the container, you can use it right?
Maybe you can try adding the 
prefect
command explicitly to the PATH of your container like this ?
🙏🏽 1
j

jack

11/04/2021, 3:16 AM
Yes,
Copy code
$ docker run -it --rm my-image bash
bash-4.2# prefect version
0.15.7
sbash-4.2# sh
sh-4.2# prefect version
0.15.7
k

Kevin Kho

11/04/2021, 3:21 AM
This certainly looks good. Sorry, I really don’t have other suggestions other than to explicitly try adding the directory to the path. I can definitely ask the team more tomorrow for other suggestions.
j

jack

11/04/2021, 3:23 AM
This is where prefect is located:
/usr/local/bin
is already in $PATH when I run the image locally. I'm not sure if $PATH is different when run via ECS
k

Kevin Kho

11/04/2021, 3:24 AM
Gotcha. I think it should be the same. Will look for more ideas tom
j

jack

11/04/2021, 3:25 AM
Thanks for digging 😅
a

Anna Geller

11/04/2021, 9:58 AM
@jack afaik, you should be using one of PrefectHQ base images. Otherwise, your custom image would somehow need to replicate the Prefect Dockerfile: https://github.com/PrefectHQ/prefect/blob/master/Dockerfile At least two people from the community were trying to use image which was not based on PrefectHQ image and it didn’t work for them. Do you have any specific reason to use 
amazonlinux:2.0.20211005.0
? Prefect base images are optimized to work with Prefect flows, and they are all based on Python slim base image. This way, your image should be smaller in size and a bit faster to spin up on ECS than installing Python from Amazon Extras:
Copy code
RUN amazon-linux-extras enable python3.8
j

jack

11/04/2021, 1:56 PM
Attempting a simple example that uses the 
prefecthq/prefect
image and requires no external python modules. ECS errors out with "*Stopped reason:* *Essential container in task exited*"
a

Anna Geller

11/04/2021, 2:05 PM
this is not an error really, it’s only telling us that the Prefect container in the ECS task finished execution. You can find out more if you go to the CloudWatch logs for this task and see what was the reason for the container to exit
btw, how did you configure your ECSAgent - does it run in-cluster as an ECS service? Quite recently, I’ve updated the walkthrough on ECS agent set up - sharing in case it may be useful in some way: https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df
j

jack

11/04/2021, 3:00 PM
Reading the articles now
👍 1
Powered by