https://prefect.io logo
Powered by
Title
k

Ken Nguyen

03/04/2022, 6:15 PM
I’ve built a flow that continues to be stuck in the ‘Submitted’ state. Following this documentation, I was able to narrow down that the cause is with my docker image. I have multiple docker images in my repo that works fine, but using this specific docker image in my ECSRun causes my flow to be stuck in a submitted state. Could anyone let me know what might be wrong with this docker image? 
FROM ubuntu:18.04
COPY requirements.txt /requirements.txt
COPY google_secret.json $HOME/.config/gspread_pandas/google_secret.json
ENV PATH="/root/miniconda3/bin:$PATH"
ARG PATH="/root/miniconda3/bin:$PATH"
RUN apt-get update

RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/*

RUN wget \
    <https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh> \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh 
RUN conda install pip
RUN pip install -r requirements.txt
RUN conda install -c conda-forge theano-pymc -y
k

Kevin Kho

03/04/2022, 6:16 PM
It might be architecture related. You will have more insight if you can attach a logging group to your flow. Let me look for an example for you
k

Ken Nguyen

03/04/2022, 6:16 PM
That would be great, thank you!
k

Kevin Kho

03/04/2022, 6:17 PM
Check this and note you have to create the log group in cloud watch so that you can log the errors cuz ECS doesnt give you anything by default
k

Ken Nguyen

03/04/2022, 6:19 PM
Silly request but any chance you have an example where I could add those arguments in my flow, rather than as a YAML file?
k

Kevin Kho

03/04/2022, 6:40 PM
I dont have it in dict format. Maybe try reading in the yaml with 
yaml.safe_load
and I think that returns a dict and then it should be compatible to pass to ECSRun
k

Ken Nguyen

03/04/2022, 6:43 PM
I believe I’ve found an example here: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/s3_ecs_run_task_definition_as_dict.py
👍 1
k

Kevin Kho

03/04/2022, 6:48 PM
Yeah that’s good. I should’ve probably linked it. I just didnt cuz it didnt have the logging configuration lol
k

Ken Nguyen

03/04/2022, 7:00 PM
Am I right to assume that arguments with 
-
in the YAML becomes 
_
when defined in the flow file? 
awslogs-region -> awslogs_region
k

Kevin Kho

03/04/2022, 7:33 PM
I actually don’t know. Does it error if you don’t?
k

Ken Nguyen

03/04/2022, 9:44 PM
I can’t seem to get the 
logConfigurations
set up correctly within the flow .py file. I’ve tried multiple approaches and each one would give me vague errors that don’t quite help with debugging like 
'str' object has no attribute 'get'
I think the difficulty here is I don’t exactly know what arguments to pass through, and what format those arguments would want to accept (following documentation for ECSRun doesn’t provide enough details for the 
task_definition
argument). Discourse also doesn’t have a lot of information on this either. Is there any chance that @Anna Geller could help me with the process and develop some specific example for future references in the future?
k

Kevin Kho

03/04/2022, 9:46 PM
I can probably try a bit later tonight, but I agree it’s not straightforward using the dictionary as well
Have you tried this syntax? 
{
    "containerDefinitions": [
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-west-2",
                    "awslogs-create-group": "true",
                    "awslogs-stream-prefix": "firelens"
                }
            }
}
This might be a better link
a

Anna Geller

03/04/2022, 10:25 PM
@Ken Nguyen I'm afk now, I can check tomorrow
k

Ken Nguyen

03/04/2022, 10:25 PM
Thank you Anna!!
@Kevin Kho Yes that format was the most recent one I’ve tried! I was getting this error while trying: 
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.image should not be null or empty.
Which is weird because I definitely do have an image: 
task_definition=
                    {
                        "containerDefinitions": [
                            {
                                "name": "top-down-light-test",
                                "image": "prefecthq/prefect",
                                "logConfiguration": {
                                    "logDriver": "awslogs",
                                    "options": {
                                        "awslogs-group": "/ecs/prefect-agent",
                                        "awslogs-region": "us-west-1",
                                        "awslogs-stream-prefix": "ecs"
                                    }
                                }
                            }
                        ]
                    }
I tried with different images (in the most recent try I tried with the prefect image)
k

Kevin Kho

03/04/2022, 10:27 PM
You can try specifying on the RunConfig level to force it and this might be a bug with the merging of that with the RunConfig kwargs but not immediately sure
k

Ken Nguyen

03/04/2022, 10:29 PM
I’ve also tried that: 1. Set image outside of the task_definition in runconfig 2. Set image both outside of task definition AND within task_definition And got the same image should not be null or empty error
k

Kevin Kho

03/04/2022, 10:30 PM
Gotcha ok will test this myself later tonight 🙂
k

Ken Nguyen

03/04/2022, 10:48 PM
Thank you very much for your help! This really is quite a niche area to look into
k

Kevin Kho

03/05/2022, 12:25 AM
This works for me: https://github.com/kvnkho/demos/blob/main/prefect/git_ecs.py
💯 1
:upvote: 1
a

Anna Geller

03/05/2022, 11:50 AM
@Ken Nguyen first off, I checked your Dockerfile and Prefect seems to be installed fine there, see the image (a sanity check as installing new Python3 versions apart from the default 3.6 version preinstalled on Ubuntu 18.04 can be tricky) 1. Not totally related, but what storage do you use? If Docker, then you need to use flow stored as pickle here because in this Dockerfile you don't copy flow files into the image (i.e. you can't use 
stored_as_script=True
). 2. Regarding passing task definition as dict, you've found this example - Kevin's example ☝️ is great. I used the same syntax Kevin used and added example to the same repo as you requested https://github.com/anna-geller/packaging-prefect-flows/blob/6e5fb8451c32e24a8c6bd7[…]lows/s3_ecs_run_task_definition_as_dict_incl_logging_options.py Let us know should this not work for you
6 Views
#prefect-communityJoin Slack
Powered by