https://prefect.io logo
Join Slack
Powered by
I’ve built a flow that continues to be stuck in th...
# ask-community
k
I’ve built a flow that continues to be stuck in the ‘Submitted’ state. Following this documentation, I was able to narrow down that the cause is with my docker image. I have multiple docker images in my repo that works fine, but using this specific docker image in my ECSRun causes my flow to be stuck in a submitted state. Could anyone let me know what might be wrong with this docker image?
Copy code
FROM ubuntu:18.04
COPY requirements.txt /requirements.txt
COPY google_secret.json $HOME/.config/gspread_pandas/google_secret.json
ENV PATH="/root/miniconda3/bin:$PATH"
ARG PATH="/root/miniconda3/bin:$PATH"
RUN apt-get update

RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/*

RUN wget \
    <https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh> \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh 
RUN conda install pip
RUN pip install -r requirements.txt
RUN conda install -c conda-forge theano-pymc -y
k
It might be architecture related. You will have more insight if you can attach a logging group to your flow. Let me look for an example for you
k
That would be great, thank you!
k
Check this and note you have to create the log group in cloud watch so that you can log the errors cuz ECS doesnt give you anything by default
k
Silly request but any chance you have an example where I could add those arguments in my flow, rather than as a YAML file?
k
I dont have it in dict format. Maybe try reading in the yaml with 
yaml.safe_load
and I think that returns a dict and then it should be compatible to pass to ECSRun
k
k
Yeah that’s good. I should’ve probably linked it. I just didnt cuz it didnt have the logging configuration lol
k
Am I right to assume that arguments with 
-
in the YAML becomes 
_
when defined in the flow file? 
awslogs-region -> awslogs_region
k
I actually don’t know. Does it error if you don’t?
k
I can’t seem to get the 
logConfigurations
set up correctly within the flow .py file. I’ve tried multiple approaches and each one would give me vague errors that don’t quite help with debugging like 
'str' object has no attribute 'get'
I think the difficulty here is I don’t exactly know what arguments to pass through, and what format those arguments would want to accept (following documentation for ECSRun doesn’t provide enough details for the 
task_definition
argument). Discourse also doesn’t have a lot of information on this either. Is there any chance that @Anna Geller could help me with the process and develop some specific example for future references in the future?
k
I can probably try a bit later tonight, but I agree it’s not straightforward using the dictionary as well
Have you tried this syntax?
Copy code
{
    "containerDefinitions": [
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-west-2",
                    "awslogs-create-group": "true",
                    "awslogs-stream-prefix": "firelens"
                }
            }
}
This might be a better link
a
@Ken Nguyen I'm afk now, I can check tomorrow
k
Thank you Anna!!
@Kevin Kho Yes that format was the most recent one I’ve tried! I was getting this error while trying: 
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.image should not be null or empty.
Which is weird because I definitely do have an image:
Copy code
task_definition=
                    {
                        "containerDefinitions": [
                            {
                                "name": "top-down-light-test",
                                "image": "prefecthq/prefect",
                                "logConfiguration": {
                                    "logDriver": "awslogs",
                                    "options": {
                                        "awslogs-group": "/ecs/prefect-agent",
                                        "awslogs-region": "us-west-1",
                                        "awslogs-stream-prefix": "ecs"
                                    }
                                }
                            }
                        ]
                    }
I tried with different images (in the most recent try I tried with the prefect image)
k
You can try specifying on the RunConfig level to force it and this might be a bug with the merging of that with the RunConfig kwargs but not immediately sure
k
I’ve also tried that: 1. Set image outside of the task_definition in runconfig 2. Set image both outside of task definition AND within task_definition And got the same image should not be null or empty error
k
Gotcha ok will test this myself later tonight 🙂
k
Thank you very much for your help! This really is quite a niche area to look into
k
a
@Ken Nguyen first off, I checked your Dockerfile and Prefect seems to be installed fine there, see the image (a sanity check as installing new Python3 versions apart from the default 3.6 version preinstalled on Ubuntu 18.04 can be tricky) 1. Not totally related, but what storage do you use? If Docker, then you need to use flow stored as pickle here because in this Dockerfile you don't copy flow files into the image (i.e. you can't use 
stored_as_script=True
). 2. Regarding passing task definition as dict, you've found this example - Kevin's example ☝️ is great. I used the same syntax Kevin used and added example to the same repo as you requested https://github.com/anna-geller/packaging-prefect-flows/blob/6e5fb8451c32e24a8c6bd7[…]lows/s3_ecs_run_task_definition_as_dict_incl_logging_options.py Let us know should this not work for you
6 Views
Open in Slack
PreviousNext
Powered by