Ken Nguyen
03/04/2022, 6:15 PMFROM ubuntu:18.04
COPY requirements.txt /requirements.txt
COPY google_secret.json $HOME/.config/gspread_pandas/google_secret.json
ENV PATH="/root/miniconda3/bin:$PATH"
ARG PATH="/root/miniconda3/bin:$PATH"
RUN apt-get update
RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/*
RUN wget \
<https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh> \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
RUN conda install pip
RUN pip install -r requirements.txt
RUN conda install -c conda-forge theano-pymc -y
Kevin Kho
03/04/2022, 6:16 PMKen Nguyen
03/04/2022, 6:16 PMKevin Kho
03/04/2022, 6:17 PMKen Nguyen
03/04/2022, 6:19 PMKevin Kho
03/04/2022, 6:40 PMyaml.safe_load
and I think that returns a dict and then it should be compatible to pass to ECSRunKen Nguyen
03/04/2022, 6:43 PMKevin Kho
03/04/2022, 6:48 PMKen Nguyen
03/04/2022, 7:00 PM-
in the YAML becomes _
when defined in the flow file?
awslogs-region -> awslogs_region
Kevin Kho
03/04/2022, 7:33 PMKen Nguyen
03/04/2022, 9:44 PMlogConfigurations
set up correctly within the flow .py file. I’ve tried multiple approaches and each one would give me vague errors that don’t quite help with debugging like 'str' object has no attribute 'get'
I think the difficulty here is I don’t exactly know what arguments to pass through, and what format those arguments would want to accept (following documentation for ECSRun doesn’t provide enough details for the task_definition
argument). Discourse also doesn’t have a lot of information on this either.
Is there any chance that @Anna Geller could help me with the process and develop some specific example for future references in the future?Kevin Kho
03/04/2022, 9:46 PM{
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "firelens-container",
"awslogs-region": "us-west-2",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
}
}
Anna Geller
03/04/2022, 10:25 PMKen Nguyen
03/04/2022, 10:25 PMAn error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.image should not be null or empty.
Which is weird because I definitely do have an image:
task_definition=
{
"containerDefinitions": [
{
"name": "top-down-light-test",
"image": "prefecthq/prefect",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/prefect-agent",
"awslogs-region": "us-west-1",
"awslogs-stream-prefix": "ecs"
}
}
}
]
}
Kevin Kho
03/04/2022, 10:27 PMKen Nguyen
03/04/2022, 10:29 PMKevin Kho
03/04/2022, 10:30 PMKen Nguyen
03/04/2022, 10:48 PMKevin Kho
03/05/2022, 12:25 AMAnna Geller
03/05/2022, 11:50 AMstored_as_script=True
).
2. Regarding passing task definition as dict, you've found this example - Kevin's example ☝️ is great. I used the same syntax Kevin used and added example to the same repo as you requested https://github.com/anna-geller/packaging-prefect-flows/blob/6e5fb8451c32e24a8c6bd7[…]lows/s3_ecs_run_task_definition_as_dict_incl_logging_options.py
Let us know should this not work for you