https://prefect.io logo
Title
d

Daniel Nilsen

03/02/2022, 2:59 PM
Hi Im am using docker storage and I am trying to make a docker image with my flow. I am getting an error. Any idea about how I can solve this?
Step 15/15 : RUN python /opt/prefect/healthcheck.py '["flow-data_transformation.py"]' '(3, 10)'
 ---> Running in 2853f6886cd5
  File "/opt/prefect/healthcheck.py", line 17
    def system_check(python_version: str):
                                   ^
SyntaxError: invalid syntax
the image is using
node:16
as base and has
python3
installed
a

Anna Geller

03/02/2022, 3:08 PM
I have two options for you: a thorough one and a hack - choose wisely 😄 #1 The thorough one - there is a reason why the health check is failing and that's probably due to some Python version mismatch between your local flow registration environment and the Python version used in your image, or some things in your flow not being serializable. This docs page dives deeper into how you can debug this. #2 The hack: you can ignore the healthcheck
flow.storage = Docker(ignore_healthchecks=True)
To give you some background on option #1 - by default, Prefect pickles your flow using cloudpickle during the flow registration (build time). The error indicates python version mismatch which may have the effect that your flow gets serialized incorrectly and cannot be deserialized at runtime. If you want to avoid pickling your flow completely, you can copy your flow file into your Docker image within your Dockerfile and use this approach with
stored_as_script=True
.
d

Daniel Nilsen

03/02/2022, 3:23 PM
The health check is not really failing. It has a "syntax error" within itself. When calling
python
in the container it runs
Python 2.7.16
. I am guessing the health check needs python 3.5+ ? 🤔 Is there a way I can have prefect Docker builder call
python3
instead?
a

Anna Geller

03/02/2022, 3:34 PM
Can you share your Dockerfile? Two notes on that: 1. If you go to the setup.py in the Prefect repo for any release, you can see the minimal required Python version that is supported https://github.com/PrefectHQ/prefect/blob/master/setup.py#L144 2. You don't have to use the Prefect Docker build process even if you use Docker storage. The above link I shared with script storage requires you to build the image yourself and just provide it as a reference - I think especially in use cases such as yours, it can be beneficial to build the Docker image yourself so that you have full control over the process. The Docker builder is more for simple use cases where you may just need a handful of packages to be installed and you don't have any complex dependencies (and given the npm and python 2.7, you certainly do)
d

Daniel Nilsen

03/02/2022, 3:45 PM
Ah oki I see. never mind
node.js
. I do not have a dependency on
python2
. My deps are
python3
and some
pip
packages.
FROM node:16

COPY ./src/flows/data_transformation ./
COPY ./requirements.txt ./requirements.txt

RUN apt-get update
RUN apt-get install -y python3-pip

RUN pip3 install --no-cache --upgrade pip setuptools
RUN pip3 install -r requirements.txt
a

Anna Geller

03/02/2022, 3:54 PM
I see. I'd recommend building your own image then. The repo I shared provides an example Dockerfile. Btw, It would be beneficial to move your COPY commands below to speed up your builds. This way, the previous Docker layers are cached and when your file dependencies change, you can just copy them over without having to reinstall your code dependencies (because they are cached).
FROM node:16

RUN apt-get update
RUN apt-get install -y python3-pip

RUN pip3 install --no-cache --upgrade pip setuptools
COPY ./requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt

COPY ./src/flows/data_transformation ./
d

Daniel Nilsen

03/02/2022, 3:56 PM
ah nice! thanks for the advice:)
👍 1