Hi, I'm using docker image pushed on the container registry as storage. I have a lot of custom dependency py files needed to run my flow. The directory looks something like this (image below) . And my Dockerfile content is

# specify a base image

FROM python:3.8-slim

# copy all folder contents to the image

COPY . .

# install all dependencies

RUN apt-get update && apt-get -y install libpq-dev gcc && pip install psycopg2

RUN pip install -r requirements.txt

My understanding is

COPY . .

should copy all the files required to run the flow into the image. But I'm getting an error saying no module found (image attached). Also Here's my STORAGE and RUN_CONFIG

STORAGE = Docker(registry_url='<http://aws_id.dkr.ecr.region.amazonaws.com/|aws_id.dkr.ecr.region.amazonaws.com/>',

                 

image_name='name',

                 

image_tag='tag',

                 

dockerfile='Dockerfile')

RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'cluster-name'},

                   

execution_role_arn='arn:aws:iam::aws_id:role/role',

                   

labels=['dev-modelling', 'flow-test'])

Am I missing something?? Really appreciate the help. Thanks in advance!!

Hi @Sridhar, so it will copy the files, but that doesn’t mean it’s available in the Python path. I think you might be able to get this working by specifying the

WORKDIR

of your image. Or you could make it a Python package and install it to make it available in the container whatever path the flow is running from. I have a guide here . It won’t be a lot of extra steps considering what you have already

I have found one alternative... say I have a local directory called

helpers

which is setup as a python package (has a

___init___.py

etc), I do this in my dockerfile:

COPY helpers/ /usr/local/lib/python$PYTHON_RUNTIME/helpers

Maybe there is some reason this is a bad idea, but it definitely makes the package available to be imported in the flow.

Thanks @Kevin Kho, @Billy McMonagle I'll give a go at both and see what works best. Also, just noticed, it's looking for modules on a local path. Is this because the python code was not packaged properly?

You need to point where the file lives in the Docker storage so Prefect knows where to get it. You can specify

stored_as_script=True

and

path

to point to the flow location. Docs here

Thanks for that! Works fine now.