Thread
#prefect-community
    s

    Sridhar

    10 months ago
    Hi, I'm using docker image pushed on the container registry as storage. I have a lot of custom dependency py files needed to run my flow. The directory looks something like this (image below) . And my Dockerfile content is
    # specify a base image
    FROM python:3.8-slim
    # copy all folder contents to the image
    COPY . .
    # install all dependencies
    RUN apt-get update && apt-get -y install libpq-dev gcc && pip install psycopg2
    RUN pip install -r requirements.txt
    My understanding is
    COPY . .
    should copy all the files required to run the flow into the image. But I'm getting an error saying no module found (image attached). Also Here's my STORAGE and RUN_CONFIG
    STORAGE = Docker(registry_url='<http://aws_id.dkr.ecr.region.amazonaws.com/|aws_id.dkr.ecr.region.amazonaws.com/>',
                     
    image_name='name',
                     
    image_tag='tag',
                     
    dockerfile='Dockerfile')
    RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'cluster-name'},
                       
    execution_role_arn='arn:aws:iam::aws_id:role/role',
                       
    labels=['dev-modelling', 'flow-test'])
    Am I missing something?? Really appreciate the help. Thanks in advance!!
    Kevin Kho

    Kevin Kho

    10 months ago
    Hi @Sridhar, so it will copy the files, but that doesn’t mean it’s available in the Python path. I think you might be able to get this working by specifying the
    WORKDIR
    of your image. Or you could make it a Python package and install it to make it available in the container whatever path the flow is running from. I have a guide here . It won’t be a lot of extra steps considering what you have already
    Billy McMonagle

    Billy McMonagle

    10 months ago
    I have found one alternative... say I have a local directory called
    helpers
    which is setup as a python package (has a
    ___init___.py
    etc), I do this in my dockerfile:
    COPY helpers/ /usr/local/lib/python$PYTHON_RUNTIME/helpers
    Maybe there is some reason this is a bad idea, but it definitely makes the package available to be imported in the flow.
    s

    Sridhar

    10 months ago
    Thanks @Kevin Kho, @Billy McMonagle I'll give a go at both and see what works best. Also, just noticed, it's looking for modules on a local path. Is this because the python code was not packaged properly?
    Kevin Kho

    Kevin Kho

    10 months ago
    You need to point where the file lives in the Docker storage so Prefect knows where to get it. You can specify
    stored_as_script=True
    and
    path
    to point to the flow location. Docs here
    s

    Sridhar

    10 months ago
    Thanks for that! Works fine now.