https://prefect.io logo
Powered by
Title
a

Apoorva Desai

04/14/2022, 12:20 AM
Hello! I have a custom docker container for my prefect flows and I have copied over a python file with some custom functions onto this docker container. However, my prefect flow is unable to import these functions. I am getting a module not found error. How do you access functions from python files in custom docker containers in prefect flow?
✅ 1
I just found this so I am going to implement my code like this now: https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a
👍 1
a

Anna Geller

04/14/2022, 12:39 AM
yup, this blog post is a great resource - there are also more resources and background info on this error here
👍 1
a

Apoorva Desai

04/14/2022, 3:33 AM
I've followed all of these instructions and I can import the module 
src
on my docker container and yet prefect doesn't seem to be able to do it. 
[2022-04-14T03:30:34.389Z]     from src.prefect_state_handlers import FlowStateHandler

[2022-04-14T03:30:34.389Z]   ModuleNotFoundError: No module named 'src'
k

Kevin Kho

04/14/2022, 3:33 AM
Can I see your Dockerfile?
a

Apoorva Desai

04/14/2022, 3:34 AM
Directory structure and dockerfile
k

Kevin Kho

04/14/2022, 3:35 AM
Your Dockerfile is inside the folder so 
ADD . .
would not copy source in right? Does it actually when you exec in?
a

Apoorva Desai

04/14/2022, 3:35 AM
yeah it does
I did a 
pip freeze
during flow execution (without 
import src
) and see this 
src @ file:///root/prefect-docker
I also did an 
ls
during my flow execution and see all of those files listed there
k

Kevin Kho

04/14/2022, 3:37 AM
can i see the 
setup.py
?
a

Apoorva Desai

04/14/2022, 3:39 AM
from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name="src",
    version='0.1.0',
    description='Python library for use across repositories and prefect flows',
    packages=find_packages(),
    install_requires=requirements
)
k

Kevin Kho

04/14/2022, 3:41 AM
i don’t see anything immediately wrong. can you try going in the container and starting a new python terminal and importing src?
a

Apoorva Desai

04/14/2022, 3:42 AM
yeah I did that and it works
k

Kevin Kho

04/14/2022, 3:43 AM
can you try changing directory and still seeing if it works?
a

Apoorva Desai

04/14/2022, 3:45 AM
change directory in the docker container and try importing src through a python terminal? Or change directory structure in the docker container?
k

Kevin Kho

04/14/2022, 3:46 AM
cd somewhere else
in the container and try importing 
src
to make sure it’s really in the python path after the pip install because this might work if you just happen to be in the right directory also
a

Apoorva Desai

04/14/2022, 3:48 AM
yeah it works anywhere in the container
k

Kevin Kho

04/14/2022, 3:50 AM
What does your RunConfig look like?
a

Apoorva Desai

04/14/2022, 3:53 AM
Is there a special way you need to register flows when you have custom functions being imported from custom modules?
Wondering if this is the problem: 
prefect backend server
prefect register --project data -p flows/
Because it's failing on the registration step
flow.run_config = KubernetesRun(job_template=K8S_JOB_TEMPLATE, image = "******.<http://dkr.ecr.us-east-1.amazonaws.com/oh-data-utils:PR-1.16.b7ed902%22|dkr.ecr.us-east-1.amazonaws.com/oh-data-utils:PR-1.16.b7ed902">)
@Kevin Kho ^ RunConfig
k

Kevin Kho

04/14/2022, 3:56 AM
Oh if it’s failing during registration, that’s because you didn’t pip install it in your machine. You can just move the import inside a task so it will run in flow execution
Or you need to install it on your local machine that is building the image
a

Apoorva Desai

04/14/2022, 4:06 AM
So the import is for a state handler that needs to be passed like this 
with Flow("everysundayweekly-flow", state_handlers=[statehandler.slack_failure_notification]) as flow:
So importing it in a task is causing the flow to not understand where it's coming from. How would you do this?
k

Kevin Kho

04/14/2022, 4:09 AM
Ah yeah you can’t defer that. I would just install the library locally as well with the same pip install so that it’s available for importing
a

Apoorva Desai

04/14/2022, 4:10 AM
By that you mean in the prefect flow like a shell bash task?
I still don't see how/where I would import that and use that as a state handler and call it like this https://prefect-community.slack.com/archives/CL09KU1K7/p1649909196570089?thread_ts=1649895625.292669&amp;cid=CL09KU1K7
k

Kevin Kho

04/14/2022, 4:20 AM
You said the error is during registration time right? You can just do the same 
pip install -e .
on your local machine so that it’s importable by the python installation. Or am I missing something?
Yes you should be able to just 
import src

with Flow("everysundayweekly-flow", state_handlers=[src.statehandler.slack_failure_notification]) as flow:
a

Apoorva Desai

04/14/2022, 4:38 AM
So the registration is being done by a jenkins job kicked off by a github action. So I am not entirely sure what you mean by "local machine" and where I should be installing it 😅
k

Kevin Kho

04/14/2022, 4:40 AM
that jenkins job will need that library because it calls the registration which runs the python file
for imports that tasks use it’s easier to workaround because you can just move the import line inside the task but for this, i think you really need it
a

Apoorva Desai

04/14/2022, 4:46 AM
Yeah I figured that'd be the problem. I'll look into it. Thank you 🙂
👍 1
2 Views
#prefect-communityJoin Slack
Powered by