https://prefect.io
Join Slack
Hello! I have a custom docker container for my prefect flows and I have copied over a python file wi...
# ask-community
a
Hello! I have a custom docker container for my prefect flows and I have copied over a python file with some custom functions onto this docker container. However, my prefect flow is unable to import these functions. I am getting a module not found error. How do you access functions from python files in custom docker containers in prefect flow?
✅ 1
👍 1
a
yup, this blog post is a great resource - there are also more resources and background info on this error here
👍 1
a
I've followed all of these instructions and I can import the module 
src
on my docker container and yet prefect doesn't seem to be able to do it.
Copy code
[2022-04-14T03:30:34.389Z]     from src.prefect_state_handlers import FlowStateHandler

[2022-04-14T03:30:34.389Z]   ModuleNotFoundError: No module named 'src'
k
Can I see your Dockerfile?
a
Directory structure and dockerfile
k
Your Dockerfile is inside the folder so 
ADD . .
would not copy source in right? Does it actually when you exec in?
a
yeah it does
I did a 
pip freeze
during flow execution (without 
import src
) and see this 
src @ file:///root/prefect-docker
I also did an 
ls
during my flow execution and see all of those files listed there
k
can i see the 
setup.py
?
a
Copy code
from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name="src",
    version='0.1.0',
    description='Python library for use across repositories and prefect flows',
    packages=find_packages(),
    install_requires=requirements
)
k
i don’t see anything immediately wrong. can you try going in the container and starting a new python terminal and importing src?
a
yeah I did that and it works
k
can you try changing directory and still seeing if it works?
a
change directory in the docker container and try importing src through a python terminal? Or change directory structure in the docker container?
k
cd somewhere else
in the container and try importing 
src
to make sure it’s really in the python path after the pip install because this might work if you just happen to be in the right directory also
a
yeah it works anywhere in the container
k
What does your RunConfig look like?
a
Is there a special way you need to register flows when you have custom functions being imported from custom modules?
Wondering if this is the problem:
Copy code
prefect backend server
prefect register --project data -p flows/
Because it's failing on the registration step
Copy code
flow.run_config = KubernetesRun(job_template=K8S_JOB_TEMPLATE, image = "******.<http://dkr.ecr.us-east-1.amazonaws.com/oh-data-utils:PR-1.16.b7ed902%22|dkr.ecr.us-east-1.amazonaws.com/oh-data-utils:PR-1.16.b7ed902">)
@Kevin Kho ^ RunConfig
k
Oh if it’s failing during registration, that’s because you didn’t pip install it in your machine. You can just move the import inside a task so it will run in flow execution
Or you need to install it on your local machine that is building the image
a
So the import is for a state handler that needs to be passed like this 
with Flow("everysundayweekly-flow", state_handlers=[statehandler.slack_failure_notification]) as flow:
So importing it in a task is causing the flow to not understand where it's coming from. How would you do this?
k
Ah yeah you can’t defer that. I would just install the library locally as well with the same pip install so that it’s available for importing
a
By that you mean in the prefect flow like a shell bash task?
I still don't see how/where I would import that and use that as a state handler and call it like this https://prefect-community.slack.com/archives/CL09KU1K7/p1649909196570089?thread_ts=1649895625.292669&amp;cid=CL09KU1K7
k
You said the error is during registration time right? You can just do the same 
pip install -e .
on your local machine so that it’s importable by the python installation. Or am I missing something?
Yes you should be able to just
Copy code
import src

with Flow("everysundayweekly-flow", state_handlers=[src.statehandler.slack_failure_notification]) as flow:
a
So the registration is being done by a jenkins job kicked off by a github action. So I am not entirely sure what you mean by "local machine" and where I should be installing it 😅
k
that jenkins job will need that library because it calls the registration which runs the python file
for imports that tasks use it’s easier to workaround because you can just move the import line inside the task but for this, i think you really need it
a
Yeah I figured that'd be the problem. I'll look into it. Thank you 🙂
👍 1
46 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by