https://prefect.io logo
Title
d

Daniel Nilsen

03/03/2022, 9:53 AM
Hi I have a problem running my flow. The problem has occurred before with the
local agent
. The fix for that was to run the agent from the folder of the flow. How do I fix this for the docker agent when the flow is dockerized?
Failed to load and execute flow run: ModuleNotFoundError("No module named 'parameters'")
a

Anna Geller

03/03/2022, 10:17 AM
The best way would be to package your dependencies into a Docker image - this thread provides much more information
d

Daniel Nilsen

03/03/2022, 10:25 AM
what do you mean by "package your dependencies into a Docker image"? All my dependencies are in my docker image
root
- flow.py
- paramenters.py
I use this Dockerfile command:
COPY ./src/flows/data_transformation ./
All my files are in the data_transformation folder
btw. When setting up the flow storage I set
_stored_as_script_=True
a

Anna Geller

03/03/2022, 10:42 AM
Gotcha, thanks for providing more info. For script storage there is a couple of more things you need to do https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows_no_build/docker_script_docker_run_local_image.py
Also, can you share the current state of your Dockerfile? Based on this, you may need to add a couple more steps because you are using a different base image than the one from Prefect especially with script storage. With pickle storage, Prefect can modify your Dockerfile before build to add things like entry point but with Script storage you would need to add those missing steps - check those in the Prefect main repo, there is a Dockerfile Prefect uses https://prefect-community.slack.com/archives/CL09KU1K7/p1646235910252089?thread_ts=1646233144.405739&cid=CL09KU1K7
Alternatively, switch to Prefect base image and run the node stuff separately perhaps? You haven't explained yet why do you need node :)
d

Daniel Nilsen

03/03/2022, 10:58 AM
This is now my docker file:
FROM node:16

RUN apt-get update
RUN apt-get install -y python3-pip

RUN pip3 install --no-cache --upgrade pip setuptools

COPY ./requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt

COPY ./src/flows/data_transformation ./
there are several python files inside the data_transformation folder I use node to run a javascript file as a subprocess, so I need node.js for my flow
When you mentioned "packaging my dependencies", did you mean using
setuptools
?
a

Anna Geller

03/03/2022, 11:03 AM
Yes, I meant adding setup.py So this Dockerfile won't work with script storage but it may work with pickle storage if you use the same Python and Prefect version in your local registration environment as you used in your Dockerfile
For script based Docker storage you have to use PrefectHQ base image or extend your Dockerfile significantly by adding commands similar to https://github.com/PrefectHQ/prefect/blob/master/Dockerfile
d

Daniel Nilsen

03/03/2022, 11:12 AM
mhm 🤔 I'll try using the prefect docker base. thank you for your help!:)
🙌 1
a

Anna Geller

03/03/2022, 11:15 AM
Good choice, it's also more performant and faster to build than installing Python manually 👍