https://prefect.io
Join Slack
Hi I have a problem running my flow. The problem has occurred before with the `local agent`. The fix...
# ask-community
d
Hi I have a problem running my flow. The problem has occurred before with the 
local agent
. The fix for that was to run the agent from the folder of the flow. How do I fix this for the docker agent when the flow is dockerized?
Copy code
Failed to load and execute flow run: ModuleNotFoundError("No module named 'parameters'")
a
The best way would be to package your dependencies into a Docker image - this thread provides much more information
d
what do you mean by "package your dependencies into a Docker image"? All my dependencies are in my docker image
Copy code
root
- flow.py
- paramenters.py
I use this Dockerfile command:
Copy code
COPY ./src/flows/data_transformation ./
All my files are in the data_transformation folder
btw. When setting up the flow storage I set 
_stored_as_script_=True
a
Gotcha, thanks for providing more info. For script storage there is a couple of more things you need to do https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows_no_build/docker_script_docker_run_local_image.py
Also, can you share the current state of your Dockerfile? Based on this, you may need to add a couple more steps because you are using a different base image than the one from Prefect especially with script storage. With pickle storage, Prefect can modify your Dockerfile before build to add things like entry point but with Script storage you would need to add those missing steps - check those in the Prefect main repo, there is a Dockerfile Prefect uses https://prefect-community.slack.com/archives/CL09KU1K7/p1646235910252089?thread_ts=1646233144.405739&cid=CL09KU1K7
Alternatively, switch to Prefect base image and run the node stuff separately perhaps? You haven't explained yet why do you need node :)
d
This is now my docker file:
Copy code
FROM node:16

RUN apt-get update
RUN apt-get install -y python3-pip

RUN pip3 install --no-cache --upgrade pip setuptools

COPY ./requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt

COPY ./src/flows/data_transformation ./
there are several python files inside the data_transformation folder I use node to run a javascript file as a subprocess, so I need node.js for my flow
When you mentioned "packaging my dependencies", did you mean using 
setuptools
?
a
Yes, I meant adding setup.py So this Dockerfile won't work with script storage but it may work with pickle storage if you use the same Python and Prefect version in your local registration environment as you used in your Dockerfile
For script based Docker storage you have to use PrefectHQ base image or extend your Dockerfile significantly by adding commands similar to https://github.com/PrefectHQ/prefect/blob/master/Dockerfile
d
mhm 🤔 I'll try using the prefect docker base. thank you for your help!:)
🙌 1
a
Good choice, it's also more performant and faster to build than installing Python manually 👍
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by