https://prefect.io logo
Join Slack
Powered by
Hello :slightly_smiling_face:, I deployed Prefect ...
# ask-community
o
Hello 🙂, I deployed Prefect server 0.14.14. Everything works fine locally. I am a bit confused so if anyone can help me: • I have a workload of Machine learning tasks That i would like to execute inside a docker container • I have everything packaged inside a container • I have an agent running on my machine "`prefect agent docker start --show-flow-logs -l agent-name --no-pull`" • When i do 
docker run my_image
, i get this error each time:
Copy code
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
Should i run the agent inside the container ? Thank you
k
Hey @Oussama Louati! No you don’t run the agent inside the container. The agent will poll Prefect for Flows that need to be run, and then run it. The DockerAgent is responsible for executing the Flow. Did you register the Flow with Prefect Server?
o
@Kevin Kho , Yes i did register my flow. - when running without docker, and using local agent, i can see my flow running on the web ui and everything is fine. -Now, i just packaged the same code inside a docker container, ran a DockerAgent, but iam getting that error. I don't know what am i doing wrong!
k
o
@Kevin Kho I will try detail more so bear with me please : • let's say i have this repo structure:
Copy code
|-- code/
|   |-- README.md
|   |-- code/
|   |-- code/
|-- code/
|-- workflow/
|   |-- flow.py/
|-- environment.yml
|-- Dockerfile
|-- requirements.txt
• This is a valid workflow that can be ran using 
workflow/flow.py
• So next step i build the image from that Dockerfile: 
docker build -t test:latest .
Copy code
FROM prefecthq/prefect:0.14.14-python3.7

 ... Run some steps and installations etc etc...

ADD . .

SHELL ["/bin/bash", "-c"]

CMD  ["python", "/webbeds/workflow/flow.py"]
• in my flow.py i have something like this :
Copy code
flow = Flow("Flow name")

flow.executor = LocalDaskExecutor(scheduler="threads")
id=flow.register(project_name="project_name")
Client=Client()
Client.create_flow_run(flow_id=id, run_config=DockerRun(), run_name="run_name", labels=["agent_name"])
• I even tried to pass 
DockerRun(image="test:latest")
• Then I ran : 
prefect agent docker start --show-flow-logs -l agent-name --no-pull
 • Then i ran my image: 
docker run test:latest
• The agent picks up the flow as shown in the screenshot below. • On the web ui i just get the error:
Copy code
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
k
Thanks for the detail. Can I ask why you have 
create_flow_run
in the flow.py? Are you trying to run immediately?
o
Yes i want to run it immediatly, we have multiple members in the teams that would like to launch multiple experiments (ML training basically) so we want to run it immediatly.
k
Have you gotten a simpler flow up an running with Server?
Ok I think this will be simpler if we try to get the flow working first. Can you make a simpler 
flow.py
like:
Copy code
@task
def test_task():
    return "hello"

with Flow("test_flow") as flow:
    test_task()

flow.register("project_name")
We’ll worry about the immediate run later
o
But, That same flow, with LocalRun and local agent without docker runs perfectly.
k
You mean when you run it with 
python flow.py
right?
o
Yes
k
Yes so it’s not exactly comparable. For example, we serialize things when it gets registered. Sometimes running it like that will work but registering won’t because running it locally will not try to serialize. Similarly, file paths can work on local, but sometimes fail when registering and running because the root path may be different.
What we want to do is get a simple flow up registered and then scheduled for a quick run to verify that we can do that. And then we’ll move on to the Docker
o
Ok, i will try to make it simpler now and we can see
k
Just wanna make sure you did 
prefect backend server
right?
o
Yes i did that
k
For this simple one we can use the Local agent. 
prefect agent local start
o
For some reason the agent isn't picking up any flows
k
With the simple 
flow.py
we had? I think this is because the Flows are picked up by Agents with the same labels. Let’s try putting labels on both the Flow and the Agent so the Agent will know it can pick up that flow.
o
My bad sorry, it works fine now.
On the web ui i went to run and ran the scheduled flow and it works fine
k
Nice to hear 🙂. We’ll get there! So just to confirm, You were able to register a Flow in Server. You started up the 
LocalAgent
and it picked up when you hit the “Run Now”?
o
Yes that's it
k
We’ll now move on to Docker. Configure a Docker RunConfig like here. Attach it to your 
Flow()
with 
Flow(run_config = DockerRun(whatever))
o
Done
k
Then do your 
prefect agent docker start
and other stuff you need like this: https://docs.prefect.io/orchestration/agents/docker.html#agent-configuration
Let’s run the Python script again to register
and try running the simple Flow again
o
I got the same error but this time : 
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/oussamalouati/'")
k
I see do you have a custom module you’re installing in the Docker container?
Can I see your RunConfig?
o
custom module? i have some dependencies which are in a requirements.txt that's it
I used DockerRun("my_image")
I used DockerRun() also and got same error
k
Would you be able to share your full Dockerfile?
Hey @Oussama Louati, looks like there is another person with a similar issue as you with a default image. Will try to get you a minimum working example
o
Yes i can share my full dockerfile:
Copy code
FROM prefecthq/prefect:0.14.14-python3.7



ENV PYTHONPATH /app:$PYTHONPATH

RUN apt-get update --fix-missing && \
    apt-get install -y wget bzip2 ca-certificates curl git && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

ENV CONDA_DIR $HOME/miniconda3
RUN wget --quiet <https://repo.anaconda.com/miniconda/Miniconda3-py38_4.8.3-Linux-x86_64.sh> -O ~/miniconda.sh && \
    chmod +x ~/miniconda.sh && \
    ~/miniconda.sh -b -p $CONDA_DIR && \
    rm ~/miniconda.sh

ENV PATH ${CONDA_DIR}/bin:$PATH

ADD environment.yml .

RUN conda env create -f environment.yml && \
    rm -rf $CONDA_DIR/pkgs/* && \
    conda init && \
    echo "conda activate myEnv" >> /root/.bashrc

ENV PATH ${CONDA_DIR}/envs/melqart/bin:$PATH

ADD . .


SHELL ["/bin/bash", "-c"]

CMD  ["python", "/app/workflow/flow.py"]
k
thank you will work on it
o
Also please remember i have this repo structure:
Copy code
|-- components/
|   |-- README.md
|   |-- code/
|   |-- code/
|-- code/
|-- workflow/
|   |-- flow.py/
|-- environment.yml
|-- Dockerfile
|-- requirements.txt
And in my imports, i have imports like : 
from *components*.whatever import whatever
Because i am adding 
/app
to the $PYTHONPATH in the dockerfile
Many thanks , i will be waiting for your reply
k
It might come later, but will definitely get back to you
✅ 1
Hello @Oussama Louati! The reason this was giving an error is because we have a default 
RunConfig
and default 
Storage
that are both Local. 
LocalRun
and 
LocalStorage
. What happens when you register a flow with these is that Prefect saves a serialized form in 
Users/username
, the default LocalStorage location. When Prefect runs a flow, it looks for this serialized version and unpickles it and runs it.
The problem is when we combine 
LocalStorage
with 
DockerRun
. The LocalStorage default is 
Users/username
so Prefect starts looking for that location in the Docker container and trying to find the serialized script to run. This is the error.
The fix here is to configure LocalStorage relative to where that script will live in the Docker container
For you it should be something like: 
Flow("name", storage=Local(path="/app/workflow/flow.py", stored_as_script=True)
. The stored as script is to specify to not pickle it.
I have spoken with the core team and will be creating an issue to have more detailed error messaged around this.
o
@Kevin Kho I thank you so much for your quick reply: but i am still getting this error: 
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/app/workflow/flow'")
k
Can you show be how you define Storage in the Flow?
Oh I see it. One sec
Try 
path="/app/workflow/"
. I think it has to be a directory.
o
flow = Flow("flow name", run_config=DockerRun(), storage=Local(path="/app/workflow/flow.py", stored_as_script=True))
I tried 
path="/app/workflow/"
, same error
k
This is just a matter of changing that path to point to the file in the Docker container.
Ok I’ll really try to make a minimum example for you. Might be over the weekend 🙂
o
That would be very helpful, many thanks. I hope we can make it work.
k
@Oussama Louati, I replicated your setup and recreated your error, then got it working on my end with this setup. Instructions are outlined in the README. The Dockerfile is very similar to yours. I know the first question you had was how to “run on demand”. Ask in the Slack channel once we confirm DockerRun is working, that part should be easier 🙂
o
@Kevin Kho I really thank you for your effort and appreciate what you have done. My previous error is well solved but now i am getting this one : 
Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
😕
k
This might be a mismatch of server version and flow registration Prefect version. Could you confirm your versions?
I also suggest you can try this Prefect Cloud if possible. You get 10000 free task runs with no credit card required so it’s a lot easier to setup.
Your server version should be 0.14.14 or higher
o
We are trying to adopt prefect company wide, and honestly we are running hundreds of ML workloads everyday so hosting it is the right option for now. My server is 0.14.14 and in the dockerfile i am using 
prefecthq/prefect:0.14.14-python3.7
k
Totally understandable if you don’t want to try Cloud. Those versions look good. How about prefect in the local Python environment used to register the flow?
o
It is 0.14.14 also
k
Ok I’ll need to touch base with a teammate about this error cuz I haven’t seen it yet. Will respond to you tom. Just to confirm were you able to get the simpler flow working?
o
Yes simpler flow worked fine
k
Ok perfect. Is it the same Dockerfile you posted above?
I’ll look into it a bit, but it might be hard to replicate error on my own. I may have to ask you for a minimum reproducible example tomorrow to help me pinpoint the issue.
o
Yes the same i just added the ( 
pip install -e .
to install my components). Yes, we see about it tomorrow.
k
Alright. Enjoy the rest of your weekend 🙂
o
Many thanks, you too 🙂
k
I can’t recreate this. I think what is happening is that you’re installing things in the conda environment and not activating it during the flow run. This means the pip install is not working. The easiest thing here is to use pip to install all dependencies because the container provides an isolated environment anyway.
I would likely need a minimum working example to really diagnose this.
RUN pip install -e .
is after the 
ADD . .
right?
Article for troubles with conda in Docker https://pythonspeed.com/articles/activate-conda-dockerfile/
o
Now it works for me , many thanks and apologize for all this inconvenience. I think just one question that i might ask is how can i run the flow the flow immediately from my script !
k
Perfect! Can you tell me how you fixed it?
o
I just got rid of miniconda also on my local machine i had python 3.7 and on the docker file i used 3.8. So i changed both to 3.8.
k
I see. I will answer you on the immediate flow run. Just confirming with the team
So we are working on a simpler way to do something like 
flow.run(with_backend="cloud")
, but in the mean time, the best way to run a flow instantly is using the GraphQL API. There is a 
create_flow_run
mutation and you can use.
Oh you already had this in the original Client code. That should work I think
So register and then use the 
create_flow_run
o
yes i have :
Copy code
flow = Flow(FLOW_NAME,storage=Local(path="/app/workflowflow.py",stored_as_script=True), run_config=DockerRun(image=IMAGE_NAME, labels=[AGENT_NAME]))
    .... .....
    .... .....
flow.executor = LocalDaskExecutor(scheduler=EXECUTOR_SCHEDULER)
id= flow.register(project_name= PROJECT_NAME)
Client=Client()
Client.create_flow_run(flow_id=id, run_name= RUN_NAME, labels=[AGENT_NAME])
When run it gives me : 
Failed to load and execute Flow's environment: ValueError('One of flow_id or version_group_id must be provided')
k
This worked for me. I don’t think it’s related but maybe you should use lower case for 
client=Client()
. Try also printing your 
id
to see if that was successful.
Copy code
id = flow.register('tutorial')

from prefect import Client
client=Client()
client.create_flow_run(flow_id=id, run_name="test")
2 Views
Open in Slack
PreviousNext
Powered by