https://prefect.io logo
Title
t

Tom Tom

08/25/2021, 11:46 PM
Hey everyone, on my local agent everything works fine. But if i want to use 
DockerRun(image="prefecthq/prefect")
  with
prefect agent docker start
 i´m getting this Error:
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'C'")
I know that i can´t import other dependencies but i only import following modules from prefect in the script:
from prefect import Flow, task
from prefect.run_configs import LocalRun,DockerRun
Or is it possible to run 
DockerRun()
 in an existing Container?
n

nicholas

08/25/2021, 11:55 PM
Hi @Tom Tom - any modules/packages needed to run your flows in a container will either need to be pip-installable by the container or manually placed there during the build process. You can accomplish this by either publishing modules or creating a custom image to use for your flows
🙌 1
t

Tom Tom

08/26/2021, 12:11 AM
Ah ok, i thought prefect is already installed in image
"prefecthq/prefect"
. I´ll build a custom image on base on
"prefecthq/prefect"
with the prefect dependencies, hope it works 🙂
n

nicholas

08/26/2021, 12:12 AM
Oh I'm sorry I may have misunderstood - the prefect image should have prefect dependencies installed
The "No module named 'C'" doesn't seem like it's coming from a prefect import though, that looks like something else
t

Tom Tom

08/26/2021, 12:23 AM
Oh, ok, well that´s my python script for testing:
from prefect import Flow, task
from prefect.run_configs import LocalRun,DockerRun

#classes
class eins:
    @task
    def function():
        return 1

Test=eins()

#Flow
with Flow("Prefect Test Flow") as flow:
    a = Test.function()

#flow.run_config = LocalRun()
flow.run_config = DockerRun(image="new")
flow.register(project_name="test")
I start my server on CMD with
prefect server start
and my docker agent also on CMD with
prefect agent docker start --label Andromeda
. Andromeda is my PC-Name, if I don´t define it, the agent will not run anything and the tasks will be "late" but not executing.
k

Kevin Kho

08/26/2021, 12:50 AM
Hey, I think what is happening is that your flow is using LocalStorage by default, which writes the Flow in
~/.prefect
by default when you register. I think you are on a Windows machine so your home evaluates to
C:/users/something
. And then when you use
DockerRun
it imports your flow from
LocalStorage
and looks for it under the
~.prefect
folder. But you don’t have
C:/users/something
.
In order to use DockerRun, you need a LocalStorage explicitly defined that shows the path of the flow relative to you Docker container. And then you need to add this flow to the Docker container.
Flow("docker_example", storage=Local(path="/app/workflow/flow.py",stored_as_script=True)
will mean that you copied the Flow with filename
flow.py
to
/app/workflow/
I have an example of the setup here , but this is a full Python module that I brought over to the container.
👀 1
:upvote: 1
t

Tom Tom

08/26/2021, 1:41 AM
Hey @Kevin Kho , thank you very much for your help. I´ll tried several path combinations but no success, i´ll try it tomorrow again and will reply, maybe i´m just to tired at 3:40 AM 😅
😴 1
So i tried at least 20 different paths, but no success, it returns every time
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named"test.py"
. I use Docker Desktop on Windows and so my Docker Container should be stored in
C:\Users\Tom\AppData\Local\Docker\wsl\data
. I used different paths like:
/test.py
/Desktop/test.py
../../../../../Desktop/test.py
/var/lib/docker/test.py
I run my agent and server from
C:\Users\Tom
k

Kevin Kho

08/26/2021, 7:38 PM
Can I see your Dockerfile and your flow?
t

Tom Tom

08/26/2021, 7:41 PM
I just use
run_config=DockerRun(image="prefecthq/prefect")
, so i use the prefect standard docker image and i dont have any Dockerfile(i think?).
with Flow("Prefect_Test_Flow",storage=Local(path="/Desktop/test.py",stored_as_script=True), run_config=DockerRun(image="prefecthq/prefect")) as flow:
    a = Test.function()
k

Kevin Kho

08/26/2021, 7:46 PM
The
Local(path="Desktop/test.py")
will look for that path inside the container so you need to get the file inside the container somehow.
t

Tom Tom

08/26/2021, 7:49 PM
creates
run_config=DockerRun(image="prefecthq/prefect")
a container? because at the moment no container is running.
k

Kevin Kho

08/26/2021, 7:52 PM
The docker agent will pull
prefecthq/prefect
and then look for the local path (relative to the container), and then load the flow and run it on top. This path thing is trickier with
Local
but with something like
GitHub
or
S3
you don’t need to worry about that script being stored in the image since it will be downloaded during runtime.
Local
just needs it to be in the local directory. If you are using
DockerRun
, it’s a path in the container. If you are using
LocalRun
, it’s a path on the host machine.
t

Tom Tom

08/26/2021, 8:07 PM
Ah ok i can follow you, e.g. with
docker cp <src-path> <container>:<dest-path>
  i need to copy the file into the container, so the agent find/load and can run the flows. But this is tricky cause during runtime ofc i cant use comands, maybe i can start a container manually, copy file, and then run the flow with
DockerRun()
and pass them the container ID?
k

Kevin Kho

08/26/2021, 8:13 PM
I think you would need to make a new image because I think Prefect starts a container as it calls
dockerpy's
create_container
so I don’t believe you can just pass a container to DockerRun. DockerRun also does not take a container id as an argument.
🙌 1
:upvote: 1
t

Tom Tom

08/26/2021, 8:14 PM
ok, great that was my second idea to create an own image based on prefecthq/prefect. I´ll try that out 😃
OH MY GOD
[26 August 2021 10:48pm]: All reference tasks succeeded.
You guys are awesome, thank you very much for your help 🎉 @Kevin Kho @nicholas To other ppl with the same problem, Dockerfile looks like:
FROM prefecthq/prefect:latest
WORKDIR /app
COPY test.py /app/test.py
And the flow/DockerRun is the same as above, with new image ID and
path="/app/test.py"
👍
👍 1
k

Kevin Kho

08/26/2021, 8:50 PM
Nice work!
t

Tom Tom

08/26/2021, 9:01 PM
ok, one additional question. If i add 2 or more .py files in the dockerfile, can i use in
DockerRun()
more than one path/file? Or 1 flow 1 file?
k

Kevin Kho

08/26/2021, 9:18 PM
So 1 flow is limited to one file. That place the flow is stored is a
Storage
concept rather than the
RunConfiguration
, but yes you can include multiple files in the container. Check this for more information on Docker storage. If those are files you need to import, it’s best to install as a Python module. If those are files you read in like
.sql
or
.yaml
then just including them in the container for your Flow to use will work.
🎉 1
t

Tom Tom

08/26/2021, 9:22 PM
U´re the best, thank you very much and have a good day 👏
👍 1