https://prefect.io logo
Title
r

Romain

05/20/2021, 2:01 PM
Hi folks, Have an issue with running a flow using a Kubernetes Agent, and a Module storage. So here is what I do. The agent docker image is based on prefecthq/prefect:0.14.19, and in my dockerfile, I have the following folder hierarchy:
/A/
  B/
     flows/
         my_flow.py
in my_flow.py, imagine something like that:
def get_flow():
    with Flow('my_flow', storage=Module("B.flows.my_flow:get_flow")) as flow:
        ....
    return flow
In the dockerfile, I ensure that the
PYTHONPATH
env var holds the folder
A
so that I can import
<http://B.flows.my|B.flows.my>_flow
setting this at the end of the DockerFile:
ENV PYTHONPATH "${PYTHONPATH}:/A"
After that I register this flow with a
KubernetesRun
and a `DaskExecutor`:
flow.run_config = KubernetesRun()
flow.executor = DaskExecutor()
flow.register(project_name='my_project',
              idempotency_key=flow.serialized_hash())
Then from a prefect server, I trigger the flow run, but I get the following error:
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'A'")
I don't really get why my module is not found. I have tested a local deployment using docker-compose with a local agent (running in the compose stack), and it was working fine. So I am missing something here. Any ideas?
Shall the module be also available on the prefect server apollo pod?
k

Kevin Kho

05/20/2021, 2:04 PM
Hi @Romain! Yes you need it installed and available in the execution location.
r

Romain

05/20/2021, 3:24 PM
But I thought that in prefect , the server was somewhat agnostic of the execution code. And only the agent was required to know the code. How would it work if I would be using prefect cloud and a Module storage then?
k

Kevin Kho

05/20/2021, 4:04 PM
With Docker storage, the container is downloaded and executed on the agent. For Module storage, the module would need to exist on the execution side also because it doesn’t download it from a location. This is true for Cloud as well.