Pierre Monico
08/19/2021, 3:15 PMGit
storage class source, am I correct to assume that it’s not possible to pull from a local repository?Kevin Kho
Pierre Monico
08/19/2021, 3:17 PMKevin Kho
Pierre Monico
08/19/2021, 3:22 PMKevin Kho
Pierre Monico
08/19/2021, 3:54 PMModule
but I get an ImportError
. I am unclear if the Module
storage is accessed by the context in which the Agent lives in, or inside the run_config (Docker
) in my case.Pierre Monico
08/19/2021, 3:55 PMflows
package available to the Docker image of my run config, or just to my agent (running in a pipenv
virtualenv in my case)Kevin Kho
Pierre Monico
08/19/2021, 4:00 PMimport flows
it works though…Kevin Kho
Pierre Monico
08/19/2021, 4:04 PMModule
storage only matters for the agent?Pierre Monico
08/19/2021, 4:04 PMFROM python:3.7-slim-buster
WORKDIR /app
COPY . .
RUN pip3 install pipenv
RUN pipenv lock --keep-outdated --requirements > requirements.txt
RUN apt-get update
RUN apt-get -y install libpq-dev gcc
RUN pip3 install -r requirements.txt
Kevin Kho
Module
storage, you need the setup.py
and pip install -e .
to install the module as a library in the container.Pierre Monico
08/19/2021, 4:11 PMModule
storage in conjuction with the Docker
agent and run config you’re telling the agent: spin up that container, and then pull the flow from a module within that container?Kevin Kho
Local
storage. It’s pulling from the local storage relative to the Docker containerPierre Monico
08/19/2021, 4:13 PMDockerStorage
then? Isn’t that the exact same thing?
• With this approach, you’d have to rebuild you Docker Image every time a flow changes, which I thought was the exact opposite of what Module
is trying to achieve?Kevin Kho
Pierre Monico
08/19/2021, 4:15 PMLocal
is incompatible with the Docker
agent, according to this page)Pierre Monico
08/19/2021, 4:16 PMPierre Monico
08/19/2021, 4:22 PMKevin Kho
LocalRun
is incompatible with Docker
agent but Local Storage
can be used with it.Pierre Monico
08/19/2021, 4:29 PMStorage
option for all flows. Flows using local storage are stored as files in the local filesystem. This means they can only be run by a local agent running on the same machine.`Kevin Kho
Pierre Monico
08/19/2021, 4:32 PMPierre Monico
08/19/2021, 4:34 PMModule
doc and it’s pretty explicit:
you can useSo I guess I just have to make it available in the Docker image and it should all work 🙂storage to reference and load them at execution time (provided the module is installed and importable in the execution environment)Module
Kevin Kho
Local
storage with DockerRun so I think that docs is fine. I see, I was asking because if it’s all on the same box you could use Local
everything. But yeah, I think you might need to use Docker. Where do you store the Docker Hub.Pierre Monico
08/19/2021, 4:39 PMPierre Monico
08/19/2021, 4:53 PMENV PYTHONPATH "${PYTHONPATH}:/app/flows"
😍Pierre Monico
08/19/2021, 4:56 PMModule
to just point to the correct module within the flows
package. Execution is done with Docker Agent and Docker run using that image.
It’s nice because everything can live in the same repo and the only thing my pipeline needs to do is build this simple Docker image and push it to the registry.
I am very happy 🙂Pierre Monico
08/19/2021, 4:56 PMKevin Kho
Zanie
Pierre Monico
08/20/2021, 6:39 AMmyproject
- mypackage1
-- mymodule.py
- mypackage2
- flows
-- flow1.py
-- flow2.py
I use Module
storage and DockerRun
run config. This was raising an ImportError
for the flows
package, so I’ve added it to the PYTHONPATH
of my Docker image. Now it all works, but I am wondering about one thing: in my flow files I import modules from e.g. package1
and it simply works. So basically I am wondering how Module
storage resolves all of this.
• For the flows (Module("flows.flow1")
) you need to add the package to the PYTHONPATH
as if the mechanism was not inside WORKINGDIR
of the Docker image, otherwise it won’t find the flows
package (ImportError
)
• But when actually running flows that rely on importing packages that live in WORKINGDIR
, the mechanism seems to be able to resolve them just fine as if executed from WORKINGDIR
Don’t get me wrong, it all works now - I just found the documentation to not be very clear on this and I am wondering how it works exactly. Tried reading the source but it was a bit too involved/layered for me to understand within a few minutes.