Anyone using editable / source dependencies in your flows?

Kind of. You mean like a module that is always changing right? I have seen someone git clone and pip install using the ENTRYPOINT of their Dockerfile

it works from the dockerfile, but there's a check when building the 'docker' storage

Ah you can defer the import by putting it inside a task

Ah no. If you put the editable library inside the Dockerfile it should work. Do you have the library installed locally?

root: pipfile,

cd infer/prefect_utils && pip install -e .
cd ../..
pipenv install

dockerfile:

FROM prefecthq/prefect:0.15.4-python3.8

RUN pip install --upgrade pip setuptools wheel twine \
    && pip install pipenv \
    && apt-get update \
    && apt-get install -y --no-install-recommends curl gcc python3-dev libssl-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /opt/prefect

COPY Pipfile* packages.yml profiles.yml .user.yml .python-version dbt_project.yml postinstall.py ./
COPY infer ./infer
COPY dbt ./dbt

# <https://docs.pipenv.org/en/latest/advanced.html#managing-system-dependencies>
# or else /opt/prefect/healthcheck.py:152: UserWarning: Flow uses module which is not importable.
# Refer to documentation on how to import custom modules
# <https://docs.prefect.io/api/latest/storage.html#docker>
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy
ENV PATH="/opt/prefect/.venv/bin:$PATH"

RUN python postinstall.py

COPY flows /opt/prefect/flows

Wait, if you want to make your own Docker image with the dependencies, you can do that and just run the

docker build

yourself. And then on the Flow side, just use

DockerRun

with that image you build. Do you want the flow inside that image or outside like S3 or Github?

Inside the image. What's DockerRun?

The RunConfiguration for the Flow so that when the flow starts, it will load an image and run the flow on top of the image.

But I still do

flows = sorted(Path("flows").glob("*.py"))


# Add flows
for file in flows:
    flow = extract_flow_from_file(file_path=file)
    <http://logger.info|logger.info>("Extracted flow from file before build")

    docker.add_flow(flow)

But I still need to do

flow = extract_flow_from_file(file_path=file)

otherwise I can't assign the storage to the flow, right?

I think you can pair

Local Storage

with

KubernetesRun

and if the image is already created and the flow is already inside, you just point your storage like `Local(path=...,, stored_as_script=True)`and then the flow will be relative to the image.

But IMO KubernetesRun is not configured here, it's configured using the cloud k8s agent

No, but I specify the JobTemplate and such

Yeah you're right. Will look deeper at this tom and see what causes the failing healthcheck. That's when building the Docker storage right?

Yes

Gotcha ok will try to replicate that failed healthcheck with docker storage tom

But really, what is DockerRun? I'm not running anything on Docker; it's running on GKE with containerd / runc on COS

DockerRun pairs with the Docker agent and then you specify an image so that the Flow will be loaded and run in that container. For example you can load a flow from S3 and then run it on an image you specify. So you can pack all the dependencies if they don't change frequently and then just run flows on top of it.

But can the k8s agent just pull an image then?

You can. You're doing it that way currently right?

and if the k8s / cloud agent pulls an image, is it still dockerrun?

It is not DockerRun. Ignore the DockerRun if you are on Kubernetes. The Kubernetes agent combines it's job template with the KubernetesRun RunConfig when deploying a flow so you can indeed just set it on the agent side

Yes you need VSCode

Sure

Hmm, no huddle / video buttons?

Oh I thought you meant paste the code. I'm about to sleep 😅

😄

Lol. Leave a message though. I'll look into it tom

You've helped me enough 🙂