Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey everyone,
has somebody tried to run a custom docker image as a part of the flow?
i'm using k8s workers, and want to start a pod with a custom image as a part of pipeline

Hey Validslav, <https://docs.prefect.io/v3/deploy/infrastructure-examples/docker#automatically-build-a-custom-docker-image-with-a-local-dockerfile|yes you can>!

Just to clarify the pattern you're looking to achieve:
Are you trying to have a parent flow that is running in one pod kick off a new pod with a custom image to run a child/downstream flow?

Hi, thanks, basically i'm looking for something similar to airflows `GKEStartPodOperator`  or `KubernetesPodOperator` .
Using k8s worker i'd like to run a pod with custom image(non prefect/python), but i'd like prefect to monitor this pod for a status(successful/failed run, running, etc)

&gt; Are you trying to have a parent flow that is running in one pod kick off a new pod with a custom image to run a child/downstream flow?
yes, running a new pod with a custom image as a part of the flow

<@U081C9SGT1C> Yes, you can. Build your custom image from the prefect one like so:
```FROM prefecthq/prefect:2.14.11-python3.10

# Do something custom
COPY requirements.txt .
RUN pip install -r requirements.txt```
Next, point your k8s worker to pull this image. Now any new pods using this worker will communicate normally with the prefect API and appear in the prefect UI as usual

<@U06KP9C16N7> thank you, so basically the image should be something based on prefect?
I have a binary with a complex business logic that i wanna run as a part of pipeline, and i want mage to spawn a pod with a container with this binary and watch the pod until it finishes with status 0(or error, which should stop pipeline right away)
i definitely can run it in a prefect container, but it will be adding a little bit of overhead

Yes, it has to be based on Prefect client. You can find how it's built <https://github.com/PrefectHQ/prefect/blob/d64c0d777cd4f609a2ce08c55be29b8b961c6564/Dockerfile#L128|here> (if you want to build it from scratch yourself).

> I have a binary with a complex business logic that i wanna run as a part of pipeline, and i want mage to spawn a pod with a container with this binary and watch the pod until it finishes with status 0
If you are using Kuberentes, you can define it as a Kubernetes job. Ultimately if you are using any container orchestrator I am sure there's a mechanism to watch for jobs.

&gt; If you are using Kuberentes, you can define it as a Kubernetes job. Ultimately if you are using any container orchestrator I am sure there's a mechanism to watch for jobs.
yeah, but i have to first prepare the data, and after processing i have another steps, so i prefer orchestrating it using a single solution, to have a better observability and control over the whole pipeline

I see. Yeah Prefect/Airflow/Dagster might be better suited for that. Good luck