https://prefect.io logo
d

david

07/12/2023, 7:50 PM
@Marvin Hey Marvin, I have a docker image that I want to use when I run my flows in k8s environment. The image is in artifact registry and not dockerhub. Is there a way to spin up a container based on that image where I can run my flow code? For context, my agent is successfully scheduling jobs in my k8s env but the execution is failing on some required dependencies. Hence why I want to use my pre built/vetted docker image for the execution environment. Thanks!
m

Marvin

07/12/2023, 7:51 PM
Ah, the sweet scent of documentation. It's like a map in the wilderness of software. Here's what I found: Prefect agents rely on Docker images for executing flow runs using DockerContainer or KubernetesJob infrastructure. If you do not specify an image, Prefect will use a default image tag that matches your local Prefect and Python versions. However, if you're building your own image, you may find it useful to use one of the Prefect images as a base. You can find more details in the Prefect documentation. If the image is stored anywhere other than a public Docker Hub registry, you should use a corresponding registry block. Prefect automatically sets a Docker image matching the Python and Prefect version you're using at deployment time. You can see all available images at Docker Hub. More details can be found here. For KubernetesJob,
kubectl
must be available and you must configure remote Storage. In terms of deployment, you can use the Prefect Helm chart to deploy your Prefect worker in your Kubernetes cluster. You can find a step-by-step guide in the Prefect documentation. Remember, the key to a successful software project is like the key to a successful joke: timing. And in your case, the right Docker image.
n

Nate

07/12/2023, 8:09 PM
hey @david - I think this would just be a matter of setting the
image
field on your KubernetesJob block used for a given deployment however if you're using work pools, you'd set the image there (and override it by deployment if you need to)
🙌 1
d

david

07/12/2023, 8:20 PM
Thank you @Nate!
When I try to run a job - execution fails on the set_working_directory step where prefect is trying to call a directory that exists in the local environment where I registered the flow from, but not in container whose image is specified in the KubernetesJob block. 1. Is the flow code executing in the image? 2. if so, why is local directory set as the working directory in the container executing the flow? 3. Is there a way to specify a path that exists in the container when deploying the flow locally?