Hey there 🙂 I am currently struggling with running a flow as a job on k8s. • I have deployed prefect using helm charts for both prefect-server and prefect-worker on an on-prem cluster • I have a healthy Kubernetes Workpool configured that picks up my flow-runs • My flow is backed into a docker image that resides in a private registry (gitlab) which the job i am trying to execute should pull from • When running the flow, I get

ErrImagePull

followed by

ImagePullBackOff

within the created job in k8s When checking the events of the job using

kubectl -n prefect get events | grep <job_name>

I can see that there is an issue with pulling the image from the registry.

8m19s       Normal    Scheduled          pod/teal-harrier-n6h8b-w6dc6      Successfully assigned prefect/teal-harrier-n6h8b-w6dc6 to data-cx31-pool-workload-worker1
6m47s       Normal    Pulling            pod/teal-harrier-n6h8b-w6dc6      Pulling image "<registry-url>"
6m46s       Warning   Failed             pod/teal-harrier-n6h8b-w6dc6      Failed to pull image "<registry-url>": rpc error: code = Unknown desc = failed to pull and unpack image "<registry-url>": failed to resolve reference "<registry-url>": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
6m46s       Warning   Failed             pod/teal-harrier-n6h8b-w6dc6      Error: ErrImagePull
3m19s       Normal    BackOff            pod/teal-harrier-n6h8b-w6dc6      Back-off pulling image "<registry-url>"
6m35s       Warning   Failed             pod/teal-harrier-n6h8b-w6dc6      Error: ImagePullBackOff
8m20s       Normal    SuccessfulCreate   job/teal-harrier-n6h8b            Created pod: teal-harrier-n6h8b-w6dc6

I know these error messages from other projects and this is somewhat telling me that the flow can’t pull the image from my private registry. Is there any particular place I need to configure a pullSecret so that my job can actually pull the image it is supposed to run? Any hints are appreciated 🙏

In the prefect UI, go to your work pool -> three dots -> edit, then select the advanced tab. Scroll down to the part of the JSON that has the job manifest, and add an

"imagePullSecrets":

and whatever you need to provide to it to the inner

spec

like so? I tried running the flow after adding this piece in the job manifest but it does not yet work.

looks like you're missing an "s" at the end of "Secrets"

Got a new Error 😄 Is there any documentation on the correct syntax here? I suppose that valid json should be sufficient but i am getting the following error in the flows logs now:

Job in version \"v1\" cannot be handled as a Job: json: cannot unmarshal object into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type []v1.LocalObjectReference

This question here made me think that syntax might be wrong even though i double checked that the json is valid https://stackoverflow.com/questions/54221964/kubernetes-job-in-version-v1-cannot-be-handled-as-a-job

ah it needs to be an array of objects

@Kevin Grismore worked! I managed to pull the image, now stuck on something else. I’ll switch off for today and check things tomorrow. Thanks for your support on this 🙏