Hello - Im getting the following error when trying to pull image from the azure registry. My setup is • K8s self hosted server • K8s hosted worker • prefect-server is the service account which is being used • K8s Job has been supplied with image pull secrets job_variables

job_variables:
      image: soaapreg.azurecr.io/sentinel-flow
      imagePullSecrets:
        - name: acr-secret

Worker 'KubernetesWorker a59877db-07ff-4ba4-abe2-36c607de7349' submitting flow run 'fb8f9a8f-415b-4d46-b447-393e335479de'
12:06:16 PM
prefect.flow_runs.worker
INFO
Creating Kubernetes job...
12:06:16 PM
prefect.flow_runs.worker
INFO
Completed submission of flow run 'fb8f9a8f-415b-4d46-b447-393e335479de'
12:06:17 PM
prefect.flow_runs.worker
INFO
Job 'psi3-edasich-x-z58jm': Pod has status 'Pending'.
12:06:17 PM
prefect.flow_runs.worker
ERROR
Job 'psi3-edasich-x-z58jm': Pod never started.
12:07:17 PM
prefect.flow_runs.worker
INFO
Job event 'SuccessfulCreate' at 2024-07-05 12:06:17+00:00: Created pod: psi3-edasich-x-z58jm-68rkt
12:07:17 PM
prefect.flow_runs.worker
INFO
Pod event 'Scheduled' at 2024-07-05 12:06:17.100731+00:00: Successfully assigned default/psi3-edasich-x-z58jm-68rkt to aks-soaap1z3514a-11953152-vmss000001
12:07:17 PM
prefect.flow_runs.worker
INFO
Pod event 'Pulling' (3 times) at 2024-07-05 12:06:57+00:00: Pulling image "soaapreg.azurecr.io/sentinel-flow:latest"
12:07:17 PM
prefect.flow_runs.worker
INFO
Pod event 'Failed' (3 times) at 2024-07-05 12:06:57+00:00: Failed to pull image "soaapreg.azurecr.io/sentinel-flow:latest": failed to pull and unpack image "soaapreg.azurecr.io/sentinel-flow:latest": failed to resolve reference "soaapreg.azurecr.io/sentinel-flow:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to <https://soaapreg.azurecr.io/oauth2/token?scope=repository%3Asentinel-flow%3Apull&service=soaapreg.azurecr.io>: 401 Unauthorized
12:07:17 PM
prefect.flow_runs.worker
INFO
Pod event 'Failed' (3 times) at 2024-07-05 12:06:57+00:00: Error: ErrImagePull
12:07:17 PM
prefect.flow_runs.worker
INFO
Pod event 'BackOff' (3 times) at 2024-07-05 12:07:11+00:00: Back-off pulling image "soaapreg.azurecr.io/sentinel-flow:latest"
12:07:17 PM
prefect.flow_runs.worker
INFO
Pod event 'Failed' (3 times) at 2024-07-05 12:07:11+00:00: Error: ImagePullBackOff
12:07:17 PM
prefect.flow_runs.worker
INFO
Reported flow run 'fb8f9a8f-415b-4d46-b447-393e335479de' as crashed: Flow run infrastructure exited with non-zero status code -1.

Hi Parash, it looks like the Kubernetes job can’t pull the image from the Azure Container Registry (ACR) due to an authorization issue. The

401 Unauthorized

error suggests that the credentials for accessing the ACR are either wrong or not being applied correctly. Make sure the image pull secret (

acr-secret

) is properly created and referenced in your Kubernetes job. Check the secret with:

kubectl get secret acr-secret -o yaml

Also you can double check that the

prefect-server

service account has the right permissions to use the image pull secret. You can bind the secret to the service account like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prefect-server
  namespace: default
secrets:
- name: acr-secret

Thats correct Taylor, thank you for your response. I was earlier trying to use the "prefect-server" service account which was automatically created while prefect server installation. Although I tried passing the secret to K8s job using this account, it was failing. To resolve what I did is 1. Created a new service account 2. Using this service account, created a new worker 3. Then patched the service account with credentials 4. Passed the credentials to the job this made the trick. Thank you again