Hi, has anyone experienced the following error tha...
# prefect-communityp
Parwez Noori
08/30/2022, 11:20 AMHi, has anyone experienced the following error that the Kubernetes Jobs cannot pull an public image. It gives the following error: Inserted in the thread.
✅ 1
Parwez Noori
08/30/2022, 11:20 AM Copy code
Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/library/nginx:latest|docker.io/library/nginx:latest>": failed to resolve reference "<http://docker.io/library/nginx:latest|docker.io/library/nginx:latest>": failed to do request: Head "<https://registry-1.docker.io/v2/library/nginx/manifests/latest>": dial tcp: lookup <http://registry-1.docker.io|registry-1.docker.io> on [::1]:53: read udp [::1]:38431->[::1]:53: read: connection refusedParwez Noori
08/30/2022, 1:49 PMIs it possible for the kubernetes cluster to fetch the image locally?
c
Christopher Boyd
08/30/2022, 2:58 PMHi Parwez, this seems like a network / dns issue with your cluster
Christopher Boyd
08/30/2022, 2:59 PMread connection refused [:1]53 seems like perhaps an ipv6 network connection that’s failing a dns lookup
Christopher Boyd
08/30/2022, 3:00 PMYou could try to curl docker.io just from some shell in one of your pods to confirm
Christopher Boyd
08/30/2022, 3:01 PMregarding fetching your image locally , you can setup your own container / image repository - I believe every cloud has their own version, or you can host one internally if you’re using on-prem
Christopher Boyd
08/30/2022, 3:02 PMWhat image tag are you using in your deployment file for your kubernetes jobs ?
p
Parwez Noori
08/30/2022, 5:13 PMHi @Christopher Boyd,
Thanks a lot for the answer! I'm able to curl docker.io and google.com.
I have solved it by setting the image pull policy to 'Never' to make it pull the image locally. This works for now, but will not when I try to setup CI/CD.
I'm using Azure but I'm having issues with adding the container registry to the subnet. Do you have any guidelines or recommendations? 🙌
c
Christopher Boyd
08/30/2022, 5:25 PMhttps://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-azure-cli - I’d just set up your container registry in the same resource group if you have control over it
Christopher Boyd
08/30/2022, 5:25 PMChristopher Boyd
08/30/2022, 5:26 PMif your image pull policy is never, it’s not “pulling” the image, it’s just using your local cache?
p
Parwez Noori
08/30/2022, 5:36 PMYes, I'm able to move into the same resource group as the Kubernetes Service! I'm currently moving it.
Thanks Christopher! 🙌
Ahh! makes much better sense, because I don't have locally currently, but it still fetches the image.
👋 1
Parwez Noori
08/30/2022, 5:41 PMNow I get this issue _____<http://.azurecr.io/v2/prefect2/manifests/latest|.>azurecr.io/v2/prefect2/manifests/latest": dial tcp: lookup acraros.azurecr.io on [:1]53: read udp [:1]45437->[:1]53: read: connection refused.
Does it maybe takes some time before it works?
Parwez Noori
08/30/2022, 5:48 PMI have also done this:
By default, the allow trusted services setting is enabled in a new Azure container registry.
To disable or re-enable the setting in the portal:
1. In the portal, navigate to your container registry.
2. Under Settings, select Networking.
3. In Allow public network access, select Selected networks or Disabled.
4. Do one of the following:
◦ To disable access by trusted services, under Firewall exception, uncheck Allow trusted Microsoft services to access this container registry.
◦ To allow trusted services, under Firewall exception, check Allow trusted Microsoft services to access this container registry.
5. Select Save.
Parwez Noori
08/31/2022, 9:27 AM@Christopher Boyd just wanted to mentioned that it has been resolved. Once again thank you for your help!! 🙌
🙌 1
c
Christopher Boyd
08/31/2022, 12:29 PMGreat to hear Parwez, happy coding!