https://prefect.io logo
t

Tomas Thornquist

06/15/2020, 7:23 AM
We're having major issues with imagepullsecrets occasionally not being added to the k8s jobs causing the jobs to fail. Since this only happens occasionally it's very difficult to debug.
j

josh

06/15/2020, 11:48 AM
Hey @Tomas Thornquist can you provide more context on this?
t

Tomas Thornquist

06/16/2020, 1:45 AM
Sometimes our jobs fail because we get a 403 from our container image registry. I checked the k8s config of the jobs that failed and I notice that there is no imagePullSecrets in the config which is why it's unable to download the image. This happens maybe 1% of the time.
j

josh

06/16/2020, 12:39 PM
Interesting that it is happening non-deterministically! Could you open an issue for this?