Sometimes our jobs fail because we get a 403 from our container image registry. I checked the k8s config of the jobs that failed and I notice that there is no imagePullSecrets in the config which is why it's unable to download the image. This happens maybe 1% of the time.