Sean Turner
09/23/2022, 4:21 PMv2
)? My EKS agent picked up a flow off of a queue but the pod never started?
prefect deployment build main.py:foo \
-n sean-k8s-test-deployment \
-q company-name \
-sb s3/company-name-prefect-staging/sean-turner/foo \
-ib kubernetes-job/test \
--apply
EKS agent logs:
16:09:45.420 | INFO | prefect.agent - Submitting flow run 'c8f0bba7-b104-40d2-aa96-8a278c800f1e'
16:09:45.766 | INFO | prefect.agent - Completed submission of flow run 'c8f0bba7-b104-40d2-aa96-8a278c800f1e'
16:10:45.785 | ERROR | prefect.infrastructure.kubernetes-job - Job 'test6qzkh': Pod never started.
My kubernetes-job/test
infra block has {"EXTRA_PIP_PACKAGE": "s3fs"}
and does not have a kubeconfig set.
My S3 infra block s3/company-name-prefect-staging/sean-turner/foo
does not have credentials because the agent and orion pods are assuming IAM roles that give permissions to read and write to the bucket.Zanie
09/23/2022, 5:00 PMpod_watch_timeout_seconds
defaults to 60 but can be set higherSean Turner
09/23/2022, 5:06 PM{"EXTRA_PIP_PACKAGES": "s3fs,awscli"}
aws-auth
configmap to allow the Agent IRSA IAM role? I'm kind of grasping at straws, I don't think this sort of thing is documented anywhereZanie
09/23/2022, 5:28 PMkubectl describe job <job-name>
get you?kubectl get pods --selector=job-name=<job-name>
Sean Turner
09/23/2022, 5:31 PMEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 5m29s (x9 over 27m) job-controller Error creating: pods "testkbhpn-" is forbidden: error looking up service account prefect/prefect: serviceaccount "prefect" not found
Zanie
09/23/2022, 5:54 PM