Any idea what to make of this `v2` My EKS agent picked up a

Question

Any idea what to make of this `v2` My EKS agent picked up a flow off of a queue but the pod never started ```prefect deployment build main py foo n sean k8s test deployment q company name sb s3 company name prefect staging sean turner foo ib kubernetes job test apply``` EKS agent logs ```16 09 45 420 | INFO | prefect agent Submitting flow run c8f0bba7 b104 40d2 aa96 8a278c800f1e 16 09 45 766 | INFO | prefect agent Completed submission of flow run c8f0bba7 b104 40d2 aa96 8a278c800f1e 16 10 45 785 | ERROR | prefect infrastructure kubernetes job Job test6qzkh Pod never started ``` My `kubernetes job test` infra block has ` EXTRA PIP PACKAGE s3fs ` and does not have a kubeconfig set My S3 infra block `s3 company name prefect staging sean turner foo` does not have credentials because the agent and orion pods are assuming IAM roles that give permissions to read and write to the bucket

Zanie · Answer

It s possible the pod timeout is just too low

Zanie · Answer

`pod watch timeout seconds` defaults to 60 but can be set higher

Sean Turner · Answer

Hmm just upped it but I don t feel like that should be the issue Is there anywhere I can get additional logs I also added the attached heavily redacted kubeconfig and updated the environment variable to include the awscli ` EXTRA PIP PACKAGES s3fs awscli `

Sean Turner · Answer

Maybe I need to update my EKS `aws auth` configmap to allow the Agent IRSA IAM role I m kind of grasping at straws I don t think this sort of thing is documented anywhere

Sean Turner · Answer

Updated the EKS aws auth and experiencing the same issue Is there a resource anywhere about setting the kubeconfig

Zanie · Answer

What does `kubectl describe job lt job name gt ` get you

Zanie · Answer

You can also get all pods for the job `kubectl get pods selector=job name= lt job name gt `

Sean Turner · Answer

Ahhh interesting I didn t realize that the jobs were actually being created

Sean Turner · Answer

```Events Type Reason Age From Message Warning FailedCreate 5m29s x9 over 27m job controller Error creating pods testkbhpn is forbidden error looking up service account prefect prefect serviceaccount prefect not found```

Sean Turner · Answer

Okay huge progress I m unblocked Thank you so much

Zanie · Answer

thumbsup I believe there s an issue to report these errors via the agent as well