https://prefect.io logo
Join Slack
Powered by
Any idea what to make of this (`v2`)? My EKS agent...
# ask-community
s
Any idea what to make of this (
v2
)? My EKS agent picked up a flow off of a queue but the pod never started?
Copy code
prefect deployment build main.py:foo \
    -n sean-k8s-test-deployment \
    -q company-name \
    -sb s3/company-name-prefect-staging/sean-turner/foo \
    -ib kubernetes-job/test \
    --apply
EKS agent logs:
Copy code
16:09:45.420 | INFO    | prefect.agent - Submitting flow run 'c8f0bba7-b104-40d2-aa96-8a278c800f1e'
16:09:45.766 | INFO    | prefect.agent - Completed submission of flow run 'c8f0bba7-b104-40d2-aa96-8a278c800f1e'
16:10:45.785 | ERROR   | prefect.infrastructure.kubernetes-job - Job 'test6qzkh': Pod never started.
My 
kubernetes-job/test
infra block has 
{"EXTRA_PIP_PACKAGE": "s3fs"}
and does not have a kubeconfig set. My S3 infra block 
s3/company-name-prefect-staging/sean-turner/foo
does not have credentials because the agent and orion pods are assuming IAM roles that give permissions to read and write to the bucket.
Untitled.py
✅ 1
z
It’s possible the pod timeout is just too low?
pod_watch_timeout_seconds
defaults to 60 but can be set higher
s
Hmm, just upped it but I don't feel like that should be the issue. Is there anywhere I can get additional logs? I also added the attached heavily redacted kubeconfig and updated the environment variable to include the awscli 
{"EXTRA_PIP_PACKAGES": "s3fs,awscli"}
Untitled.txt
Maybe I need to update my EKS 
aws-auth
configmap to allow the Agent IRSA IAM role? I'm kind of grasping at straws, I don't think this sort of thing is documented anywhere
Updated the EKS aws-auth and experiencing the same issue. Is there a resource anywhere about setting the kubeconfig?
z
What does 
kubectl describe job <job-name>
get you?
🙌 1
You can also get all pods for the job 
kubectl get pods --selector=job-name=<job-name>
s
Ahhh interesting! I didn't realize that the jobs were actually being created
Copy code
Events:
  Type     Reason        Age                  From            Message
  ----     ------        ----                 ----            -------
  Warning  FailedCreate  5m29s (x9 over 27m)  job-controller  Error creating: pods "testkbhpn-" is forbidden: error looking up service account prefect/prefect: serviceaccount "prefect" not found
Okay huge progress. I'm unblocked. Thank you so much!
z
👍 I believe there’s an issue to report these errors via the agent as well
Open in Slack
PreviousNext
Powered by