https://prefect.io logo
Channels
Powered by
l

Leon Kozlowski

12/08/2021, 4:18 PM
For a k8s agent, should all of the pods created for a flow runs inherit the 
Environment:*
from the agent? I am having issues persisting a service account and role ARN that give flows privileges to hit aws resources (details in thread)
When I describe my agent pod I am seeing the correct service account and role ARN with privs:
Copy code
Environment:
      AWS_DEFAULT_REGION:           us-west-1
      AWS_REGION:                   us-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::<ID>:role/<ROLE_NAME>
      AWS_WEB_IDENTITY_TOKEN_FILE: <TOKEN_LOCATION>
But when a job pod is created I no longer see those values and I am getting an AccessDenied error
Copy code
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Query operation: User <USER> is not authorized to perform: dynamodb:Query
When the role listed in the agent is authorized to preform the query
I tried including the env var 
SERVICE_ACCOUNT_NAME
in the run config in the UI for a quick test and still got the same error
k

Kevin Kho

12/08/2021, 4:28 PM
This is beyond me. Will have to ask around.
l

Leon Kozlowski

12/08/2021, 4:30 PM
Thanks @Kevin Kho
j

Jamie Zieziula

12/08/2021, 4:58 PM
Hey Leon - just to confirm, your AWS role allows the mentioned dynamodb:query action?
And if you describe one of your job pods, you are no longer seeing those env values present?
l

Leon Kozlowski

12/08/2021, 5:01 PM
Hi Jamie - yes, the role I created has dynamodb:query and dynamodb:scan for the table and index that I’m hitting
When I describe my agent pod I can see the role ARN for the correct role + a mount for the serviceaccount
and when I show the yaml 
kubectl get deployment <NAME> -o yaml
- I can see the 
serviceAccount
and 
serviceAccountName
as they should be
then when I describe an in progress job for a flow run, I no longer see the role in 
Environment:
or the mount
j

Jamie Zieziula

12/08/2021, 5:44 PM
are you describing the job itself or the pod hosting the job?
i would assume the job pods would inherit configuration from the agent pod but let me double check on this behavior
l

Leon Kozlowski

12/08/2021, 5:51 PM
I was describing the pod sorry
Like this: 
kubectl describe pod prefect-job-XXXXXX-XXXXX
j

Jamie Zieziula

12/08/2021, 6:00 PM
ah got it. ok let me look into this
can you share your kubernetes run config here?
@Leon Kozlowski actually i think this is all you need - https://docs.prefect.io/orchestration/agents/kubernetes.html#service-account
you can either specify the name of the service account in your run config 
service_account_name = arn:aws:iam::<ID>:role/<ROLE_NAME>
or you can define it when you whereever you are starting the prefect agent
l

Leon Kozlowski

12/08/2021, 7:00 PM
it should be set to the arn? or the service account name
j

Jamie Zieziula

12/08/2021, 7:03 PM
yeah the SA, sorry about that!
l

Leon Kozlowski

12/08/2021, 7:17 PM
Can this be done in the job_template?
Was able to get it working by setting 
serviceAccountName
in my 
job_template.yaml
- thanks @Jamie Zieziula
m

Mariia Kerimova

12/08/2021, 7:35 PM
Hi! I think you could add annotation to your kubernetes service account like this: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html Make sure to annotate correct kubernetes service account (the one that used by your prefect job).
upvote 1
👍 1
a

Anna Geller

12/08/2021, 7:40 PM
I was reading on IAM roles for Service Accounts, but what Maria shared seems to be equivalent because it also results in the same IAM role annotation - in both cases, you would need to create OIDC provider for your EKS cluster (need to be done only once) if you want to use IAM roles for SA.
l

Leon Kozlowski

12/08/2021, 7:54 PM
Thanks all
8 Views
Powered by