Thread
#prefect-community
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    For a k8s agent, should all of the pods created for a flow runs inherit the
    Environment:*
    from the agent? I am having issues persisting a service account and role ARN that give flows privileges to hit aws resources (details in thread)
    When I describe my agent pod I am seeing the correct service account and role ARN with privs:
    Environment:
          AWS_DEFAULT_REGION:           us-west-1
          AWS_REGION:                   us-west-1
          AWS_ROLE_ARN:                 arn:aws:iam::<ID>:role/<ROLE_NAME>
          AWS_WEB_IDENTITY_TOKEN_FILE: <TOKEN_LOCATION>
    But when a job pod is created I no longer see those values and I am getting an AccessDenied error
    botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Query operation: User <USER> is not authorized to perform: dynamodb:Query
    When the role listed in the agent is authorized to preform the query
    I tried including the env var
    SERVICE_ACCOUNT_NAME
    in the run config in the UI for a quick test and still got the same error
    Kevin Kho

    Kevin Kho

    9 months ago
    This is beyond me. Will have to ask around.
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    Thanks @Kevin Kho
    Jamie Dick

    Jamie Dick

    9 months ago
    Hey Leon - just to confirm, your AWS role allows the mentioned dynamodb:query action?
    And if you describe one of your job pods, you are no longer seeing those env values present?
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    Hi Jamie - yes, the role I created has dynamodb:query and dynamodbšŸ˜’can for the table and index that Iā€™m hitting
    When I describe my agent pod I can see the role ARN for the correct role + a mount for the serviceaccount
    and when I show the yaml
    kubectl get deployment <NAME> -o yaml
    - I can see the
    serviceAccount
    and
    serviceAccountName
    as they should be
    then when I describe an in progress job for a flow run, I no longer see the role in
    Environment:
    or the mount
    Jamie Dick

    Jamie Dick

    9 months ago
    are you describing the job itself or the pod hosting the job?
    i would assume the job pods would inherit configuration from the agent pod but let me double check on this behavior
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    I was describing the pod sorry
    Like this:
    kubectl describe pod prefect-job-XXXXXX-XXXXX
    Jamie Dick

    Jamie Dick

    9 months ago
    ah got it. ok let me look into this
    can you share your kubernetes run config here?
    @Leon Kozlowski actually i think this is all you need - https://docs.prefect.io/orchestration/agents/kubernetes.html#service-account
    you can either specify the name of the service account in your run config
    service_account_name = arn:aws:iam::<ID>:role/<ROLE_NAME>
    or you can define it when you whereever you are starting the prefect agent
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    it should be set to the arn? or the service account name
    Jamie Dick

    Jamie Dick

    9 months ago
    yeah the SA, sorry about that!
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    Can this be done in the job_template?
    Was able to get it working by setting
    serviceAccountName
    in my
    job_template.yaml
    - thanks @Jamie Dick
    m

    Mariia Kerimova

    9 months ago
    Hi! I think you could add annotation to your kubernetes service account like this:https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html Make sure to annotate correct kubernetes service account (the one that used by your prefect job).
    Anna Geller

    Anna Geller

    9 months ago
    I was reading on IAM roles for Service Accounts, but what Maria shared seems to be equivalent because it also results in the same IAM role annotation - in both cases, you would need to create OIDC provider for your EKS cluster (need to be done only once) if you want to use IAM roles for SA.
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    Thanks all