https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Michail Melonas

05/04/2022, 12:45 PM
I’ve have a scheduled flow that is failing due to a credential issue:
Error downloading Flow from S3: An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.
I’m using AWS SSO credentials to upload the flow to S3:
Copy code
flow.storage = S3(
    bucket=Config.PREFECT_S3_BUCKET,
    client_options={
        "aws_access_key_id": Config.AWS_ACCESS_KEY_ID,
        "aws_secret_access_key": Config.AWS_SECRET_ACCESS_KEY,
        "aws_session_token": Config.AWS_SESSION_TOKEN,
    },
)
Initially, the flow runs. However, flows scheduled after the token expiry fail with the above error. How do I both use SSO to upload flows to S3 and have them scheduled on Prefect Cloud?
a

Anna Geller

05/04/2022, 1:01 PM
Did you try to do
aws configure
with AWS CLI on the host from which you register your flow? Do you register from your local machine, some shared VM or from CI?
m

Michail Melonas

05/04/2022, 1:35 PM
We’re using SSO, not an IAM key pair. These are short lived credentials generated by
aws sso login
or pasted from the SSO UI.
The flow was registered from my local machine
a

Anna Geller

05/04/2022, 1:49 PM
In that case I believe the easiest way would be to run this aws sso login command before you register the flow from your terminal? Would that work for you? Not sure how otherwise approach it, especially given that SSO would require you to continue in the browser, correct?
m

Michail Melonas

05/04/2022, 2:02 PM
I can successfully register the flow in question using SSO credentials. The issue occurs when I make use of the “schedule” functionality on Prefect Cloud: when the flow gets scheduled to run after the token expiry, I get the above
ExpiredToken
error.
The flow runs on a K8s cluster that has appropriate (non-SSO) AWS credentials as a Kubernetes secret. What I want is to: 1. register the flow from my local machine using SSO credentials (i.e., to use SSO to upload the flow to S3) 2. when the flow gets triggered to run, the credentials saved as a Kubernetes secret to be used (i.e., to use said secret to download the flow from S3).
Is that possible?
a

Anna Geller

05/04/2022, 2:20 PM
I see - I understand your issue now. I don't think that using SSO will work for the remote agent. SSO are credentials bound to a person (here: to you). For an agent, you need long-lived programmatic access credentials or an IAM role
m

Michail Melonas

05/04/2022, 2:25 PM
Okay. So I can’t use two different types of credentials for uploading flows from my local machine and the agent downloading them?
a

Anna Geller

05/04/2022, 3:09 PM
you totally can, AWS S3 permissions can be set in a variety of ways - for registering from your laptop you may use your SSO credentials and for the agent you may use a totally different IAM user do you have some DevOps engineer in your team who can help you set this up?
m

Michail Melonas

05/04/2022, 3:14 PM
Yes, I do. This has already been configured — I use SSO credentials and the Agent has programmatic credentials. With reference to the above code snippet, how do I create a
Flow
instance such that two different sets of credentials can be used for upload/download of the flow?
a

Anna Geller

05/04/2022, 3:48 PM
I'm not sure what's the best way of explaining it is There are two processes: build-time process and runtime process Build-time This is the part where you register your flow. While registering your flow, Prefect may push your flow code to S3 if you want. If you do this from your laptop, then authenticating via SSO and then registering your flow will work and this concludes the build time process Runtime This is the part where your agent deploys scheduled flow runs and while doing that, it needs to pull your flow code from S3. To do that, your agent (your execution layer) needs to have valid credentials. Given that you use the Kubernetes agent, you could configure that using IAM roles for service accounts attached to your Kubernetes cluster.
IAM roles for service accounts are the AWS-recommended way of setting credentials to interact with AWS services within Kubernetes workloads. This applies to flow run pods but also to Dask nodes deployed within the same cluster as those permissions can be applied on the cluster level. The easiest way to set this up is using eksctl: https://eksctl.io/usage/iamserviceaccounts/
p

Patrick Tan

06/01/2022, 7:30 PM
@Anna Geller, I am in similar situation, I registered flow using AWS temporary credentials. My agent (local agent) is deployed in ECS task. I attached task role with full permission to S3 bucket where Flow is stored. I am still getting ExpiredToken error
a

Anna Geller

06/01/2022, 7:54 PM
Can you post it in a separate thread @Patrick Tan? your issue is very different since it's for ECS, not K8s
👍 1
25 Views