https://prefect.io logo
Join Slack
Powered by
Hey everyone, qq. When using CodeCommit as your fl...
# ask-community
j
Hey everyone, qq. When using CodeCommit as your flow storage, is there a way to pass in the credentials / region name for the boto3 authentication via PrefectSecret() ?
z
hey Joseph, 
CodeCommit
flow storage should look for a secret named 
AWS_CREDENTIALS
to get boto3 authentication A secret named "AWS_CREDENTIALS" should be a json string that looks something like
Copy code
{
  "ACCESS_KEY": XXX,
  "SECRET_ACCESS_KEY": XXX,
  "SESSION_TOKEN": XXX (I think this is optional)
}
in your 
CodeCommit
storage, you'll need to specify the secret being used, and you can pass the aws region
Copy code
storage = CodeCommit(
    ... # other stuff
    secrets = ["AWS_CREDENTIALS"],
    client_options={"region_name": "us-east-1"}
)
🙌 1
j
AWESOME! thank you! I'm probably going to need region-name in there too right?
you beat me to it 🙂
🙂 1
z
I haven't tested the region name kwarg personally, let me know if you run into issues with that one
j
ah I'm still running into trouble, I've got this error:
Copy code
Failed to load and execute Flow's environment: PartialCredentialsError('Partial credentials found in explicit, missing: aws_secret_access_key')
so then I set my AWS_CREDENTIALS to:
Copy code
{
  "aws_access_key_id": "XXX",
  "aws_secret_access_key": "XXX"
}
and resulted in another error:
Copy code
Failed to load and execute Flow's environment: NoCredentialsError('Unable to locate credentials')
z
hmmm your first setup should have the correct configuration
what did your AWS_CREDENTIALS secret look like when you ran into the first error? (omitting actual secret values of course)
j
exactly what you sent over, except for the session token. In my console I'm able to connect to prefect.utilities "aws" using the same two variables, ACCESS_KEY and SECRET_ACCESS_KEY like your example
👍 1
z
hmm and what does your 
CodeCommit
storage definition look like now?
j
Copy code
with Flow('Intraday Strategy Greeks',
        run_config = LocalRun(
                labels=['sradev68']
        ),
        storage = CodeCommit(
                repo='repo_name',
                path='path_to_flow.py',
                commit='branch_name',
                secrets=["AWS_CREDENTIALS"],
                client_options = {"region_name":"us-east-2"})
          ) as flow:
whoops we're good now! Weird, I just re-registered and it skipped because metadata is unchanged, yet now it's running from the CLI
z
nice! 🎉
💯 1
j
@Zach Angell this is working terrific! I do have a follow up question regarding AWS and Prefect. I'm trying to load a secret from AWS using thethe proper Prefect AWS Task. I was able to get it with this logic, but I'm not sure if using .run() in a flow here is the correct method.
Copy code
with Flow('Load AWS Secret') as flow:
    cred = PrefectSecret("AWS_CREDENTIALS")
    secret_value = AWSSecretsManager(secret="AWS_SECRET").run(credentials=cred)
The above works, but I'm wondering if there is a better way, like using a task decorator and passing in the PrefectSecret as an argument....would that work?
z
Hi @Joseph Loss the way you're doing things here is correct and best practice The reason you have to use 
.run()
here is because 
AWSSecretsManager
is itself a 
Task
. Something like this would also work
Copy code
secret_manger_task = AWSSecretsManager() # initialize a task
with Flow('load secret') as flow:
    cred = PrefectSecret("AWS_CREDENTIALS")
    secret_value = secret_manager_task(secret="MY_SECRET", credentials=cred) # calling the task will run it in the flow
Open in Slack
PreviousNext
Powered by