https://prefect.io
Join Slack
Hey everyone, qq. When using CodeCommit as your flow storage, is there a way to pass in the credenti...
# ask-community
j
Hey everyone, qq. When using CodeCommit as your flow storage, is there a way to pass in the credentials / region name for the boto3 authentication via PrefectSecret() ?
z
hey Joseph, 
CodeCommit
flow storage should look for a secret named 
AWS_CREDENTIALS
to get boto3 authentication A secret named "AWS_CREDENTIALS" should be a json string that looks something like
Copy code
{
  "ACCESS_KEY": XXX,
  "SECRET_ACCESS_KEY": XXX,
  "SESSION_TOKEN": XXX (I think this is optional)
}
in your 
CodeCommit
storage, you'll need to specify the secret being used, and you can pass the aws region
Copy code
storage = CodeCommit(
    ... # other stuff
    secrets = ["AWS_CREDENTIALS"],
    client_options={"region_name": "us-east-1"}
)
🙌 1
j
AWESOME! thank you! I'm probably going to need region-name in there too right?
you beat me to it 🙂
🙂 1
z
I haven't tested the region name kwarg personally, let me know if you run into issues with that one
j
ah I'm still running into trouble, I've got this error:
Copy code
Failed to load and execute Flow's environment: PartialCredentialsError('Partial credentials found in explicit, missing: aws_secret_access_key')
so then I set my AWS_CREDENTIALS to:
Copy code
{
  "aws_access_key_id": "XXX",
  "aws_secret_access_key": "XXX"
}
and resulted in another error:
Copy code
Failed to load and execute Flow's environment: NoCredentialsError('Unable to locate credentials')
z
hmmm your first setup should have the correct configuration
what did your AWS_CREDENTIALS secret look like when you ran into the first error? (omitting actual secret values of course)
j
exactly what you sent over, except for the session token. In my console I'm able to connect to prefect.utilities "aws" using the same two variables, ACCESS_KEY and SECRET_ACCESS_KEY like your example
👍 1
z
hmm and what does your 
CodeCommit
storage definition look like now?
j
Copy code
with Flow('Intraday Strategy Greeks',
        run_config = LocalRun(
                labels=['sradev68']
        ),
        storage = CodeCommit(
                repo='repo_name',
                path='path_to_flow.py',
                commit='branch_name',
                secrets=["AWS_CREDENTIALS"],
                client_options = {"region_name":"us-east-2"})
          ) as flow:
whoops we're good now! Weird, I just re-registered and it skipped because metadata is unchanged, yet now it's running from the CLI
z
nice! 🎉
💯 1
j
@Zach Angell this is working terrific! I do have a follow up question regarding AWS and Prefect. I'm trying to load a secret from AWS using thethe proper Prefect AWS Task. I was able to get it with this logic, but I'm not sure if using .run() in a flow here is the correct method.
Copy code
with Flow('Load AWS Secret') as flow:
    cred = PrefectSecret("AWS_CREDENTIALS")
    secret_value = AWSSecretsManager(secret="AWS_SECRET").run(credentials=cred)
The above works, but I'm wondering if there is a better way, like using a task decorator and passing in the PrefectSecret as an argument....would that work?
z
Hi @Joseph Loss the way you're doing things here is correct and best practice The reason you have to use 
.run()
here is because 
AWSSecretsManager
is itself a 
Task
. Something like this would also work
Copy code
secret_manger_task = AWSSecretsManager() # initialize a task
with Flow('load secret') as flow:
    cred = PrefectSecret("AWS_CREDENTIALS")
    secret_value = secret_manager_task(secret="MY_SECRET", credentials=cred) # calling the task will run it in the flow
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by