I’d like to use a Secret stored in Prefect Cloud in my run config. I have stored the secret using `...

Stephen Lloyd

04/11/2022, 3:39 PM

I’d like to use a Secret stored in Prefect Cloud in my run config. I have stored the secret using

Copy code

>>> from prefect import Client
>>> client = Client()
>>> client.set_secret(name="test", value="test")
>>> client.set_secret(name="AWS_ACCOUNT_ID-dev", value="123456789098")

It seems like I should be able to use

Copy code

from prefect.client import Secret

aws_account_id = Secret('AWS_ACCOUNT_ID-' + RUN_ENV).get()

However, I receive the following error:

Copy code

ValueError: Local Secret "AWS_ACCOUNT_ID-dev" was not found.

How can I retrieve a secret value outside of a task?

Kevin Kho

04/11/2022, 3:43 PM

If executing locally, there is a config/env variable and the default is to pull local secrets. Set

Copy code

export PREFECT__CLOUD__USE_LOCAL_SECRETS=false

and this will pull the Cloud Secret

Kevin Kho

04/11/2022, 3:44 PM

This is set to false for agent/cloud runs

Stephen Lloyd

04/12/2022, 3:32 AM

Thanks. Where should I have found tis parameter/config setting?

Kevin Kho

04/12/2022, 3:41 AM

I guess it’s not documented on the Secrets page but more generally configurable stuff can be found in the config.toml

Stephen Lloyd

04/12/2022, 3:44 AM

Ah. Should I have this config file stored locally? Do the config selections get uploaded with a flow?

Kevin Kho

04/12/2022, 3:52 AM

You need them stored on where the flow is running, or you can pass the env variables

Copy code

PREFECT__CLOUD__XXX_XXX

for example to your flow to them to take effect

Stephen Lloyd

04/13/2022, 4:16 AM

Thanks so much!

Stephen Lloyd

04/13/2022, 10:16 AM

@Kevin Kho Following up on this… • If I set via

export PREFECT…SECRETS=false

I get the expected behavior. • If I change the default config path with

export PREFECT__USER_CONFIG_PATH=/Users/slloyd/projects/my_path/.prefect

with

use_local_secret=false

I do not get the expected behavior • If I put the file in

$HOME/.prefect/config.toml

with

use_local_secret=false

in a docker image I do not get the expected behavior. Do you know what I am missing?

Kevin Kho

04/13/2022, 1:49 PM

I have noticed the same thing too with the

config.toml

in Docker. Would you be willing to write an issue about it? I can also open one and you can add your details there? You can use the RunConfig as that works

7 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.