https://prefect.io
Join Slack
Good evening from Germany, is there a difference how I activate cloud secret storage on different cl...
# ask-community
f
Good evening from Germany, is there a difference how I activate cloud secret storage on different clients? Is there an example of how I turn in on in the config? The documentation is not very helpful about how to actually do it, for me at least.
k
Hi @Florian Kühnlenz, do you mean like 
PREFECT__CLOUD__USE_LOCAL_SECRETS=false
?
f
Yes. I guess so. But it is not clear to me where I set it for example for the docker client. Is it needed as env var inside the containers? Or just when starting the client? The documentation also mentions the config.toml but without an example.
Okay, I think I tried every combination I could think of. I could not make the cloud secret load via PrefectSecret inside a DockerAgent.
k
that sounds like it would work. It should already pull the Secret from Cloud when you’re running it on the Docker agent. Did you try the RunConfig? Do you get an error saying something like “Local Secret not found?”
f
Yes I get Local Secret not found. Which run config?
k
flow.run_config = DockerRun(env={"PREFECT__CLOUD__USE_LOCAL_SECRETS": "false"})
upvote 1
f
I can try this out. But just for clarification what is the canocial way to set it for the docker agent?
This seems to result in the same error. Am I missing something else to configure?
k
I don’t think there is anything else to configure. It should work. What is your Prefect version?
f
0.14.22 But looking at the implementation of the prefect secret it also seems to check if the backend is set to cloud. Does the flow inside the docker container started form the agent know anything about the backend?
k
Wait are you on Server or Cloud? It does check but I think the check for Server in that code was to give a definitive error message that Secrets are not supported for Server
f
I am on cloud. I am just wondering if that is correctly visible for the flow.
k
Can you try this and see if it works with the 
"PREFECT__CLOUD__USE_LOCAL_SECRETS": "false"
Copy code
from prefect.client import Secret
from prefect.client.client import Client
client = Client()
test = client.set_secret(name="MYSECRET", value="MY SECRET VALUE")
print(Secret("MYSECRET").get())
Ok I got clarification on this from the core team. The use_local_secrets is 
true
by default when you are developing locally. This means it pulls from environment variables. When you run with Cloud as the backend, this value is 
false
by default. It should pull the Cloud secret by default when you run your Flow.
Testing with the script above and setting 
use_local_secrets
to false will help you see if you can pull the secret you’re trying to pull
f
okay, I did never set up a local env to test interaction with cloud, but this would not explain at all why it did not work out of the box in cloud when deployed.
k
Just wanna be sure, are you using the Secret in a Task or Flow?
f
We use 
password = PrefectSecret("PWD")
within the flow context
k
Let me check about PrefectSecret. It might be deprecated in favor of Secret
f
k
Gotcha this should work one sec I’ll try to replicate
f
I am also trying to get some log output for 30 min but the slow build chain and my brain at the end of a day are really holding me back...
Okay so I put in a 
<http://logger.info|logger.info>(prefect.config.backend)
and this prints server. So the flow inside the docker agent does not know the correct backend.
k
How are you starting the Docker agent?
f
via the commandline 🙂. Anything specific to look for?
k
There is a backend.toml file that should say “cloud”. Maybe you can run 
prefect backend cloud
where the agent is being spun up from?
f
So passing the token to the agent and setting the api is not enough? If that is the case it is a little intransparent. I will check tomorrow if this is set.
k
Oh I see your point. I think because you can set the API, but that just saves the config. There are some users to toggle back and forth with 
prefect backend server
and 
prefect backend cloud
so even if you configure, you still need to specify the backend
f
I just realized I have the following problem: on the same VM I need to have two docker agents running, where one is connecting to server and one to cloud.
k
Were you able to get it working? I think you can do it with the ENTRYPOINT?
f
Well it seems to be working now. Maybe would it would be good to have a better logging message to make this configuration error easier discoverable?
k
You mean logging message of what the backend is right?
f
Sorry for the late reply. I mean two things: 1. When to docker agents starts it could log what mode it is in. 2. The error message if not secret is found could mention the mode
k
I definitely agree with number 2, and one should not be hard. 1 may likely be in logged in debug mode. I’ll open an issue for this. Thanks ! 🙂
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by