Hello Prefect Team, I'm trying to use local secret...
# prefect-serverc
Charles Leung
03/10/2021, 6:45 PMHello Prefect Team, I'm trying to use local secrets with the core server setup to test some things. However, I constantly get an key error even as i set the environment variable via the agent as such:
prefect agent start docker --env PREFECT__CONTEXT__SECRETS__VAULT_TOKEN=<REDACT> --api <REDACT>
Unexpected error: HTTPError('400 Client Error: Bad Request for url: <REDACT>/graphql')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/client/secrets.py", line 137, in get
value = secrets[self.name]
File "/usr/local/lib/python3.7/site-packages/box/box.py", line 335, in getitem
raise BoxKeyError(str(err)) from None
box.exceptions.BoxKeyError: "'vault_token'"
Am i missing something in my setup? use_local_secrets is set to true in the config.
j
Jim Crist-Harif
03/10/2021, 6:49 PMIt looks like you're setting your key as
VAULT_TOKENvault_tokenc
Charles Leung
03/10/2021, 6:49 PMI actually tried that first, and got the same issue 😞
j
Jim Crist-Harif
03/10/2021, 6:50 PMWhat version of prefect is your agent running?
c
Charles Leung
03/10/2021, 6:50 PM0.13.5
Charles Leung
03/10/2021, 6:50 PMi am following this flow example: https://docs.prefect.io/core/concepts/secrets.html#local-context
j
Jim Crist-Harif
03/10/2021, 6:52 PMAh, that's very old. Several bugs when using secrets with prefect server have been fixed since then, I'd suggest upgrading to see if that fixes the issue.
c
Charles Leung
03/10/2021, 6:52 PMok got it, thanks!
Charles Leung
03/10/2021, 9:10 PMHey Jim, Thanks for your quick response. I've updated my version and i now get the following error:
Copy code
Unexpected error: ValueError('Local Secret "VAULT_TOKEN" was not found.')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 869, in get_task_run_state
logger=self.logger,
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 299, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/secrets/base.py", line 68, in run
return _Secret(name).get()
File "/usr/local/lib/python3.7/site-packages/prefect/client/secrets.py", line 142, in get
) from None
ValueError: Local Secret "VAULT_TOKEN" was not found.Charles Leung
03/10/2021, 9:14 PMIs there a way to debug inside the environment?
j
Jim Crist-Harif
03/10/2021, 9:15 PMHow are you accessing the secret? With a
PrefectSecretc
Charles Leung
03/10/2021, 9:16 PMyes
Copy code
import os
import prefect
from prefect.tasks.secrets import PrefectSecret
from prefect.environments.storage.docker import Docker
flow = prefect.Flow('vault-test')
fetch_token = PrefectSecret('VAULT_TOKEN')
flow.add_task(fetch_token)j
Jim Crist-Harif
03/10/2021, 9:25 PMHmmm. Try registering and running the following flow and see what it outputs (with the same agent configuration):
Copy code
import os
import prefect
from prefect import Flow, task
@task(log_stdout=True)
def log_env():
print("Logging prefect environment variables")
for k, v in os.environ.items():
if k.startswith("PREFECT_"):
print(f"ENVAR: {k} = {v}")
@task(log_stdout=True)
def log_secrets():
print("Logging prefect secrets")
for k in prefect.context.get("secrets", {}):
print(f"SECRET: {k}")
with Flow("debug") as flow:
log_env()
log_secrets()c
Charles Leung
03/10/2021, 10:47 PMPerfect Jim 🙂 this task helped me find what i needed. I was using jinja templating to create a systemctl prefect agent service, and a \n between the --env and the initial command was breaking it. Thanks!
👍 1
j
Jim Crist-Harif
03/10/2021, 10:50 PMGlad you figured it out!
2 Views