https://prefect.io logo
Title
c

Charles Leung

03/10/2021, 6:45 PM
Hello Prefect Team, I'm trying to use local secrets with the core server setup to test some things. However, I constantly get an key error even as i set the environment variable via the agent as such: prefect agent start docker --env PREFECT__CONTEXT__SECRETS__VAULT_TOKEN=<REDACT> --api <REDACT> Unexpected error: HTTPError('400 Client Error: Bad Request for url: <REDACT>/graphql') Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/prefect/client/secrets.py", line 137, in get value = secrets[self.name] File "/usr/local/lib/python3.7/site-packages/box/box.py", line 335, in getitem raise BoxKeyError(str(err)) from None box.exceptions.BoxKeyError: "'vault_token'" Am i missing something in my setup? use_local_secrets is set to true in the config.
j

Jim Crist-Harif

03/10/2021, 6:49 PM
It looks like you're setting your key as
VAULT_TOKEN
, but requesting it as
vault_token
(keys are case sensitive). I'd start trying with matching case and see if that works.
c

Charles Leung

03/10/2021, 6:49 PM
I actually tried that first, and got the same issue 😞
j

Jim Crist-Harif

03/10/2021, 6:50 PM
What version of prefect is your agent running?
c

Charles Leung

03/10/2021, 6:50 PM
0.13.5
j

Jim Crist-Harif

03/10/2021, 6:52 PM
Ah, that's very old. Several bugs when using secrets with prefect server have been fixed since then, I'd suggest upgrading to see if that fixes the issue.
c

Charles Leung

03/10/2021, 6:52 PM
ok got it, thanks!
Hey Jim, Thanks for your quick response. I've updated my version and i now get the following error:
Unexpected error: ValueError('Local Secret "VAULT_TOKEN" was not found.')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 869, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 299, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/secrets/base.py", line 68, in run
    return _Secret(name).get()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/secrets.py", line 142, in get
    ) from None
ValueError: Local Secret "VAULT_TOKEN" was not found.
Is there a way to debug inside the environment?
j

Jim Crist-Harif

03/10/2021, 9:15 PM
How are you accessing the secret? With a
PrefectSecret
object?
c

Charles Leung

03/10/2021, 9:16 PM
yes
import os
import prefect
from prefect.tasks.secrets import PrefectSecret
from prefect.environments.storage.docker import Docker

flow = prefect.Flow('vault-test')
fetch_token = PrefectSecret('VAULT_TOKEN')
flow.add_task(fetch_token)
j

Jim Crist-Harif

03/10/2021, 9:25 PM
Hmmm. Try registering and running the following flow and see what it outputs (with the same agent configuration):
import os
import prefect
from prefect import Flow, task


@task(log_stdout=True)
def log_env():
    print("Logging prefect environment variables")
    for k, v in os.environ.items():
        if k.startswith("PREFECT_"):
            print(f"ENVAR: {k} = {v}")


@task(log_stdout=True)
def log_secrets():
    print("Logging prefect secrets")
    for k in prefect.context.get("secrets", {}):
        print(f"SECRET: {k}")


with Flow("debug") as flow:
    log_env()
    log_secrets()
c

Charles Leung

03/10/2021, 10:47 PM
Perfect Jim 🙂 this task helped me find what i needed. I was using jinja templating to create a systemctl prefect agent service, and a \n between the --env and the initial command was breaking it. Thanks!
👍 1
j

Jim Crist-Harif

03/10/2021, 10:50 PM
Glad you figured it out!