https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Given a JSON secret block like `{"use...
# marvin-ai
m
@Marvin Given a JSON secret block like 
{"username": "...", "password": "..."}
, how can I use interpolation to get a selected attribute value? I want to set it as an environment job variable for a deployment.
m
✅ thought for 114.9 seconds
✅ thought for 1607.6 seconds
✅ thought for 912.4 seconds
Great question. You can interpolate specific keys from a JSON-style block directly into deployment job variables (like environment variables) using Jinja in Prefect 3.x. First, a quick clarification to give you the right syntax: - Are you storing that object in a Prefect JSON block, or in a Secret block that contains a JSON string? The best practice for selecting fields is to use a JSON block. If you store JSON inside a Secret block, it’s treated as a single string and can’t be sub-selected without parsing. Assuming a JSON block named 
db-creds
with value:
Copy code
{
  "username": "alice",
  "password": "s3cr3t"
}
You can set environment variables in your deployment like this: In prefect.yaml:
Copy code
deployments:
  - name: my-flow
    job_variables:
      env:
        DB_USER: "{{ block('json', 'db-creds').username }}"
        DB_PASS: "{{ block('json', 'db-creds')['password'] }}"
Or in Python when deploying:
Copy code
flow.from_source("<https://github.com/owner/repo>", entrypoint="flows/my_flow.py:flow").deploy(
  name="my-flow",
  job_variables={
    "env": {
      "DB_USER": "{{ block('json', 'db-creds').username }}",
      "DB_PASS": "{{ block('json', 'db-creds')['password'] }}",
    }
  }
)
Notes: - You can use either dot access or bracket access for keys. If your key has special characters or starts with a number, prefer 
['key']
. - If you currently have a Secret block that stores a JSON string, the 
block('secret', 'name')
interpolation returns the entire string. Prefect’s templating does not guarantee a built-in JSON-parse filter, so you won’t be able to select 
username
directly. In that case, consider: - Moving the data to a JSON block, or - Splitting into multiple Secret blocks (e.g., 
db-username
, 
db-password
). Docs for reference: - Customize job variables - Store secrets If you can share which block type you’re using (JSON vs Secret), I can tailor the exact snippet for your setup.
n
thats.... not valid at all @Marvin there's no 
block
template thing that you're suggesting there im not immediately sure if you can refer by key to sub-objects using the template string syntax surefire way would be to modify 
os.environ
manually by reading the block once you start, though i understand why you might not want to do that
m
✅ thought for 239.6 seconds
n
sounds like a decent enchancement request, if its not possible
👍 1
m
It’s a Secret block in my case.
m
✅ thought for 48.4 seconds
Short answer: - If it’s a JSON block, you can subselect keys in Jinja. - If it’s a Secret block that contains a JSON string, you cannot subselect; switch to a JSON block or split secrets. Examples (Prefect 3.x): In prefect.yaml:
Copy code
deployments:
  - name: my-flow
    job_variables:
      env:
        DB_USER: "{{ block('json', 'db-creds').username }}"
        DB_PASS: "{{ block('json', 'db-creds')['password'] }}"
Or in Python when deploying:
Copy code
flow.from_source("<https://github.com/owner/repo>", entrypoint="flows/my_flow.py:flow").deploy(
  name="my-flow",
  job_variables={
    "env": {
      "DB_USER": "{{ block('json', 'db-creds').username }}",
      "DB_PASS": "{{ block('json', 'db-creds')['password'] }}",
    }
  }
)
Notes: - Use 
block('json', '<block-name>')
for JSON blocks and select fields with 
.key
or 
['key']
. - If you currently have a Secret block that stores the entire JSON as a string, 
block('secret', '<block-name>')
returns the whole string and you can’t pick out 
username
via interpolation. Prefer a JSON block for structured values or split into separate Secret blocks. Docs: - Customize job variables - Store secrets If you tell me whether your block is type JSON or Secret (and its name), I can give you the exact snippet.
You’re right — my earlier suggestion was off. There isn’t a 
block(...)
macro you can use in job variable templates, and you can’t sub-select keys from a JSON block via the Jinja templating used in 
job_variables
. What is supported in Prefect 3.x templates (e.g., in 
prefect.yaml
→ 
job_variables
) are things like: - 
env('NAME')
to pull from the worker process environment - workspace variables (e.g., 
var('NAME')
) — useful if you split values into simple strings So for your use case (pulling one key out of a JSON/Secret block) there isn’t a direct interpolation syntax. Here are workable approaches: Option 1: Pass-through environment variables - Set the env on your worker (or work pool’s base job template), then pass it through in the deployment:
Copy code
deployments:
  - name: my-flow
    job_variables:
      env:
        DB_USER: "{{ env('DB_USER') }}"
        DB_PASS: "{{ env('DB_PASS') }}"
Docs: Customize job variables Option 2: Workspace variables - Store them separately as workspace variables (e.g., 
db_username
, 
db_password
) and reference them:
Copy code
deployments:
  - name: my-flow
    job_variables:
      env:
        DB_USER: "{{ var('db_username') }}"
        DB_PASS: "{{ var('db_password') }}"
This avoids parsing JSON in templates by keeping each value as a string variable. Docs: Customize job variables Option 3: Set env in code at runtime (the “surefire” way) - Load the block and set 
os.environ
before your flow logic. This is the most flexible if you must keep a single JSON/Secret block. For a JSON block:
Copy code
import os, json
from prefect.blocks.system import JSON as JSONBlock

def configure_env_from_json_block(block_name: str):
    b = JSONBlock.load(block_name)
    # In Prefect 3, JSON block stores the data as a Python object (commonly under `value`)
    data = getattr(b, "value", None) or {}
    os.environ["DB_USER"] = data["username"]
    os.environ["DB_PASS"] = data["password"]

# Call early (e.g., at module import or start of your flow)
configure_env_from_json_block("db-creds")
For a Secret block that stores a JSON string:
Copy code
import os, json
from prefect.blocks.system import Secret

def configure_env_from_secret_json(block_name: str):
    raw = Secret.load(block_name).get()  # returns the string
    data = json.loads(raw)
    os.environ["DB_USER"] = data["username"]
    os.environ["DB_PASS"] = data["password"]

configure_env_from_secret_json("db-creds")
If you can share which worker type you’re using (Docker/Kubernetes/Process) and whether you can use workspace variables, I can suggest the cleanest path for your setup. For reference on storing secrets/blocks: Store secrets
Great question. In Prefect 3.x you can interpolate values from blocks directly in prefect.yaml, and use them in job environment variables. If your block is a system.JSON block whose value is: {"username": "...", "password": "..."} then reference the specific fields via the JSON block’s value attribute:
Copy code
deployments:
- name: my-deployment
  entrypoint: flows/my_flow.py:flow
  job_variables:
    env:
      DB_USERNAME: "{{ prefect.blocks.system.json.my-credentials.value.username }}"
      DB_PASSWORD: "{{ prefect.blocks.system.json.my-credentials.value.password }}"
Key points: - Use the full path: 
prefect.blocks.system.json.<block-slug>.value.<key>
- Replace 
my-credentials
with your block’s slug. If instead you used a system.Secret block: - A Secret block resolves to a single string value. Placeholders will return the entire secret value, e.g.:
Copy code
API_TOKEN: "{{ prefect.blocks.system.secret.my-token }}"
- You cannot select a nested field from a Secret block if it contains JSON as a string. If you need per-field values, use a system.JSON block (recommended for structured data) or separate Secret blocks for each value. Helpful docs: - prefect.yaml templating - Customize job variables - Blocks overview If you can share which block type you created (system.JSON vs system.Secret) and its slug, I can tailor the exact placeholder for you.
n
sorry, not sure why marvin is spamming this thread
m
Probably, because I fix some typos and edited my question.
n
ahhh yea that'll do it
m
@Marvin Could I make the interpolation work for environment variables in deployments by creating a custom block type (e.g having a separate 
username
and 
password
property)?
m
✅ thought for 50.8 seconds
Short answer: creating a custom block with separate `username`/`password` fields won’t make those fields interpolatable into 
job_variables.env
. Today, block references are not resolved in that templating context. The supported interpolation there is: - 
{{ env('NAME') }}
for environment variables available to the worker - 
{{ var('NAME') }}
for Prefect workspace variables Workable patterns: - Use workspace variables per field and reference them:
Copy code
job_variables:
  env:
    DB_USER: "{{ var('db_username') }}"
    DB_PASS: "{{ var('db_password') }}"
- Pass through OS env vars available on the worker:
Copy code
job_variables:
  env:
    DB_USER: "{{ env('DB_USER') }}"
    DB_PASS: "{{ env('DB_PASS') }}"
- Set env in code at runtime by loading your block (the “surefire” approach you mentioned):
Copy code
import os, json
from prefect.blocks.system import JSON as JSONBlock, Secret

def configure_from_json_block(name):
    data = (JSONBlock.load(name).value) or {}
    os.environ["DB_USER"] = data["username"]
    os.environ["DB_PASS"] = data["password"]

def configure_from_secret_json(name):
    raw = Secret.load(name).get()
    data = json.loads(raw)
    os.environ["DB_USER"] = data["username"]
    os.environ["DB_PASS"] = data["password"]
If you want a single source of truth but still use interpolation, the most “templating-friendly” approach is splitting the values into separate workspace variables (or separate Secret blocks per field) and referencing them with 
var()
. Docs: - Customize job variables - Store secrets If you share your worker type (Process/Docker/Kubernetes) and where you prefer to manage secrets (Prefect variables, OS env, platform secrets), I can suggest the cleanest setup for that environment.
n
i think marvin is wrong, that should work fine
one sec lemme see if we handle this now
👍 1
Copy code
import os
from pathlib import Path

from prefect import flow
from prefect.blocks.system import Secret

secret = Secret(value=dict(a=1, b=2))
secret.save("my-secret", overwrite=True, _sync=True)


@flow
def test_flow():
    value = os.getenv("SECRET")
    print(value, type(value))
    return value


if __name__ == "__main__":
    test_flow.from_source(
        source=str(Path(__file__).parent),
        entrypoint="try_template_secret.py:test_flow",
    ).deploy(
        name="test-flow",
        work_pool_name="local",
        job_variables=dict(env=dict(SECRET="{{ prefect.blocks.secret.my-secret.a }}")),
    )
sorry for delay, we support this today
Copy code
» uv run sandbox/try_template_secret.py
Looks like you're deploying to a process work pool. If you're creating a deployment for local development, calling `.serve` on your flow is a great way to get started. See the
documentation for more information: <https://docs.prefect.io/latest/how-to-guides/deployments/run-flows-in-local-processes>  Set `ignore_warnings=True` to suppress this message.
Successfully created/updated all deployments!

                Deployments
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Name                ┃ Status  ┃ Details ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ test-flow/test-flow │ applied │         │
└─────────────────────┴─────────┴─────────┘

To schedule a run for this deployment, use the following command:

        $ prefect deployment run 'test-flow/test-flow'


You can also run your flow via the Prefect UI: <http://127.0.0.1:4200/deployments/deployment/241501c4-a31e-4938-9597-51d128794600>

» prefect deployment run 'test-flow/test-flow'
Creating flow run for deployment 'test-flow/test-flow'...
Created flow run 'crazy-mosquito'.
└── UUID: bfe4f36c-e001-4f17-a3d0-e54f0b8375d2
└── Parameters: {}
└── Job Variables: {}
└── Scheduled start time: 2025-09-05 11:22:25 CDT (now)
└── URL: <http://127.0.0.1:4200/runs/flow-run/bfe4f36c-e001-4f17-a3d0-e54f0b8375d2>
» prefect worker start --pool 'local' --run-once
Discovered type 'process' for work pool 'local'.
Worker 'ProcessWorker aa2fe605-4c88-483b-b6d2-c0eab4eb502c' started!
11:22:43.582 | INFO    | prefect.flow_runs.worker - Worker 'ProcessWorker aa2fe605-4c88-483b-b6d2-c0eab4eb502c' submitting flow run 'bfe4f36c-e001-4f17-a3d0-e54f0b8375d2'
11:22:44.301 | INFO    | prefect.flow_runs.runner - Opening process...
11:22:44.420 | INFO    | prefect.flow_runs.worker - Completed submission of flow run 'bfe4f36c-e001-4f17-a3d0-e54f0b8375d2'
11:22:45.798 | INFO    | Flow run 'crazy-mosquito' -  > Running set_working_directory step...
11:22:47.068 | INFO    | Flow run 'crazy-mosquito' - Beginning flow run 'crazy-mosquito' for flow 'test-flow'
11:22:47.069 | INFO    | Flow run 'crazy-mosquito' - View at <http://127.0.0.1:4200/runs/flow-run/bfe4f36c-e001-4f17-a3d0-e54f0b8375d2>
11:22:47.176 | INFO    | Flow run 'crazy-mosquito' - Finished in state Completed()
1 <class 'str'>
11:22:47.589 | INFO    | prefect.flow_runs.runner - Process for flow run 'crazy-mosquito' exited cleanly.
Worker 'ProcessWorker aa2fe605-4c88-483b-b6d2-c0eab4eb502c' stopped!
Copy code
job_variables=dict(env=dict(SECRET="{{ prefect.blocks.secret.my-secret.a }}"))
m
@Nate Thank you! I can see it working in your code. However, I can’t get it to work when I set the secret value via the Prefect UI. I tried setting the secret block value 
{"a": "1", "b": "2"}
as 
password
, 
string
, and 
JSON
. In all cases I got the following error while running the flow:
Copy code
ValueError: Invalid template: '{{ prefect.blocks.secret.my-secret.a }}'. Could not resolve the keypath in the block document data.
One difference I noticed between setting the secret value via the UI and 
secret.save(,,,)
is that the value does not seem to be obfuscated when I inspect the block: After setting the secret block value via the UI (as 
JSON
type and value 
{"a": "1", "b": "2"}
):
Copy code
prefect block inspect secret/my-secret      
                                                                                                                       secret/my-secret                                                                                                                       
┌─────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Block Type                                                  │ Secret                                                                                                                                                                                       │
│ Block id                                                    │ c87de180-6759-49a6-9a18-500b6ee10cd2                                                                                                                                                         │
├─────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ value                                                       │ ********                                                                                                                                                                                     │
└─────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
After setting the secret value via 
Secret(value=dict(a=1, b=2)).save("my-secret", overwrite=True, _sync=True)
Copy code
prefect block inspect secret/my-secret      
                                                                                                                       secret/my-secret                                                                                                                       
┌─────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Block Type                                                  │ Secret                                                                                                                                                                                       │
│ Block id                                                    │ c87de180-6759-49a6-9a18-500b6ee10cd2                                                                                                                                                         │
├─────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ value                                                       │ {'a': 1, 'b': 2}                                                                                                                                                                             │
└─────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
I would have expected 
value
to be obfuscated in both cases.
Open in Slack
PreviousNext
Powered by