<@ULVA73B9P> how to use blocks when developing loc...
# marvin-ai
k
@Marvin how to use blocks when developing locally with prefect?
m
thought for 177.8 seconds
Great question! Assuming you’re on Prefect 3.x, here’s a practical way to use Blocks during local development. Quick overview - Blocks are configuration objects stored in the Prefect API (Cloud or your local server). - You define a Block class in Python, save a named instance (a “block document”) to the API, then load it in your flows. - For persistent local dev, run a local Prefect server; otherwise, blocks saved to the ephemeral API won’t persist across sessions. Step-by-step: local workflow 1) Start a local server (for persistence) - In one terminal, start the server:
Copy code
prefect server start
- Prefect will use a local SQLite DB by default and expose the API at http://localhost:4200/api and the UI at http://localhost:3000. - Ensure your shell points to the local API when running code in another terminal:
Copy code
export PREFECT_API_URL=<http://localhost:4200/api>
2) Create and save a block - You can use built-in blocks or define your own. Example using the built-in Secret:
Copy code
from prefect.blocks.system import Secret

# create and save a block document
Secret(value="super-secret-token").save("my-api-token", overwrite=True)
- Custom block example:
Copy code
from prefect.blocks.core import Block
from pydantic import SecretStr

class MyCredentials(Block):
    api_key: SecretStr
    base_url: str = "<https://api.example.com>"

# save a named instance
MyCredentials(api_key="abc123", base_url="<https://api.example.com>").save("dev-creds", overwrite=True)
3) Load a block in your flows - In a sync flow:
Copy code
from prefect import flow
from prefect.blocks.system import Secret

@flow
def use_secret():
    token = Secret.load("my-api-token").get()
    print(f"Loaded token of length: {len(token)}")

use_secret()
- In an async flow, use `aload`:
Copy code
from prefect import flow
from prefect.blocks.system import Secret

@flow
async def use_secret_async():
    secret = await Secret.aload("my-api-token")
    print(secret.get())

use_secret_async()
4) View and manage blocks - In the UI: open http://localhost:3000, go to Blocks, then Catalog (create) or Documents (list existing). - From Python: list/inspect via the client if needed. - From CLI: use the
prefect block
command group (run
prefect block --help
to see what’s available on your version). Where blocks are stored locally - If you’re running
prefect server start
, blocks are stored in the server’s database (SQLite by default, typically under
~/.prefect/prefect.db
). - Without a running server (or without
PREFECT_API_URL
pointing at one), Prefect may use an ephemeral in-memory API — any saved blocks will be lost when the process ends. Export/import during local dev - You can serialize block instances (careful with secrets) to move between environments:
Copy code
from prefect.blocks.system import Secret

# export
s = Secret.load("my-api-token")
json_str = s.model_dump_json()  # does not include raw secret by default

# import
restored = Secret.model_validate_json(json_str)
restored.save("my-api-token-copy", overwrite=True)
Gotchas and tips - In async flows, prefer
await MyBlock.aload("name")
. - The Python class must be importable where you load it (the server stores the data, not the class code). - If you see “Block document not found”, verify your code’s
PREFECT_API_URL
matches the server you saved to. Helpful links - Prefect UI (local): Open local UI - Prefect API (local): Local API endpoint - Blocks in the UI: Blocks UI Want me to tailor this to your setup? Are you using the local server only, or Prefect Cloud too?
k
@Marvin I'm using Prefect Cloud too, but this requires me to put secrets in my code. What is the best practice here?
m
thought for 52.6 seconds
Great question — you shouldn’t put secrets in code. Here’s the best-practice approach for Prefect Cloud (and local dev) in Prefect 3.x: Core principles - Never hardcode secrets in Python or in your repo. - Store secrets in Prefect Cloud as Blocks (e.g., Secret, credential blocks) or in your cloud provider’s secret manager and reference them via a Prefect block. - In code, only reference block names and load them at runtime. - Authenticate to Cloud using an API key via environment variables or
prefect cloud login
, not in code. Recommended patterns 1) Create secrets as blocks in Prefect Cloud (no code) - In the Cloud UI (Blocks > Catalog), create a Secret (or the relevant credential block) and give it a name like
my-api-token
. - This keeps secrets out of code and your repo; Prefect encrypts secret fields at rest and scrubs them in logs. 2) Load the block in your flow - Sync flow:
Copy code
from prefect import flow
from prefect.blocks.system import Secret

@flow
def run():
    token = Secret.load("my-api-token").get()
    # use token (avoid logging or printing it)
- Async flow:
Copy code
from prefect import flow
from prefect.blocks.system import Secret

@flow
async def run_async():
    secret = await Secret.aload("my-api-token")
    token = secret.get()
3) Authenticate to Prefect Cloud without embedding credentials in code - Use
prefect cloud login
(stores auth locally) OR set env vars:
Copy code
export PREFECT_API_URL=<https://api.prefect.cloud>
export PREFECT_API_KEY=your_cloud_api_key
- In CI/agents, use a service account API key stored in your CI secret store or worker environment, not in code. 4) Local development options without secrets in code - Point your local dev to Prefect Cloud so your code loads the same Cloud blocks:
Copy code
export PREFECT_API_URL=<https://api.prefect.cloud>
export PREFECT_API_KEY=your_cloud_api_key
- Or, if you prefer a local server for dev, bootstrap local blocks from environment variables (still no secrets in code):
Copy code
import os
from prefect.blocks.system import Secret

Secret(value=os.environ["MY_API_TOKEN"]).save("my-api-token", overwrite=True)
Then set
MY_API_TOKEN
only in your local shell or .env (not committed), and load the block by name in your flows. 5) Rotation and environment separation - Rotate by updating the block document in Cloud; your flows pick up the new value without code changes. - Keep separate blocks per environment (e.g.,
my-api-token-dev
,
my-api-token-prod
) and/or separate workspaces in Cloud. Reference the appropriate block name from each deployment/flow. 6) Optional: Use a cloud secret manager via a Prefect block - If your org standard is AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault, use the corresponding Prefect integration block to fetch secrets at runtime. Your code still references a block name; the block reads the secret from your provider. Common pitfalls to avoid - Printing or logging secret values (even length can sometimes leak info). - Saving blocks programmatically with raw literals in code. If scripting block creation, read secrets from environment variables. - Mixing local server and Cloud without adjusting
PREFECT_API_URL
— you’ll get “Block document not found” if pointed at the wrong API. If you’d like, tell me which services you need creds for (e.g., AWS, GCP, DB) and your preferred workflow (Cloud-only vs. local dev + Cloud), and I can provide a tailored block setup and a bootstrap script that avoids secrets in code.
k
@Marvin I don't want Cloud Secrets, I want to use different values for local development
m
thought for 58.6 seconds
Got it — you want different secrets/values locally without storing secrets in Prefect Cloud. Here’s a clean pattern that keeps secrets out of your code and lets you switch sources by environment. What to do - Keep local secrets in local Blocks (on your local Prefect server). - For Cloud runs, don’t use Cloud Secrets; read from your org’s source of truth (env vars on workers/CI or a cloud secret manager). - Switch behavior based on which API you’re targeting (local vs Cloud) or an explicit env flag. How it works 1) Local development (Prefect Block) - Point your shell to your local API (e.g., `PREFECT_API_URL=http://localhost:4200/api`) so blocks are stored locally. - Bootstrap your local block from an env var (so secrets aren’t in code or the repo):
Copy code
# set in your local shell only (or a .env that is not committed)
export LOCAL_API_TOKEN=super-secret-for-local

# run once to create/update the local block
from prefect.blocks.system import Secret
import os

Secret(value=os.environ["LOCAL_API_TOKEN"]).save("my-api-token", overwrite=True)
- Your flows can then load
my-api-token
locally:
Copy code
from prefect.blocks.system import Secret
from prefect import flow

@flow
def run_local():
    token = Secret.load("my-api-token").get()
    ...
2) Cloud runs (no Cloud Secrets) - Point your runtime to Prefect Cloud via env:
Copy code
export PREFECT_API_URL=<https://api.prefect.cloud>
export PREFECT_API_KEY=...  # service account key in CI/worker env
- Provide runtime secrets via your preferred method: - Environment variables on the worker/CI (e.g.,
MY_API_TOKEN
) - Your cloud secret manager (AWS/GCP/Azure) using your standard SDK or a Prefect integration block that reads from those managers (the secret itself stays in your provider, not Prefect Cloud) 3) One small helper to unify both paths - Use a single accessor so your flow code never hardcodes secrets or cares where they came from:
Copy code
import os
from prefect.blocks.system import Secret

def get_api_token():
    api_url = os.getenv("PREFECT_API_URL", "")
    is_local = api_url.startswith("<http://localhost>") or os.getenv("PREFECT_ENV") == "local"

    if is_local:
        # local dev: use the local Prefect Block
        return Secret.load("my-api-token").get()

    # cloud/prod: read from env var or your secret manager
    # env var example:
    return os.environ["MY_API_TOKEN"]
- Flow usage:
Copy code
from prefect import flow

@flow
def run():
    token = get_api_token()
    ...
Tips - Keep the same block name locally and the same env var name in Cloud to simplify code. - Never commit .env files or scripts with literal secrets. - If you prefer a secret manager (AWS/GCP/Azure), fetch in the “cloud/prod” branch of
get_api_token()
using your org’s SDK/role setup. Your code still never stores the secret in Prefect Cloud. Useful links - Prefect Cloud UI: Open Prefect Cloud - Local UI (if running a local server): Open local UI If you share which secret source you use in Cloud (env vars, AWS Secrets Manager, etc.) and how you run flows (worker type/CI), I can provide a drop-in helper and bootstrap script tailored to that.