Samuel Hinton
08/29/2025, 6:02 AMjson
artifact to Prefect?Marvin
08/29/2025, 6:02 AMMarvin
08/29/2025, 6:04 AMfrom prefect import flow
from prefect.artifacts import create_markdown_artifact
import json
@flow
def my_flow():
payload = {"a": 1, "b": [1, 2, 3]}
md = "```json\n" + json.dumps(payload, indent=2) + "\n```"
create_markdown_artifact(markdown=md, key="example-json", description="Sample JSON payload")
- If your JSON is a list of objects (tabular)
Use a Table artifact; the UI will render it as a table.
from prefect.artifacts import create_table_artifact
rows = [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"},
]
create_table_artifact(table=rows, key="users-table", description="Users as table")
- If you want to upload a file and link to it (best for large payloads)
Store the JSON in object storage (S3/GCS/Azure/etc.) and create a Link artifact to it.
from prefect.artifacts import create_link_artifact
s3_url = "<https://my-bucket.s3.amazonaws.com/path/data.json>"
create_link_artifact(link=s3_url, link_text="Download data.json", key="data-json-link")
Notes
- Artifacts are stored in Prefect’s database; for large JSON payloads it’s better to store the file in external storage and use a link artifact.
- Async variants exist (e.g., acreate_markdown_artifact
) if you’re in async code.
- The optional key
helps you identify artifacts across runs.
Docs
- Artifacts in Prefect 3.x
- Artifacts API (source)
Quick question to tailor the suggestion: Do you want the JSON contents visible directly in the UI, or just a downloadable link? And roughly how large is the JSON?Nate
08/29/2025, 3:13 PMMarvin
08/29/2025, 3:13 PMMarvin
08/29/2025, 3:17 PMprefect variable ls
prefect variable set MY_SETTING "value"
prefect variable get MY_SETTING
prefect variable unset MY_SETTING
You can add --overwrite
to update an existing value.
- Python
from prefect.variables import Variable
# Set (optionally with tags)
Variable.set("MY_SETTING", "value", tags=["prod"], overwrite=True)
# Get with default
limit = Variable.get("MAX_CONCURRENCY", default=8)
# Unset
Variable.unset("MY_SETTING")
Notes and best practices
- Keep Variables for non-sensitive data; store secrets in blocks and load them at runtime.
- Use defaults in code so flows still run if a Variable isn’t set.
- For deployments, you can template values from Variables and env vars. If you tell me which deployment field you want to parameterize (e.g., image tag, env var, job variable), I’ll share the exact snippet.
Docs
- Variables in Prefect 3
- Storing secrets (use blocks, not Variables)
If you share what you’re configuring (e.g., ECS task env var, worker queue name, Dask thread count), I can suggest whether a Variable, env var, or block is the better fit and show a concrete example.Nate
08/29/2025, 3:24 PMwhat would be the best way to upload aartifact to Prefect?json
Marvin
08/29/2025, 3:24 PMMarvin
08/29/2025, 3:26 PMimport json
from prefect import flow
from prefect.artifacts import create_markdown_artifact
@flow
def report():
payload = {"job_id": "123", "status": "ok", "metrics": {"rows": 42}}
md = "### Job output\n\n```json\n" + json.dumps(payload, indent=2) + "\n```"
create_markdown_artifact(
markdown=md,
key="job-output",
description="Raw JSON output"
)
- Link to stored JSON (e.g., S3 presigned URL):
from prefect.artifacts import create_link_artifact
# After you upload to storage and get a URL:
url = "<https://example.com/path/to/object.json>" # presigned or public URL
create_link_artifact(
link=url,
link_text="Download JSON output",
key="job-output-json",
description="Raw JSON stored externally"
)
- Table from JSON array of objects:
from prefect.artifacts import create_table_artifact
rows = [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 28}]
create_table_artifact(
table=rows,
key="people",
description="People table from JSON"
)
Notes
- Use key
to group/track a logical artifact across runs.
- For large payloads, prefer the Link approach and optionally pair it with a small Markdown summary in the UI.
Refs
- Artifacts API reference
If you share the rough size and whether you want it rendered in the UI or just retrievable, I can recommend the exact approach.Nate
08/29/2025, 3:26 PMMarvin
08/29/2025, 3:26 PMMarvin
08/29/2025, 3:29 PMVariable.get()
(e.g., parameters, toggles)
- Use Artifacts when:
- You want to display/run-scope output in the UI and track it across runs
- You want a visible record (Markdown/table/link/image) on the flow run page
- You might store large JSON externally and link to it
Examples
- Variable storing a JSON dict:
from prefect.variables import Variable
# set once
Variable.set("MY_JSON_CONFIG", {"threshold": 0.9, "regions": ["us", "eu"]}, overwrite=True)
# later, read it
cfg = Variable.get("MY_JSON_CONFIG", default={})
- “JSON artifact” shown in the UI (Markdown code block):
import json
from prefect.artifacts import create_markdown_artifact
payload = {"job_id": "123", "status": "ok"}
md = "```json\n" + json.dumps(payload, indent=2) + "\n```"
create_markdown_artifact(markdown=md, key="job-output")
- Large JSON stored elsewhere → link artifact:
from prefect.artifacts import create_link_artifact
create_link_artifact(link="<https://example.com/my.json>", link_text="Download JSON", key="job-output-json")
TL;DR: Variables are great for mutable config (JSON is fine). Artifacts are for run outputs you want visible and tracked in the UI.
Links
- Variables
- Artifacts APISamuel Hinton
08/31/2025, 11:07 PM