Hey <@ULVA73B9P> what would be the best way to upl...
# ask-community
s
Hey @Marvin what would be the best way to upload a
json
artifact to Prefect?
m
thought for 103.7 seconds
Great question! In Prefect 3.x there isn’t a dedicated “JSON artifact” type, but you’ve got a few good options depending on how you want the JSON to appear in the UI: Options - Show the JSON inline in the UI (recommended for small/medium payloads) Use a Markdown artifact with a fenced JSON code block for nice formatting.
Copy code
from prefect import flow
  from prefect.artifacts import create_markdown_artifact
  import json

  @flow
  def my_flow():
      payload = {"a": 1, "b": [1, 2, 3]}
      md = "```json\n" + json.dumps(payload, indent=2) + "\n```"
      create_markdown_artifact(markdown=md, key="example-json", description="Sample JSON payload")
- If your JSON is a list of objects (tabular) Use a Table artifact; the UI will render it as a table.
Copy code
from prefect.artifacts import create_table_artifact

  rows = [
      {"id": 1, "name": "Alice"},
      {"id": 2, "name": "Bob"},
  ]
  create_table_artifact(table=rows, key="users-table", description="Users as table")
- If you want to upload a file and link to it (best for large payloads) Store the JSON in object storage (S3/GCS/Azure/etc.) and create a Link artifact to it.
Copy code
from prefect.artifacts import create_link_artifact

  s3_url = "<https://my-bucket.s3.amazonaws.com/path/data.json>"
  create_link_artifact(link=s3_url, link_text="Download data.json", key="data-json-link")
Notes - Artifacts are stored in Prefect’s database; for large JSON payloads it’s better to store the file in external storage and use a link artifact. - Async variants exist (e.g.,
acreate_markdown_artifact
) if you’re in async code. - The optional
key
helps you identify artifacts across runs. Docs - Artifacts in Prefect 3.x - Artifacts API (source) Quick question to tailor the suggestion: Do you want the JSON contents visible directly in the UI, or just a downloadable link? And roughly how large is the JSON?
n
@Marvin perhaps variables would be appropriate here?
m
thought for 239.6 seconds
They might! To make sure we’re on the same page—what value(s) are you thinking about parameterizing? For example: environment names, S3 bucket prefixes, parallelism limits, feature flags, etc. Depending on the sensitivity and how you’ll use them, Variables can be a good fit. Quick overview (Prefect 3.x): - What Variables are: workspace-scoped, mutable key/value settings for non-sensitive config (strings/numbers/booleans). - When to use: toggles, environment names, sizes/limits, dataset identifiers. - When not to use: secrets or credentials — use Secret/Credential blocks instead. How to use Variables - UI - Navigate to Variables in your workspace, create/update values there. - CLI
Copy code
prefect variable ls
  prefect variable set MY_SETTING "value"
  prefect variable get MY_SETTING
  prefect variable unset MY_SETTING
You can add
--overwrite
to update an existing value. - Python
Copy code
from prefect.variables import Variable

  # Set (optionally with tags)
  Variable.set("MY_SETTING", "value", tags=["prod"], overwrite=True)

  # Get with default
  limit = Variable.get("MAX_CONCURRENCY", default=8)

  # Unset
  Variable.unset("MY_SETTING")
Notes and best practices - Keep Variables for non-sensitive data; store secrets in blocks and load them at runtime. - Use defaults in code so flows still run if a Variable isn’t set. - For deployments, you can template values from Variables and env vars. If you tell me which deployment field you want to parameterize (e.g., image tag, env var, job variable), I’ll share the exact snippet. Docs - Variables in Prefect 3 - Storing secrets (use blocks, not Variables) If you share what you’re configuring (e.g., ECS task env var, worker queue name, Dask thread count), I can suggest whether a Variable, env var, or block is the better fit and show a concrete example.
n
@Marvin nah dude im just talking about the original ask
what would be the best way to upload a
json
artifact to Prefect?
m
thought for 106.4 seconds
Got it — there isn’t a dedicated “JSON artifact” type in Prefect 3.x, but here are the best patterns: - Small/medium JSON you want visible in the UI: embed it in a Markdown artifact as a JSON code block. - Large JSON or binary: store it (e.g., S3/GCS) and create a Link artifact to it. - Tabular JSON (list of objects): use a Table artifact. Examples - JSON rendered in UI (Markdown):
Copy code
import json
from prefect import flow
from prefect.artifacts import create_markdown_artifact

@flow
def report():
    payload = {"job_id": "123", "status": "ok", "metrics": {"rows": 42}}
    md = "### Job output\n\n```json\n" + json.dumps(payload, indent=2) + "\n```"
    create_markdown_artifact(
        markdown=md,
        key="job-output",
        description="Raw JSON output"
    )
- Link to stored JSON (e.g., S3 presigned URL):
Copy code
from prefect.artifacts import create_link_artifact

# After you upload to storage and get a URL:
url = "<https://example.com/path/to/object.json>"  # presigned or public URL
create_link_artifact(
    link=url,
    link_text="Download JSON output",
    key="job-output-json",
    description="Raw JSON stored externally"
)
- Table from JSON array of objects:
Copy code
from prefect.artifacts import create_table_artifact

rows = [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 28}]
create_table_artifact(
    table=rows,
    key="people",
    description="People table from JSON"
)
Notes - Use
key
to group/track a logical artifact across runs. - For large payloads, prefer the Link approach and optionally pair it with a small Markdown summary in the UI. Refs - Artifacts API reference If you share the rough size and whether you want it rendered in the UI or just retrievable, I can recommend the exact approach.
n
@Marvin dude need some coffee or what? im saying that variables are essentially JSON artifacts bruv
m
thought for 166.4 seconds
I get what you mean: both can hold JSON-ish data. But they’re not the same thing in Prefect. Quick compare - Use Variables when: - You want mutable, workspace-scoped config (JSON-serializable: dict/list/str/etc.) - Not tied to a specific run, not meant for UI rendering - You’ll read it via
Variable.get()
(e.g., parameters, toggles) - Use Artifacts when: - You want to display/run-scope output in the UI and track it across runs - You want a visible record (Markdown/table/link/image) on the flow run page - You might store large JSON externally and link to it Examples - Variable storing a JSON dict:
Copy code
from prefect.variables import Variable

# set once
Variable.set("MY_JSON_CONFIG", {"threshold": 0.9, "regions": ["us", "eu"]}, overwrite=True)

# later, read it
cfg = Variable.get("MY_JSON_CONFIG", default={})
- “JSON artifact” shown in the UI (Markdown code block):
Copy code
import json
from prefect.artifacts import create_markdown_artifact

payload = {"job_id": "123", "status": "ok"}
md = "```json\n" + json.dumps(payload, indent=2) + "\n```"
create_markdown_artifact(markdown=md, key="job-output")
- Large JSON stored elsewhere → link artifact:
Copy code
from prefect.artifacts import create_link_artifact
create_link_artifact(link="<https://example.com/my.json>", link_text="Download JSON", key="job-output-json")
TL;DR: Variables are great for mutable config (JSON is fine). Artifacts are for run outputs you want visible and tracked in the UI. Links - Variables - Artifacts API
s
For context @Nate - we're trying to pass results from one flow to another, but its difficult configuring results persistence as academic workloads dont have cloud resources to share, and in terms of file systems different flows may be executed on entirely difference centres. Because the result is just a json dict with just a few elements, I was trying out using an artifact with a specific key to effective use the prefect artifact table as the result storage.