https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Tags for flows seem to be more like a...
# ask-marvin
a
@Marvin Tags for flows seem to be more like a declarative set of enums rather than an unlimited set of arbitrary strings. Say we have an ID we want to propagate across multiple flows. What's the best mechanism to do this for Prefect? Ideally, we'd like to use the Prefect UI and filter by a UUID.
m
✅ thought for 166.4 seconds
Yes — tags are arbitrary strings in Prefect 3, not a fixed enum set. If your goal is: propagate one ID across multiple flows and be able to find those runs in the UI by that ID, the best practical pattern is: 1. Pass the UUID as a flow parameter 2. Also put it somewhere UI-searchable, usually the flow run name and/or a tag My recommendation: Best overall pattern - Use a parameter like 
correlation_id: str
to propagate the value between flows - Set the flow run name to include a short or full UUID if you want easy UI search - Optionally add a tag like 
cid:<uuid>
if you want grouping/filtering by tags Example:
Copy code
python
from prefect import flow

@flow(flow_run_name="job-{correlation_id}")
def child_flow(correlation_id: str):
    print(correlation_id)

@flow
def parent_flow():
    correlation_id = "123e4567-e89b-12d3-a456-426614174000"
    child_flow(correlation_id=correlation_id)
Why this works: - Parameters are the cleanest way to propagate the ID across flow boundaries - Flow run name is visible and searchable in the UI - Tags can also carry arbitrary strings like 
cid:123e...
, but tags are better for low-cardinality grouping than high-cardinality per-run unique IDs A few tradeoffs: Using tags for UUIDs - Technically valid: tags are just strings - But operationally, unique-per-run tags can get noisy - Tags tend to be more useful for things like 
customer:acme
, 
env:prod
, 
pipeline:billing
Using run names - Best if you specifically want “paste UUID into the UI and find the run” - Very human-friendly - You can include both business context and the UUID, e.g. 
invoice-sync-<uuid>
Using parameters only - Great for propagation - But less convenient if your main need is UI filtering/search So if I were designing this, I’d do:
Copy code
python
from prefect import flow

@flow(flow_run_name="child-{correlation_id}")
def child_flow(correlation_id: str):
    pass

@flow(flow_run_name="parent-{correlation_id}")
def parent_flow(correlation_id: str):
    child_flow(correlation_id=correlation_id)
If you want tags too:
Copy code
python
from prefect import flow

@flow(tags=["pipeline:example"])
def child_flow(correlation_id: str):
    pass
Then update tags at runtime if needed via the client API, but for simple propagation I would not choose tags as the primary transport. Bottom line: Use parameters for propagation and flow run names for UI lookup. Use tags only if you also want tag-based grouping. Relevant docs: - Flows - Deployments - Flow API reference If you want, I can show a concrete pattern for: - parent flow -> child flow propagation - deployment-triggered runs - or updating tags/names dynamically at runtime.
a
Show me how to change a flow run name if we're triggering a deployment.
m
✅ thought for 6.5 seconds
I can’t access that PR directly from the link alone here. If you want, paste: - the relevant Prefect code snippet - how you’re launching downstream flows - and what kind of UI filtering you need and I can suggest the best pattern for that exact PR. If your intent is “we have a request/job UUID in this PR and want it visible across all related Prefect runs,” the shortest recommendation is still: - pass it as a 
correlation_id
parameter everywhere - include it in 
flow_run_name
- optionally add a tag like 
cid:<uuid>
if tag filtering matters Example:
Copy code
python
from prefect import flow

@flow(flow_run_name="ingest-{correlation_id}")
def ingest_flow(correlation_id: str):
    ...

@flow(flow_run_name="transform-{correlation_id}")
def transform_flow(correlation_id: str):
    ...

@flow(flow_run_name="pipeline-{correlation_id}")
def pipeline(correlation_id: str):
    ingest_flow(correlation_id=correlation_id)
    transform_flow(correlation_id=correlation_id)
If you paste the diff, I’ll tailor the recommendation to the code in that PR.
a
@Marvin Adam Liu [5:21 PM] Show me how to change a flow run name if we're triggering a deployment.
m
✅ thought for 113.6 seconds
Yep — when you trigger a deployment, pass 
flow_run_name=
. Example:
Copy code
python
from prefect.deployments import run_deployment

flow_run = run_deployment(
    name="my-flow/my-deployment",
    flow_run_name="my-custom-run-name",
    timeout=0,
)

print(flow_run.name)
If you want to include a UUID:
Copy code
python
from prefect.deployments import run_deployment

correlation_id = "123e4567-e89b-12d3-a456-426614174000"

flow_run = run_deployment(
    name="my-flow/my-deployment",
    flow_run_name=f"ingest-{correlation_id}",
    timeout=0,
)
A couple notes: - This sets the created flow run's name at trigger time - In Prefect 3, deployments themselves do not have a 
flow_run_name
template field - If you want dynamic naming from inside the flow, the flow can also rename itself later via the client API Related docs: - Deployment flow run APIs - Flows If you want, I can also show: - async 
run_deployment
- how to do this from the lower-level client - or how to combine 
flow_run_name
with a propagated 
correlation_id
parameter.
a
@Marvin honestly this also kind of isn't useful for the general pattern let's say I want to associate N different UUID based terms with a run- say a message ID, a chat ID, and so on we want to be able to search, to answer something like "what are all Prefect runs that are associated with this chat?" is there a way to do this natively in Prefect? the only thing I can think of so far is to emit some kind of message for external telemetry (and I do not need advice on how to do that, so save the tokens)
m
✅ thought for 14.9 seconds
Short answer: not really, not natively in the way you want. If your requirement is: - attach multiple arbitrary high-cardinality identifiers to a run - later search Prefect for runs by any one of those identifiers, like 
chat_id = ...
- do that cleanly in the Prefect UI/API then Prefect 3 does not currently have a first-class “search arbitrary indexed metadata on flow runs” feature. What exists natively: - Run name — searchable, but basically one string - Tags — arbitrary strings, and filterable, but awkward for lots of unique IDs - Parameters — stored, but not generally a practical indexed search surface for this use case - Labels — structured metadata exists, but not a strong native answer for “query all runs where `chat_id = X`” - Artifacts/events — useful for observability, but that’s drifting into the external-telemetry shape you explicitly don’t want So for your exact pattern — multiple UUID-ish foreign keys, later searchable by any of them — Prefect is not acting like a metadata index today. Closest native workaround Use tags with namespaced values, e.g. - 
chat_id:<uuid>
- 
message_id:<uuid>
- 
conversation_id:<uuid>
That is the nearest built-in mechanism for “attach arbitrary identifiers and later filter runs by them.” But the caveats are exactly the ones you’re worried about: - very high-cardinality tags - potentially noisy UI/filter UX - not especially elegant as a general metadata model So I’d frame it this way: - If you need a small-ish pragmatic solution inside Prefect only: use namespaced tags - If you need a real searchable association store for arbitrary IDs: Prefect doesn’t natively provide that today So the honest answer is: No, Prefect does not currently have a native, first-class, indexed metadata field for querying runs by arbitrary business identifiers like 
chat_id
across many runs. If helpful, I can give a very short opinionated recommendation on whether namespaced tags are “good enough” here, and where I’d draw the line before calling it the wrong tool.
a
@Marvin sure but I think I know the answer
m
✅ thought for 5.6 seconds
Yeah — I think you do too :) My opinionated line: - Good enough: a handful of identifier types, moderate volume, mostly human investigation in the UI - use tags like 
chat_id:<uuid>
/ 
message_id:<uuid>
- Wrong tool: lots of high-cardinality IDs, routine lookup by arbitrary ID, or this becomes part of product/runtime behavior - Prefect is orchestration metadata, not a general association index So if the question is “can Prefect be the system of record for querying runs by arbitrary business IDs?” my answer is no. If the question is “can we hack this in well enough for operator debugging?” my answer is yes, with namespaced tags, but expect it to feel hacky.
Open in Slack
PreviousNext
Powered by