<@ULVA73B9P> Can I use assets, e.g. with @materia...
# ask-marvin
c
@Marvin Can I use assets, e.g. with @materialise or other ways, but with the data URI decided run time? I have some data a task will produce, but the name of the data will be known only in the mid of the task execution Specifically I have a rest endpoint. I will post to the endpoint, which will create a document id, e.g. http://myrest.tld/api/documents/123 this URL will be the one that is materialised by the task next task run will be a different URL/document. I would like to use the assets mechanism to signal data has been created
m
thought for 391.0 seconds
Yes — in Prefect 3.x you can materialize assets whose URI is only known at runtime. The easiest pattern is: - Wrap the task with
@prefect.assets.materialize()
to make it a “materializing task” - Inside the task, once you have the final URI, register it with
prefect.assets.add_asset_metadata(...)
- Optionally include
AssetProperties
(name/description/owners) and any extra metadata Prefect will emit the asset materialization event and associate it with the running task/flow. Each run can register a different URI. Example:
Copy code
from prefect import flow
from prefect.assets import materialize, add_asset_metadata, Asset, AssetProperties
import httpx

@materialize(by="document-api")
def create_document(payload: dict) -> str:
    # Create the document and get its ID/URL at runtime
    r = <http://httpx.post|httpx.post>("<http://myrest.tld/api/documents>", json=payload)
    r.raise_for_status()
    doc_id = r.json()["id"]
    doc_url = f"<http://myrest.tld/api/documents/{doc_id}>"

    # Register the asset that was created at runtime
    # The key must be a valid URI (http/https/s3/etc.)
    asset = Asset(
        key=doc_url,
        properties=AssetProperties(
            name=f"document-{doc_id}",
            description="Document created via REST",
            owners=["<mailto:data-eng@example.com|data-eng@example.com>"]
        )
    )
    add_asset_metadata(asset, {
        "doc_id": doc_id,
        "status_code": r.status_code,
    })

    return doc_url


@flow
def my_flow():
    url = create_document({"title": "Hello", "body": "World"})
    # downstream work that uses `url`...
Notes: -
Asset.key
must be a valid URI; your REST URL works. -
add_asset_metadata
can be called multiple times to attach more metadata as you go. - If you don’t need the decorator, you can still call
add_asset_metadata
in any task; using
@materialize()
ensures the UI clearly shows “materialized by <task>” and ties the event to that task run. - If you just want a clickable link (not an asset), you can alternatively emit an artifact:
Copy code
from prefect.artifacts import create_link_artifact
  create_link_artifact(link=doc_url, description="Created document")
Does this match your use case? If you share your Prefect version and a minimal snippet of your task, I can tailor the example further.
c
@Marvin using your code, materisalize fails with `TypeError: materialize requires at least one asset argument, e.g.
@materialize(asset)
It seems that the decorator needs a further arg, which is the asset
m
thought for 436.2 seconds
You’re right — my earlier example was missing the required asset argument. In Prefect 3.x,
@materialize
must be given at least one asset at decoration time. Because your asset key (the REST URL) is only known mid-run, you have a few workable patterns: Option A — Two-step pattern with a tiny, dynamically-defined materialization task - First task creates the document and returns the URL - Then define a small task (at runtime) that materializes that specific URL as an asset and attaches any metadata Example:
Copy code
from prefect import flow, task
from prefect.assets import materialize, Asset, AssetProperties, add_asset_metadata
import httpx

@task
def create_document(payload: dict) -> dict:
    r = <http://httpx.post|httpx.post>("<http://myrest.tld/api/documents>", json=payload)
    r.raise_for_status()
    doc_id = r.json()["id"]
    return {"id": doc_id, "url": f"<http://myrest.tld/api/documents/{doc_id}>"}

def materialize_runtime_asset(url: str, *, name: str | None = None, description: str | None = None, owners: list[str] | None = None, metadata: dict | None = None):
    asset = Asset(
        key=url,
        properties=AssetProperties(name=name, description=description, owners=owners)
    )

    @materialize(asset, by="document-api", task_run_name=lambda: f"materialize {url}")
    def _emit():
        if metadata:
            add_asset_metadata(asset, metadata)
        return url

    return _emit()

@flow
def my_flow():
    out = create_document({"title": "Hello", "body": "World"})
    materialize_runtime_asset(
        out["url"],
        name=f"document-{out['id']}",
        description="Document created via REST",
        owners=["<mailto:data-eng@example.com|data-eng@example.com>"],
        metadata={"doc_id": out["id"]}
    )
Why this works: -
@materialize(asset)
requires a concrete asset at decoration time; by defining the tiny
_emit
task after you know the URL, you satisfy that requirement. - The asset materialization event is tied to the
_emit
task run and will show in the UI. - You can attach arbitrary metadata with
add_asset_metadata
. Option B — Use artifacts if you just need a clickable link/notification (not a first-class asset)
Copy code
from prefect.artifacts import create_link_artifact

create_link_artifact(
    link=doc_url,
    description="Created document via REST"
)
Option C — “Namespace asset” plus metadata (if per-document assets aren’t required) - Materialize a single static asset key like
<http://myrest.tld/api/documents>
and attach per-run metadata that includes the specific
doc_url
and
doc_id
. This gives you a single asset with rich run-by-run metadata, but doesn’t create a unique asset per document. Docs for reference: - Assets concept - Assets how-to If you want each document to be its own asset (distinct key per run), Option A is the most direct approach. Want me to adapt this pattern to your exact task/flow structure?
j
❤️ 1
c
@Jesse thanks, I was looking for something like that indeed. I thought I read something about it in the past, but couldn't find it back
🙇 1