<@ULVA73B9P> how to use materialize to log output of my flow on Prefect Cloud with Prefect 3? Is it ...
f
@Marvin how to use materialize to log output of my flow on Prefect Cloud with Prefect 3? Is it possible to use materialize without backend to store data file?
m
thought for 378.2 seconds
Great question. In Prefect 3, you use Assets to “materialize” and log the outputs of your flow to Prefect Cloud. This gives you an Assets graph, lineage, and metadata in the UI. And yes—you can use materialize purely for tracking/logging without any storage backend. Key points - @materialize logs an asset event to Prefect Cloud; it does not write any files for you. - The asset key is just an identifier (URI-like). You can use gs://, s3://, file://, or even a custom “tracking://” scheme for pure observability. - If you want an actual file in GCS, write it yourself in the task (via google-cloud-storage or prefect-gcp) and use a matching gs:// asset key. Minimal examples 1) Tracking/logging only (no storage backend)
Copy code
from prefect import flow
from prefect.assets import materialize, add_asset_metadata

@materialize("<tracking://customers/daily>")
def build_customers(date: str):
    # Do your compute here...
    add_asset_metadata(
        "<tracking://customers/daily>",
        {"date": date, "rows": 123456, "status": "ok"}
    )

@flow
def pipeline(date: str):
    build_customers(date)

if __name__ == "__main__":
    pipeline("2025-01-01")
This creates an asset in Prefect Cloud at key tracking://customers/daily and attaches your metadata—no file storage required. 2) Track a real file in GCS (you write the file yourself)
Copy code
from prefect import flow
from prefect.assets import materialize, add_asset_metadata
from google.cloud import storage

@materialize("<gs://my-bucket/datasets/daily/customers.csv>")
def build_customers(date: str):
    # Create and upload your data
    client = storage.Client()
    bucket = client.bucket("my-bucket")
    key = f"datasets/daily/customers_{date}.csv"
    bucket.blob(key).upload_from_string("id,name\n1,alice")

    # Log runtime metadata to the asset in Cloud
    add_asset_metadata(
        "<gs://my-bucket/datasets/daily/customers.csv>",
        {"date": date, "blob": key, "rows": 1}
    )

@flow
def pipeline(date: str):
    build_customers(date)

if __name__ == "__main__":
    pipeline("2025-01-01")
Optional: if you want to actually point the asset key at the specific blob (per-date), you can set the asset key dynamically at call time:
Copy code
from prefect.assets import materialize

@materialize("<gs://placeholder>")  # default; will be overridden
def build_customers(date: str): ...

@flow
def pipeline(date: str):
    dynamic = build_customers.with_options(
        assets=[f"<gs://my-bucket/datasets/daily/customers_{date}.csv>"]
    )
    dynamic(date)
Also useful: regular logs to Cloud
Copy code
from prefect import flow, task
from prefect.logging import get_run_logger

@task
def compute():
    logger = get_run_logger()
    <http://logger.info|logger.info>("This message appears in Prefect Cloud logs.")
    return 42

@flow
def pipeline():
    return compute()
Where this shows up in Cloud - You’ll see your assets in the Assets view with status (succeeded/failed), history, and any metadata you add. - Using “tracking://…” is perfectly fine when you want observability without storing a file anywhere. Docs that walk through this - Assets concepts: Assets concepts - Assets how-to: Assets how-to - Advanced assets: Advanced assets - Logging to Cloud: Logging customization Since you mentioned you’re on Prefect 2 today and GCP: - Assets and @materialize are a Prefect 3 feature. To use them, run your flow with Prefect 3 against the same Cloud workspace. - If you do want to persist files in GCS from Prefect, you can use either google-cloud-storage (as above) or prefect-gcp utilities (e.g., GcsBucket or cloud_storage_upload_blob_from_string). Would you like a short, ready-to-run example wired to your GCS bucket name and project, or are you planning to use tracking-only assets first?
f
@Marvin what's the difference between materialize and create_markdown_artifact?
m
thought for 43.2 seconds
Short answer: - materialize (Prefect 3) declares that your task produced or updated a data asset and logs lineage/metadata to the Assets graph in Prefect Cloud. - create_markdown_artifact creates a human-readable UI artifact (markdown) shown on run pages/artifacts list; it’s for reporting, not lineage. Key differences - Purpose - materialize: Data lineage and observability for datasets, tables, files, etc. Shows up in the Assets graph with dependencies and health. - create_markdown_artifact: Human-friendly output (reports, summaries) rendered in the UI. - What is stored - materialize: Stores an event with the asset key (e.g., gs://…, s3://…, snowflake://…, tracking://…) and any metadata you add. Prefect does not write your data. - create_markdown_artifact: Stores the markdown content in Prefect Cloud for display. - Where it appears in the UI - materialize: Assets view/graph, with lineage, latest status, and metadata. - create_markdown_artifact: Run page “Artifacts” section and the global Artifacts list. - Dependencies - materialize: Can infer or declare upstream asset dependencies for lineage. - create_markdown_artifact: No lineage; it’s just a UI artifact tied to the run. - Storage backend required? - materialize: No—asset keys are identifiers only; you write data yourself if you want it persisted (e.g., to GCS). - create_markdown_artifact: Content is stored in Prefect Cloud automatically. - Version note - materialize is a Prefect 3 feature. Artifacts exist in both Prefect 2 and 3. Minimal examples - materialize (logs an asset, no file required)
Copy code
from prefect import flow
from prefect.assets import materialize, add_asset_metadata

@materialize("<tracking://customers/daily>")
def build_customers(date: str):
    # Do work...
    add_asset_metadata("<tracking://customers/daily>", {"date": date, "rows": 123456})

@flow
def pipeline(date: str):
    build_customers(date)
- create_markdown_artifact (UI report)
Copy code
from prefect import flow
from prefect.artifacts import create_markdown_artifact

@flow
def pipeline():
    md = "# Daily Summary\n\n- Rows processed: 123,456\n- Status: ✅"
    create_markdown_artifact(key="daily-summary", markdown=md, description="Run summary")
You can use both together: materialize to log/update the asset and lineage, and create_markdown_artifact to attach a rendered report for humans. Docs - Assets: Assets concepts, Assets how-to, Advanced assets - Artifacts: Artifacts how-to, Artifacts concepts Since you’re on Prefect 2 Cloud today: materialize is available in Prefect 3. If you stay on 2.x, you can still use create_markdown_artifact for reporting; upgrading to 3 unlocks the Assets graph and @materialize.