<@ULVA73B9P> I have a workflow that produces stack...
# ask-marvin
t
@Marvin I have a workflow that produces stacked arrays that may need to be accessed when it's finished. Currently I serialise them and pass them in the json output of the workflow if requested. However it can be too big to make sense in a json alongside the rest of the result. I was thinking of using artifacts instead. There is an image type artifact, can I use it to serve the stacked arrays or are there constraints as to the nature of the data? I don't care that it doesn't render in the UI, I'd just like it to be downloadable after the job is completed.
m
thought for 381.5 seconds
Short answer: don’t use an Image artifact for this. In Prefect 3.x, Image artifacts only take an image_url string; they don’t store binary content or base64. All artifact payloads are stored as JSON in Prefect’s database, so they’re best for small/medium metadata, not large arrays. Recommended pattern for large outputs (e.g., stacked NumPy arrays): - Persist the data to external storage (S3, GCS, Azure Blob, etc.) in a compact format (e.g., .npz or .zarr), and - Create a Link artifact pointing to that object (public URL or a presigned URL) so you can download it after the run. Why this is preferred: - Artifacts aren’t designed for large binary payloads; there’s no documented hard limit, but they’re stored as JSON in the API/DB and can become unwieldy or hit server limits. - Image artifacts require a URL and won’t accept raw bytes/base64. - Link artifacts are lightweight, render nicely in the UI, and give you a clean, clickable download. Example with S3 (Prefect AWS) This uploads a compressed NPZ to S3 and posts a Link artifact with a presigned URL:
Copy code
import io
import numpy as np
from datetime import timedelta

from prefect import flow
from prefect.artifacts import create_link_artifact
from prefect_aws.s3 import S3Bucket
from prefect_aws.credentials import AwsCredentials
import boto3

@flow
def produce_and_link_array():
    # 1) Produce your stacked array
    arr = np.random.rand(1000, 1000, 3)  # example

    # 2) Serialize to compressed NPZ in-memory
    buf = io.BytesIO()
    np.savez_compressed(buf, data=arr)
    buf.seek(0)

    # 3) Upload to S3
    bucket_name = "my-bucket"
    key = "artifacts/run-123/stacked_array.npz"

    aws_creds = AwsCredentials.load("my-aws-creds")  # or construct with your keys/role
    s3_block = S3Bucket(bucket=bucket_name, credentials=aws_creds)
    s3_block.upload_to_bucket(data=buf.getvalue(), key=key)

    # 4) Generate a presigned URL for download
    #    (You can also use s3_block.get_s3_client() if you prefer the block)
    session = boto3.Session(
        aws_access_key_id=aws_creds.aws_access_key_id,
        aws_secret_access_key=aws_creds.aws_secret_access_key,
        aws_session_token=aws_creds.aws_session_token,
        region_name=aws_creds.region_name,
    )
    s3_client = session.client("s3")
    url = s3_client.generate_presigned_url(
        "get_object",
        Params={"Bucket": bucket_name, "Key": key},
        ExpiresIn=3600,  # 1 hour
    )

    # 5) Create a Link artifact so it’s easy to grab from the Prefect UI
    create_link_artifact(
        link=url,
        link_text="Download stacked array (.npz)",
        key="stacked-array",  # optional but helpful for versioning/discoverability
        description="Compressed NPZ for run-123",
    )

if __name__ == "__main__":
    produce_and_link_array()
Notes and options: - If you need private access only, use a short-lived presigned URL (as above). If the data can be public, you can link to a public object URL. - For GCS/Azure, use the equivalent blocks (GcsBucket / AzureBlobStorage) and create a signed URL with those SDKs, then create a Link artifact. - For smaller, human-readable summaries, consider a Markdown or Table artifact with shapes/stats, and still link out to the full binary. Docs and references: - Artifacts overview: Artifacts concepts - How-to use artifacts: Artifacts how-to - Python API for artifacts: prefect.artifacts API - AWS S3 integration: Prefect AWS integration A couple of clarifiers to make sure we steer you right: - Where do you prefer to store the arrays (S3, GCS, Azure, something else)? - Roughly how large are these arrays (MB/GB)? - Do you want the download link to be public or time-limited/private?