Theom
11/06/2025, 9:09 AMMarvin
11/06/2025, 9:09 AMMarvin
11/06/2025, 9:16 AMimport io
import numpy as np
from datetime import timedelta
from prefect import flow
from prefect.artifacts import create_link_artifact
from prefect_aws.s3 import S3Bucket
from prefect_aws.credentials import AwsCredentials
import boto3
@flow
def produce_and_link_array():
# 1) Produce your stacked array
arr = np.random.rand(1000, 1000, 3) # example
# 2) Serialize to compressed NPZ in-memory
buf = io.BytesIO()
np.savez_compressed(buf, data=arr)
buf.seek(0)
# 3) Upload to S3
bucket_name = "my-bucket"
key = "artifacts/run-123/stacked_array.npz"
aws_creds = AwsCredentials.load("my-aws-creds") # or construct with your keys/role
s3_block = S3Bucket(bucket=bucket_name, credentials=aws_creds)
s3_block.upload_to_bucket(data=buf.getvalue(), key=key)
# 4) Generate a presigned URL for download
# (You can also use s3_block.get_s3_client() if you prefer the block)
session = boto3.Session(
aws_access_key_id=aws_creds.aws_access_key_id,
aws_secret_access_key=aws_creds.aws_secret_access_key,
aws_session_token=aws_creds.aws_session_token,
region_name=aws_creds.region_name,
)
s3_client = session.client("s3")
url = s3_client.generate_presigned_url(
"get_object",
Params={"Bucket": bucket_name, "Key": key},
ExpiresIn=3600, # 1 hour
)
# 5) Create a Link artifact so it’s easy to grab from the Prefect UI
create_link_artifact(
link=url,
link_text="Download stacked array (.npz)",
key="stacked-array", # optional but helpful for versioning/discoverability
description="Compressed NPZ for run-123",
)
if __name__ == "__main__":
produce_and_link_array()
Notes and options:
- If you need private access only, use a short-lived presigned URL (as above). If the data can be public, you can link to a public object URL.
- For GCS/Azure, use the equivalent blocks (GcsBucket / AzureBlobStorage) and create a signed URL with those SDKs, then create a Link artifact.
- For smaller, human-readable summaries, consider a Markdown or Table artifact with shapes/stats, and still link out to the full binary.
Docs and references:
- Artifacts overview: Artifacts concepts
- How-to use artifacts: Artifacts how-to
- Python API for artifacts: prefect.artifacts API
- AWS S3 integration: Prefect AWS integration
A couple of clarifiers to make sure we steer you right:
- Where do you prefer to store the arrays (S3, GCS, Azure, something else)?
- Roughly how large are these arrays (MB/GB)?
- Do you want the download link to be public or time-limited/private?