@Marvin: When persisting results to S3, how long are they cached for? Will they exist after Prefect has removed the flow called it, after Prefect’s flow run retention window?

✅ thought for 244.1 seconds

we won't delete stuff from your blob

Short answer: - Persisted results in S3 are not auto-deleted by Prefect. They remain until you or an S3 lifecycle rule deletes them. - Prefect’s flow run retention (Cloud or self-hosted) only deletes database records/metrics; it does not touch S3 objects. - “Cache expiration” controls whether a previously persisted result can be reused, but it does not delete the underlying S3 object. Details (Prefect 3.x): - Two “cache” layers: - In-memory cache: exists only for the duration of a single flow run process; it’s evicted when the run ends. - Persistent results (e.g., S3): written when

persist_result=True

(or enabled via settings). You can set

cache_expiration

to influence reuse, but expired results are just treated as a cache miss on read—they are not removed from S3. - If you want time-based deletion, configure an AWS S3 Lifecycle Policy on the bucket/prefix where Prefect writes results. - Deleting/retaining flow runs in Prefect Cloud does not affect S3 objects. Example (won’t delete S3 after 7 days—just stops reusing it):

from datetime import timedelta
from prefect import task, flow

@task(persist_result=True, cache_expiration=timedelta(days=7))
def compute():
    return "hello"

@flow
def my_flow():
    return compute()

if __name__ == "__main__":
    my_flow()

Docs for deeper reading: - Results and persistence: Prefect 3 results - Caching and expiration semantics: Caching concepts and Advanced caching - Flow run/DB retention (separate from S3): Database maintenance If helpful, I can suggest a storage key prefix strategy and an S3 lifecycle rule to auto-expire results after your desired window. Would you like that?

however, the prefect server will no longer go looking for a certain key after some time if you configure expiration

@Nate: Awesome, thank you! That’s what I was hoping

Well, mostly hoping

as in, ideally we'd clear out your old blob stuff?

I’m admittedly still thinking it through, so it’s unclear. I’ve been recording metadata per runs, so we have long-term storage of what happened. This has been helpful for reporting/auditing, within our team, and to other stakeholders. E.g. “Document X has Y ready for it”. I’m also getting this from results, but sometimes those results are quite hefty. E.g. “Inference ran on document X for topics A, B, C”. I could get that from the inference results themselves in our blog storage, but there’s that heft to it.