If you're using local filesystem result storage in Prefect 3.x, the default retention is effectively forever. Key points: -

cache_expiration

defaults to

None

- That means cached results do not expire unless you set an expiration explicitly - Prefect does not automatically clean up old/expired local result files from disk So with local storage, result files remain on disk until: - you delete them manually, or - you point Prefect at a different storage location Example with explicit expiration:

python
from datetime import timedelta
from prefect import task

@task(cache_expiration=timedelta(hours=24))
def my_task():
    return "hello"

Without

cache_expiration=...

, there is no time-based retention. A couple related details: - Default local result storage path is typically

~/.prefect/storage/

- You can configure it with

PREFECT_LOCAL_STORAGE_PATH

- Expired cache entries are just *ignored for reuse*; they are not automatically deleted Relevant docs: - Results - Task caching - Task API reference If you want, I can also show you how to set a global cleanup strategy for local result files.

I’d recommend not relying on local filesystem storage alone if disk space / inode usage is a real concern. Best options, in order: 1. Set

cache_expiration

on cached tasks This prevents stale cache entries from being reused forever.

python
from datetime import timedelta
from prefect import task

@task(cache_expiration=timedelta(days=1))
def my_task(x):
    return x

But important: this does not delete files. It only makes old cache entries invalid. 2. Add an external cleanup job for local storage If you stay on local filesystem storage, the safest pattern is to periodically delete old files from your Prefect storage directory, for example

~/.prefect/storage/

or your configured

PREFECT_LOCAL_STORAGE_PATH

. For example, a cron job like:

bash
find ~/.prefect/storage -type f -mtime +7 -delete

Or if inode count is the main concern, prune aggressively based on age or total file count. 3. Prefer a remote object store with lifecycle policies If you need persistent caching but want storage management handled more cleanly, use something like S3/GCS/Azure and configure bucket lifecycle rules. That’s usually the most robust approach for production. Then you can: - keep

cache_expiration

in Prefect for cache validity - use bucket lifecycle rules for actual deletion 4. Be selective about what you persist Avoid persisting results for tasks where caching adds little value. Only cache: - expensive computations - stable inputs - frequently reused outputs For everything else, disable persistence/caching. 5. Use coarser-grained caching If you cache many tiny task outputs, inode exhaustion can happen faster than disk exhaustion. In that case: - cache fewer tasks - cache larger aggregated outputs instead of many tiny results Practical recommendation For local filesystem caching, I’d usually do this: - set

cache_expiration

to something reasonable like 1–7 days - run a scheduled cleanup job daily - monitor both disk usage and inode usage - move to object storage if cache volume grows beyond “single-machine convenience” So the short answer is: use

cache_expiration

+ OS-level cleanup. Prefect itself does not currently enforce storage quotas or garbage-collect local cached result files. Docs: - Task caching - Results If you want, I can give you a concrete cleanup script for Linux/macOS or a Prefect flow that prunes old local result files safely.

Yes — that’s a valid pattern, and it’s a common reason to use persisted task results/caching. If you’re using task result caching as a checkpoint so retries don’t re-run side-effectful or expensive steps, then cleaning up old result files is still OK — as long as you keep them long enough for the retries / reruns you care about. Rule of thumb: You can delete a cached result after you no longer need Prefect to reuse it. That means retention should be longer than the maximum window in which: - a task run might retry - a flow run might resume / be re-run and expect the same cached result - another run with the same cache key might intentionally reuse the checkpoint What happens if you delete too early? If the cached result file is gone, Prefect can’t load it, so the task will run again. For “exactly once” semantics, that may be a problem if re-execution would repeat a side effect. So the answer is: Yes, you can clean old result files, but only after your checkpointing window has passed. Recommended approach Pick a retention period based on your operational behavior, for example: - if retries only happen for a few hours, keep cache at least 1–2 days - if manual reruns may happen the next day, keep cache several days - if you depend on cache for deduplication across long periods, local filesystem cleanup becomes riskier Important distinction Using cache for “exactly once” is helpful, but it is not a perfect durability boundary by itself if stored only on local disk, because: - local files can be deleted - local disk can fail - workers may run on different machines and not share the same cache - ephemeral infrastructure may lose local state So if “exactly once” is truly critical, the stronger pattern is: - make the operation itself idempotent when possible, or - store the checkpoint/dedup marker in a durable external system - database row - object store marker - message/idempotency table - upstream/downstream system with idempotency key support Then Prefect cache becomes an optimization / convenience layer rather than the only correctness mechanism. Practical guidance If you stay with local filesystem cache: - keep cleanup conservative - retain files longer than your maximum retry/rerun horizon - avoid cleanup of very recent files - make sure all retries that need the cache run on the same persistent storage - do not assume local cache is safe across hosts unless storage is shared A reasonable setup might be: -

cache_expiration=timedelta(days=3)

- cleanup job deletes files older than 7 days That gives: - 3 days of cache validity for Prefect reuse - a few extra days of buffer before physical deletion Best practice for your use case For checkpointing side-effectful work, I’d recommend: 1. Use

cache_expiration

to define the logical checkpoint window 2. Keep filesystem cleanup retention longer than

cache_expiration

3. If correctness really matters, back it with an external idempotency/checkpoint store So: Yes, old result files can be cleaned, but don’t treat local cache as your only exactly-once guarantee. If you want, I can sketch a concrete Prefect pattern for “checkpointing with retries” that combines task caching with a durable idempotency record.