<@ULVA73B9P> I have these storage files that accum...
# marvin-ai
p
@Marvin I have these storage files that accumulate in my prefect storage, is there a possibility to not store these files or automatically delete them ?
Copy code
-rw-r--r--. 1 root root  2565 Sep 18 14:35 ffffabc887a94443978d1739462dd51c                                                                                                                 
-rw-r--r--. 1 root root  2565 Sep 25 21:55 ffffe1353cfa46b8b27e6a4a20a82af7                                                                                                                 
-rw-r--r--. 1 root root  9191 Sep 28 08:26 fffffa5defe84203b79362073e1cdd3a                                                                                                                 
(base) [root@dtftlp11 storage]# cat fffd9f55adbb40df97262154386eff33                                                                                                                        
{"metadata":{"storage_key":"/data/.prefect/storage/fffd9f55adbb40df97262154386eff33","expiration":null,"serializer":{"type":"pickle","picklelib":"cloudpickle","picklelib_version":null},"pr
efect_version":"3.4.14","storage_block_id":null},"result":"gAWVkQYAAAAAAACMFnRibGliLnBpY2tsaW5nX3N1cHBvcnSUjBJ1bnBpY2tsZV9leGNlcHRpb26U\nk5QojBRkYXRhZmV0Y2gudGFza3MuY29yZZSMElJlbW90ZUZpbGV
5weZRoHIwabWV0ZW9mcmFuY2VfcmFkYXJfZG93bmxvYWSUaB5L\nAGgfSwBoICloIUsAaCJLAGgjS0BoJEsAdWJoJUsydWJoMksyaDNoJymBlH2UKGgqaAopgZR9lCho\nDX2UaA99lChoE
WhPaBNoUHVoFWgXKYGUfZQoaBpoU2gcaFRoHksAaB9LAGggKWghSwBoIksAaCNL\nQGgkSwB1YmglSzJ1YmgySzB1YnVidWJ1YnVih5RSlE6JTnSUUpQu\n"}(base) [root@dtftlp11 storage]#
m
thought for 323.7 seconds
Those files are Prefect 3 “result storage” artifacts. Prefect writes them when a task/flow result needs to be persisted locally (e.g., it’s not JSON-serializable, it’s large, or you’ve enabled result persistence or caching). The JSON header you saw (with serializer=pickle and prefect_version) is metadata for that persisted result; the base64 blob is the serialized value. Options to avoid or clean them up: A) Don’t persist results - If you don’t need to retrieve results from the UI/API, disable persistence: - Per task/flow:
Copy code
from prefect import flow, task

    @task(persist_result=False)
    def do_work(...):
        ...

    @flow(persist_result=False)
    def my_flow(...):
        ...
- Globally (recommended to ensure nothing persists by default):
Copy code
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false PREFECT_TASKS_DEFAULT_PERSIST_RESULT=false
- Make return values JSON-serializable when possible, or explicitly return nothing, to avoid “auto-persist” when Prefect can’t inline the value. B) Persist, but avoid local files - Use a JSON serializer when your results are JSON-serializable; that lets Prefect inline data instead of writing a file: - Per task/flow:
Copy code
@task(result_serializer="json")
    def small_result():
        return {"status": "ok"}  # JSON-serializable
- Or globally:
Copy code
prefect config set PREFECT_RESULTS_DEFAULT_SERIALIZER=json
Note: Non-JSON values will still need storage (or will fail if you force JSON on them), so only do this if your results are JSON-serializable. - Send results to remote storage with its own lifecycle/retention (S3/GCS/etc.) and let the bucket policy clean up:
Copy code
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK="s3/my-results-block"
Create/configure the block in the UI first and apply a lifecycle policy on the bucket. C) Keep local files but auto-clean them - Prefect doesn’t currently garbage-collect these files automatically (the expiration field you saw isn’t used for auto-deletion). Use OS tools or a small housekeeping job: - Simple cron example (delete older than 7 days):
Copy code
find /data/.prefect/storage -type f -mtime +7 -delete
- Or point local storage at a tmp location that gets cleaned:
Copy code
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/tmp/prefect/storage
D) About caching - If you’ve enabled task caching, that can also write to storage. You can disable caching globally if you don’t use it:
Copy code
prefect config set PREFECT_TASKS_DISABLE_CACHING=true
You can inspect your current settings with:
Copy code
prefect config view --show-defaults
Docs for details: - Results and result storage: Prefect 3 Results - Settings reference: Settings reference If you can share whether you need to read results back from the UI/API, I can recommend the safest combination (e.g., turn off persistence entirely vs. switch to JSON or remote storage).