Julien
10/24/2025, 7:25 AMMarvin
10/24/2025, 7:25 AMMarvin
10/24/2025, 7:31 AMcache_policy=prefect.cache_policies.NO_CACHE
- Don’t persist results: set persist_result=False
- Don’t keep results in memory: set cache_result_in_memory=False
- Flow-level
- Affects the flow’s own result (not task results): set @flow(persist_result=False)
- You can also set a flow cache_policy, but it does not change the defaults for tasks inside the flow
- Global/deployment/worker level (via environment variables)
- PREFECT_TASKS_DISABLE_CACHING=true disables caching for all tasks
- PREFECT_TASKS_DEFAULT_PERSIST_RESULT=false makes tasks not persist results by default
- PREFECT_RESULTS_PERSIST_BY_DEFAULT=false sets a global default for result persistence
Important notes
- There is no CLI “--no-cache” flag; use the Python decorator options or the env vars above.
- If you set both persist_result=False and cache_result_in_memory=False, you cannot reliably pass very large Python objects to downstream tasks without holding them in memory. In that case, write large data to external storage (e.g., S3/GCS/disk) inside the task and return a lightweight reference (path/URL).
- Avoid calling future.result() on massive results unless you really need them; that materializes the data in memory in your flow process.
Examples
Per-task: disable caching, persistence, and in-memory retention
from prefect import task, flow
from prefect.cache_policies import NO_CACHE
@task(
cache_policy=NO_CACHE,
persist_result=False,
cache_result_in_memory=False,
)
def generate_big_payload():
# produce large data
return b"x" * (1024 * 1024 * 500) # 500MB
@flow(persist_result=False)
def my_flow():
fut = generate_big_payload.submit()
# Do NOT call fut.result() unless necessary
return "done"
Global/deployment/worker environment variables (set in your worker environment or Deployment’s Environment settings)
export PREFECT_TASKS_DISABLE_CACHING=true
export PREFECT_TASKS_DEFAULT_PERSIST_RESULT=false
export PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
Optional: “discard” persisted results if something else forces persistence
- You can point result_storage at a sink that discards writes. For example:
from prefect.filesystems import NullFileSystem
from prefect import task
@task(result_storage=NullFileSystem(), persist_result=True)
def compute():
return "large-but-throw-away"
This ensures even if persistence is on, nothing is actually written. Usually you don’t need this if you explicitly set persist_result=False.
Links
- Caching concepts: Prefect 3 docs: Caching
- Results and persistence: Prefect 3 docs: Results
- Settings reference: Prefect 3 docs: Settings reference
Quick questions to tailor the fix
- Do downstream tasks need these large objects, or can you return references instead?
- Where are you running (local process, Docker, k8s), and can you set env vars at the worker/deployment level?