Hi <@ULVA73B9P> I have a lot of memory issue with ...
# ask-marvin
j
Hi @Marvin I have a lot of memory issue with prefect when task return lots of data, it looks like prefect is caching task result, is thete a way to disable this ? on a task level or for a flow (including all task) or on a deployment / worker level ?
m
thought for 356.4 seconds
Short answer: yes. In Prefect 3.x you can turn off both result caching and persistence, either per-task, per-flow (flow result only), or globally via settings/env vars. What to change - Task-level (most direct) - Disable caching: set
cache_policy=prefect.cache_policies.NO_CACHE
- Don’t persist results: set
persist_result=False
- Don’t keep results in memory: set
cache_result_in_memory=False
- Flow-level - Affects the flow’s own result (not task results): set
@flow(persist_result=False)
- You can also set a flow
cache_policy
, but it does not change the defaults for tasks inside the flow - Global/deployment/worker level (via environment variables) -
PREFECT_TASKS_DISABLE_CACHING=true
disables caching for all tasks -
PREFECT_TASKS_DEFAULT_PERSIST_RESULT=false
makes tasks not persist results by default -
PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
sets a global default for result persistence Important notes - There is no CLI “--no-cache” flag; use the Python decorator options or the env vars above. - If you set both
persist_result=False
and
cache_result_in_memory=False
, you cannot reliably pass very large Python objects to downstream tasks without holding them in memory. In that case, write large data to external storage (e.g., S3/GCS/disk) inside the task and return a lightweight reference (path/URL). - Avoid calling
future.result()
on massive results unless you really need them; that materializes the data in memory in your flow process. Examples Per-task: disable caching, persistence, and in-memory retention
Copy code
from prefect import task, flow
from prefect.cache_policies import NO_CACHE

@task(
    cache_policy=NO_CACHE,
    persist_result=False,
    cache_result_in_memory=False,
)
def generate_big_payload():
    # produce large data
    return b"x" * (1024 * 1024 * 500)  # 500MB

@flow(persist_result=False)
def my_flow():
    fut = generate_big_payload.submit()
    # Do NOT call fut.result() unless necessary
    return "done"
Global/deployment/worker environment variables (set in your worker environment or Deployment’s Environment settings)
Copy code
export PREFECT_TASKS_DISABLE_CACHING=true
export PREFECT_TASKS_DEFAULT_PERSIST_RESULT=false
export PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
Optional: “discard” persisted results if something else forces persistence - You can point
result_storage
at a sink that discards writes. For example:
Copy code
from prefect.filesystems import NullFileSystem
from prefect import task

@task(result_storage=NullFileSystem(), persist_result=True)
def compute():
    return "large-but-throw-away"
This ensures even if persistence is on, nothing is actually written. Usually you don’t need this if you explicitly set
persist_result=False
. Links - Caching concepts: Prefect 3 docs: Caching - Results and persistence: Prefect 3 docs: Results - Settings reference: Prefect 3 docs: Settings reference Quick questions to tailor the fix - Do downstream tasks need these large objects, or can you return references instead? - Where are you running (local process, Docker, k8s), and can you set env vars at the worker/deployment level?