<@ULVA73B9P> I am seeing that If I apply the persi...
# ask-marvin
i
@Marvin I am seeing that If I apply the persist=True option to a task, all the tasks called from it are also cached. Is this correct?
m
thought for 178.6 seconds
Short answer: No. In Prefect 3.x, setting
persist_result=True
(sometimes referred to as “persist”) on a task only applies to that specific task. It does not cascade to other tasks that it calls. Propagation happens from flows to tasks, not from tasks to other tasks. A quick illustration:
Copy code
from prefect import flow, task

@task(persist_result=True)
def outer():
    return inner()  # inner does NOT inherit persist_result from outer

@task  # no explicit persistence
def inner():
    return 42

@flow  # no explicit persistence at the flow level
def run():
    return outer()
- outer() will persist its result. - inner() will not persist its result unless you: - explicitly set
@task(persist_result=True)
on it, or - enable persistence at the flow level (see below), or - have a global default set via settings. Where propagation does happen: - If you set
@flow(persist_result=True)
, tasks called within that flow will, by default, persist results unless they explicitly override it. Also, persistence and caching are different: - persist_result controls storing results to disk/backends. - caching is controlled separately (e.g., by attaching a cache policy). Enabling persistence on one task does not automatically “cache” other tasks. If you’re seeing “all tasks cached,” it may be due to: - persistence set at the flow level, - global defaults like
PREFECT_RESULTS_PERSIST_BY_DEFAULT
or
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
, - or a cache policy applied to those tasks. Docs for more detail: - Results (persistence) - Caching - Settings and environment variables If you can share a minimal snippet (flow/task decorators and any env vars) or confirm which Prefect version you’re on, I can pinpoint why you’re observing that behavior.
i
@Marvin By running the example, I am seeing two keys in ~/.prefect/storage after running the flow
m
thought for 351.1 seconds
Good observation — seeing two keys in ~/.prefect/storage usually means two separate results were persisted: one for the inner task and one for the outer task. A few clarifications: - Those files are persisted results, not an in-memory “cache.” - In Prefect 3.x, each task decides independently whether to persist. However, a task can end up persisting even if you didn’t explicitly set
persist_result=True
on it due to: - Flow-level propagation:
@flow(persist_result=True)
enables persistence for all tasks in that flow unless they override it. - Global settings: environment or profile settings like
PREFECT_RESULTS_PERSIST_BY_DEFAULT
or
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
. - A cache policy on the task (cache policies require persistable results). So if your outer task has
persist_result=True
and your inner task is persisting too, it’s typically because one of the above is causing the inner task to persist as well. That produces two keys: - Key A: persisted result for the inner task - Key B: persisted result for the outer task (which returns the inner task’s value, so both files often contain the same payload, but under different keys) How to confirm what’s happening: - Check your settings in code:
Copy code
from prefect.settings import get_current_settings
  print(get_current_settings().results.persist_by_default)
  print(get_current_settings().tasks.default_persist_result)
- Make persistence explicit on the inner task to test:
Copy code
@task(persist_result=False)
  def inner():
      return 42
With the outer task
persist_result=True
and inner
persist_result=False
, you should see a single new key (outer only). If you can share: - Your exact flow/task decorators (especially whether the flow has
persist_result=True
), - Any cache policy usage, - And whether you have any Prefect env vars set (e.g.,
PREFECT_RESULTS_PERSIST_BY_DEFAULT
,
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
), I can pinpoint why you’re seeing two entries. References: - Results (persistence): https://docs.prefect.io/latest/concepts/results/ - Caching: https://docs.prefect.io/latest/concepts/caching/ - Settings: https://docs.prefect.io/latest/concepts/settings/
i
@Marvin I get persist_by_default=False, default_persist_result=None Does it mean that the inner task gets its persistence from the outer task?
m
thought for 209.3 seconds
Yes — with those settings, that’s exactly what’s happening. Resolution order for a task’s
persist_result
in Prefect 3.x is: 1) The task’s own
@task(persist_result=...)
2)
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
3) If neither is set, Prefect falls back to the current run context: - If you’re inside another task, it uses the active TaskRunContext’s
persist_result
(i.e., the outer task’s value) - Otherwise it uses the FlowRunContext’s value - Otherwise it uses the global default (
results.persist_by_default
, which you have False) So in your case: - Outer task has
persist_result=True
- Inner task has
persist_result=None
- No task default and
persist_by_default=False
→ The inner task picks up
True
from the outer task’s active TaskRunContext, so both persist and you see two keys in
~/.prefect/storage
. If you want only the outer to persist, set the inner explicitly to `persist_result=False`:
Copy code
@task(persist_result=True)
def outer():
    return inner()

@task(persist_result=False)
def inner():
    return 42
Docs: - Results (persistence): https://docs.prefect.io/latest/concepts/results/ - Settings: https://docs.prefect.io/latest/concepts/settings/
i
@Marvin Even though I am setting PREFECT_TASKS_DEFAULT_PERSIST_RESULT=False in an .env file which is loaded before the script, I am still getting two keys in ~/.prefect/storage
m
thought for 119.0 seconds
Thanks for sharing the settings—this explains the behavior you’re seeing. Given: - PREFECT_RESULTS_PERSIST_BY_DEFAULT=False - PREFECT_TASKS_DEFAULT_PERSIST_RESULT=None - Outer task has persist_result=True - Inner task has persist_result=None What happens: - The inner task resolves its persist_result via the priority chain: 1) Its own setting (None) 2) Tasks default (None) 3) should_persist_result() → checks the current run context - At the moment should_persist_result() is evaluated for the inner task, the outer task’s TaskRunContext is still active. TaskRunContext.get() returns that context, and Prefect uses its persist_result (True). - Result: the inner task inherits True from the outer task and persists its result. The outer task also persists its own result. You see two keys in ~/.prefect/storage. How to get just one key: - Explicitly disable persistence on the inner task:
Copy code
@task(persist_result=False)
  def inner():
      return 42
- Or disable it flow-wide and opt-in only where needed:
Copy code
@flow(persist_result=False)
  def my_flow():
      return outer()
Quick self-checks: - Print effective values during the run:
Copy code
from prefect.context import TaskRunContext
  from prefect.settings import get_current_settings

  @task(persist_result=True)
  def outer():
      print("outer persist:", TaskRunContext.get().persist_result)
      return inner()

  @task
  def inner():
      print("inner persist:", TaskRunContext.get().persist_result)
      return 1
- Verify Prefect sees your env var as expected:
Copy code
from prefect.settings import get_current_settings
  s = get_current_settings()
  print("results.persist_by_default:", s.results.persist_by_default)
  print("tasks.default_persist_result:", s.tasks.default_persist_result)
If your goal is “only the outer task persists,” the most reliable approach is to set
@task(persist_result=False)
on the inner task(s).