<@ULVA73B9P> What does it mean when a flow run is ...
# ask-marvin
p
@Marvin What does it mean when a flow run is "Paused". Does it allow the infrastructure to deprovision (e.g. with a Kubernetes, shutdown the flow runs associated Kubernetes Job) and restart a new one later?
m
thought for 250.2 seconds
Short answer: - Paused = the flow run is waiting (typically for input/approval), but the Python process keeps running. Infra is NOT released. - If you want infra to deprovision and later pick back up, use Suspended instead. Details: - Paused state is entered when your flow calls
pause_flow_run(...)
. The engine stays alive and polls Prefect for a resume signal. On Kubernetes, that means the pod/job remains running. If you kill the pod while Paused, the run will crash rather than resume. - Suspended state is entered when your flow calls
suspend_flow_run(...)
. This is designed for long waits or human-in-the-loop approvals where you want to release compute. The worker/process exits cleanly; later, when you resume the run, Prefect will schedule it again and a new worker/pod will pick it up and continue from the suspension point. Timeouts: - Both
pause_flow_run
and
suspend_flow_run
default to a 1 hour timeout (
timeout=3600
). If not resumed in time, you’ll get a timeout error (e.g.,
FlowPauseTimeout
) and the run will fail. Set a larger timeout if needed. How to resume: - In the UI: Click Resume on the flow run, optionally provide input. - In code:
Copy code
from prefect.flow_runs import resume_flow_run

resume_flow_run(flow_run_id, run_input={"approved": True})
Kubernetes best practice: - For short, bounded waits where keeping the pod running is fine, use Paused. - For longer waits or when you want autoscaling/deprovisioning, use Suspended so your Kubernetes Job can complete and a new one will be created automatically on resume. Code examples - Pause (keeps infra alive):
Copy code
from prefect import flow, pause_flow_run

@flow
def my_flow():
    approval = pause_flow_run(wait_for_input=dict, timeout=3600)
    # continues immediately after resume, in the same process/pod
    if approval.get("approved"):
        ...
- Suspend (releases infra and resumes later in a new process/pod):
Copy code
from prefect import flow
from prefect.flow_runs import suspend_flow_run

@flow
def my_flow():
    request = suspend_flow_run(wait_for_input=dict, timeout=86400)
    # after resume, execution picks up here in a new process/pod
    ...
If you’re on Prefect 3.x (assumed), there is no
prefect flow-run resume
CLI subcommand; use the UI or the Python API as shown above.
p
@Marvin can i use suspend_flow_run anywhere in a flow or task? How does Prefect store and restore the state of local variables (e.g. non-serializeable objects) and references? i can't image this is working properly except in very controlled scenarios.
m
thought for 102.2 seconds
Great questions — you’re thinking about the right constraints. Short answer - You can call
suspend_flow_run
from within a flow (including subflows). It suspends the entire flow run. - You should NOT rely on it to preserve in-memory local variables or open resources. Prefect does not serialize your Python stack. - If you need state across a suspend/resume, you must persist it yourself and reload it after resume. Where can I call it? - In a flow: Recommended. It’s a “checkpoint” where the flow can safely stop and later continue. - In a subflow: Also fine; the subflow run suspends and resumes independently, and the parent waits on its completion. - Inside a task: Technically you can target the parent flow via
flow_run_id
, but it’s not a good pattern. The flow is what resumes later, not the task’s Python frame. The task will be re-executed on resume if your code calls it again. How suspend/resume really works -
suspend_flow_run
creates a “wait for input” record (optionally with a
key
) and then exits the run by transitioning it to Suspended. - On resume, Prefect schedules the same flow run again on fresh infrastructure. Your flow code is re-imported and re-executed until it reaches the suspend point again. - At that point, if matching input exists (via the same
key
),
suspend_flow_run(...)
immediately returns that input instead of suspending again. - There is no capture/restoration of your Python call stack or local variables. Only the run input and persisted results you explicitly stored are available. Implications for local variables and non-serializable objects - Local variables, in-memory objects, open DB connections, temporary files in the pod, etc. are not restored. - Any input you provide at resume must be JSON-serializable or a Pydantic model (what you pass to
run_input=
). - If you need to “carry” state across a suspend, persist it: write to durable storage (database, object storage, block), or rely on persisted task results and reload when execution resumes. Pass references (paths, keys, IDs) via
run_input
or your own storage. Best practices - Place
suspend_flow_run
at deterministic, re-entrant boundaries in your flow. - Use the
key=
parameter so the same suspend point can return existing input on re-entry and not suspend again. - Persist anything you’ll need after resume; reload it immediately after
suspend_flow_run
returns. - Keep work before the suspend minimal or idempotent since it will run again on resume. Examples - Suspend at a flow boundary (recommended):
Copy code
from prefect import flow
from prefect.flow_runs import suspend_flow_run

@flow
def approval_flow():
    # Compute/persist any needed state before suspending
    state_ref = persist_large_state_somewhere()

    approval = suspend_flow_run(
        wait_for_input=dict,  # or a Pydantic model
        timeout=86400,        # 24h
        key="approval-step-1" # idempotency key for this wait point
    )
    # Re-entry after resume: reload any state you need
    state = load_from_ref(state_ref)

    if approval.get("approved"):
        do_the_thing(state)
- Resuming with input (UI or Python):
Copy code
from prefect.flow_runs import resume_flow_run
resume_flow_run(flow_run_id, run_input={"approved": True})
What if I suspend from a task? - It will suspend the flow run, but the task’s local memory is not preserved. On resume, the flow restarts and will call that task again if your code path leads there. This often surprises people; prefer suspending at the flow level or right after a task returns a persisted reference you can reload. Timeouts -
suspend_flow_run(timeout=...)
defaults to 3600 seconds. If not resumed in time, the run will fail with a timeout. Increase it for longer approvals, or handle the timeout explicitly. If you share a small snippet of where you’re thinking to suspend, I can suggest a pattern to persist/reload just the right pieces so the resume is smooth.
p
That means i have to make sure all tasks/subflows i called up to the point of suspend_flow_run must use caching/persist their result because the flow run is actually just called another time, but now the suspend_flow_run function returns the potential user input?
@Marvin That means i have to make sure all tasks/subflows i called up to the point of suspend_flow_run must use caching/persist their result because the flow run is actually just called another time, but now the suspend_flow_run function returns the potential user input?
m
thought for 82.1 seconds
You’ve got the right mental model. - When you call
suspend_flow_run
, the flow exits cleanly. On resume, Prefect schedules the same flow run again on fresh infrastructure and your flow code starts from the top. The call to
suspend_flow_run(...)
is reached again, but this time it immediately returns the stored input (matched by its
key
) instead of suspending. - Consequently, anything before the suspension point will run again unless you make it idempotent or enable reuse via caching/persisted results. How to avoid rework before a suspend - Use task caching + persisted results so Prefect can skip recomputation on re-entry: - Give tasks deterministic cache keys (e.g., based on inputs) and set a cache policy. - Persist results so the cache can be satisfied on the next run. - Persist large/complex state to durable storage (S3/GCS/DB) and return a reference (path/key/ID). Reload it after resume. - Place
suspend_flow_run
as early as possible before expensive or side-effectful work; or do the expensive work after resume. Example pattern
Copy code
from datetime import timedelta
from prefect import flow, task
from prefect.cache_policies import DEFAULT
from prefect.flow_runs import suspend_flow_run

@task(
    cache_policy=DEFAULT,                # input-based cache
    cache_expiration=timedelta(days=7),
    persist_result=True                  # ensure result is stored for cache reuse
)
def prepare_data(params):
    ref = write_data_somewhere(params)   # durable reference, not a big in-memory object
    return ref

@flow
def main(params):
    # This task will be re-invoked on resume, but will hit the cache and skip work
    data_ref = prepare_data(params)

    approval = suspend_flow_run(
        wait_for_input=dict,
        key="approval-step-1",
        timeout=86400
    )

    # After resume, reload from the durable reference
    data = read_data_from_ref(data_ref)
    process(data)
Subflows - A subflow call before the suspend will also run again on resume. If it’s expensive, either: - Make the subflow idempotent and cache its internal tasks, or - Record that you’ve run it (e.g., write a marker keyed by inputs) and conditionally skip on re-entry, or - Move the subflow call to after the suspend if it depends on the input. When you don’t need caching/persistence - If prior steps are cheap and/or idempotent (safe to run again), you can skip caching. - For anything expensive or with side effects, use caching/persistence or guard with your own “already done” checks. Key takeaways - Yes: assume your flow will re-run from the start on resume. - Use task caching + persisted results (and/or durable references) to avoid recomputation. - Place
suspend_flow_run
at deterministic, re-entrant checkpoints and use
key=
so it doesn’t suspend again on re-entry.