Short answer: yes. Persisting results in Prefect 2.x is useful, but it’s not free. The main downsides: - Performance overhead - Extra serialization/deserialization and I/O on every task. For remote storage (S3/GCS/etc.), expect noticeable latency. - Large objects make it worse; Pickle is slower and more brittle than JSON. - Cost and quotas - You’ll pay for storage, requests, and egress on object stores. High task throughput can translate into a lot of small objects and API calls. - Operational complexity and failure modes - More things to go wrong: auth/permission issues, missing buckets, timeouts, throttling, network flakiness. - You must manage retention/lifecycle; otherwise your bucket becomes a landfill. - Compatibility and lock-in - Pickled results are tied to Python and library versions; schema/code changes can break deserialization. - Cross-environment reuse can fail if the environment that reads the result doesn’t match the one that wrote it. - Security risks - Persisted results can include sensitive data. You need encryption-at-rest, access controls, and to avoid Pickle when possible (it’s unsafe with untrusted inputs). - Caching pitfalls - Persistence enables cache reuse across runs. If your

cache_key_fn

isn’t robust, you can serve stale results and scratch your head for hours. - Side-effecting tasks shouldn’t rely on cached/persisted outputs for correctness. - Observability and UX surprises - Persisted results live in your storage; the UI shows references, not the full payload. Debugging large/complex results can be a scavenger hunt. When to think twice: - High-volume, small/fast tasks (I/O cost dominates). - Sensitive data pipelines without strong storage policies. - Workloads with unstable schemas or frequent dependency changes. - Anything where correctness hinges on re-execution rather than reuse. If you do persist: - Prefer JSON/structured serializers when possible; reserve Pickle for objects you truly can’t serialize otherwise. - Set storage lifecycle policies (TTL/archival). - Make cache keys explicit and conservative. - Keep read/write environments aligned (Python and deps). - Encrypt and lock down storage. If you’re on 2.x today, it’s fine—but consider planning for Prefect 3.x, which improves result handling and deployment ergonomics.

Short version: not really. In Prefect 2.x there’s no built-in “resume a run and only execute the previously failed tasks” unless you have something to skip the successful ones. Without persisted results, Prefect can’t supply the outputs of prior successful tasks to downstreams, so a rerun will recompute everything. Practical options that work without persisting results: - Use retries on the tasks that can fail - Add `retries`/`retry_delay_seconds` so failures are handled within the same run. - Example:

@task(retries=3, retry_delay_seconds=60)
    def fragile():
        ...

- Make tasks idempotent with external checkpoints - Have each task check external state (S3/DB/file) and no-op if work is already done. Then rerunning the flow only “does” the failed/missing parts. - Example:

@task
    def process_record(id):
        if already_processed(id):  # e.g., S3 object/DB row exists
            return marker_for(id)
        # do work, then write the marker
        write_marker(id)
        return marker_for(id)

- Split into subflows/stages with checkpoints - Break your DAG into stages that write durable markers; rerunning only reruns stages missing their markers. What doesn’t exist in 2.x: - A CLI/UI “rerun failed tasks only” for an already-failed flow run. - Cross-run caching without persisted results to reconstruct prior task outputs. If you’re open to minimal persistence, enabling cache keys + persisted results lets a fresh run skip previously successful tasks automatically. If not, external checkpoints are the usual pattern. Can you share whether this is for rerunning after a completed failed run, and whether your tasks’ outputs are consumed downstream? That will determine how strict your checkpointing needs to be.

Because the default favors speed, simplicity, and safety. Persisting results in Prefect 2.x is opt-in because: - Performance and cost: Writing every task output to storage adds serialization and I/O latency, plus object-store costs (requests, egress, retention). Most users don’t need it on every task. - Fragility and compatibility: Pickled objects can break across Python/library upgrades and are unsafe with untrusted inputs. Many objects aren’t cleanly serializable anyway. Opt-in avoids surprise failures. - Security: Persisted results may contain sensitive data; default-off prevents accidental data-at-rest without explicit controls (encryption, ACLs, lifecycle). - Correctness surprises: Cross-run reuse/caching can serve stale results if cache keys are wrong. Default-off pushes you to enable it only where you’ve thought through idempotency and cache keys. - Operational complexity: It requires configuring result storage, serializers, and lifecycle policies. Default-off keeps the out-of-the-box experience simple. When to turn it on: - You want cross-run caching/skipping of expensive tasks. - You need to resume/retry work across runs or infrastructure. - You’ve chosen safe serializers (prefer JSON/structured) and storage with proper security/lifecycle. For reference, the

@task

decorator exposes

persist_result

,

result_storage

, and

result_serializer

so you can enable it surgically where it pays off.