Great question. Assuming you’re on Prefect 3.x: the Redis instance used by the Background Service primarily holds ephemeral coordination state (for background services like scheduling) and message queues. It does not need persistence for correctness, but many teams still attach a small PV so Redis can use AOF/RDB and survive pod restarts cleanly. Key points - Persistence is optional: background-service coordination data is ephemeral and re-populates on restart. - Disk usage is driven mainly by Redis persistence (AOF/RDB) and temporary message backlogs, not by long-term storage. - Right-sizing rule of thumb: PV size ≈ 2–3x your peak Redis memory footprint (to cover AOF growth and snapshot overhead) plus headroom. Recommended PV sizes - Small workloads (dev/small prod) - Profile: ≤50 concurrent flow runs, tens of runs/hour, low event rate - PV: 1–2 GiB - Notes: Fine to run without persistence; if enabling AOF, start at 2 GiB. - Medium workloads - Profile: 50–300 concurrent flow runs, hundreds of runs/hour, moderate event rate - PV: 5–10 GiB - Notes: Use stream trimming and AOF rewrite; monitor backlog during peaks. - Large workloads - Profile: 300–1000+ concurrent flow runs, thousands of runs/hour, high event rate or bursty spikes - PV: 20–50 GiB - Notes: Consider separate Redis instances for coordination vs. messaging, keep streams aggressively trimmed, and watch AOF growth. Operational guidance - Start point: If unsure, 5–10 GiB is a safe default for most prod clusters. - Monitor and adjust: - Track Redis memory and AOF size; increase PV if AOF grows near capacity. - Keep message streams trimmed to prevent backlog bloat. - Persistence mode: - If you enable AOF: expect higher disk usage; ensure auto-rewrite is on. - If you keep Redis ephemeral: the background service will recover, but you may see transient reprocessing/lag after restarts. Quick checks to size/monitor

# Memory footprint
redis-cli INFO memory | egrep 'used_memory_human|maxmemory_human'

# AOF / RDB status and file sizes
redis-cli INFO persistence | egrep 'aof_enabled|aof_current_size|rdb_last_bgsave_status'

# Stream sizes (adjust stream key if used)
redis-cli XLEN prefect:events
redis-cli XLEN prefect:messages

# Consumer lag (replace keys/group as appropriate)
redis-cli XINFO GROUPS prefect:events

Happy to tune these numbers if you can share: - Rough concurrent flow runs and runs/hour - Whether AOF is enabled - Any observed Redis used_memory and aof_current_size during peak hours - Whether Redis is shared between background service coordination and messaging or split into separate instances/databases

Great follow-up. Here’s a practical way to choose a Redis persistence mode for the Prefect Background Service (3.x), which uses Redis for ephemeral coordination and message queues: Key decision criteria - Recovery Point Objective (RPO) – how much recent state can you afford to lose on crash? - None: lose all Redis state - RDB: lose up to last snapshot interval (e.g., minutes) - AOF everysec: up to ~1s - AOF always: near-zero, highest write latency - Recovery Time Objective (RTO) – how fast do you need to be back online? - RDB: generally fastest to load - AOF: slower to load (can be large), faster with RDB preamble and regular rewrites - None: fastest to start, but services must rebuild state from the DB (may cause lag) - Workload write intensity and latency sensitivity - High write/throughput: avoid AOF always; prefer AOF everysec or RDB-only - Latency-sensitive paths: RDB or AOF everysec (avoid AOF always) - Tolerance for transient event loss/duplication - If occasional loss of in-flight messages/leases and brief reprocessing lag is acceptable, you can run without persistence - If you need to minimize missed triggers/events after restarts, use AOF (everysec) or RDB with short intervals - Disk and IOPS budget - AOF files grow larger and write more frequently; RDB is more compact but uses bursts during snapshots - Operational complexity - None: simplest - RDB-only: simple, predictable - AOF: requires monitoring/rewrite tuning; combine with RDB preamble for faster restarts - HA topology - If you’re running a single Redis instance, persistence reduces impact of node restarts - With Redis replication/sentinel/managed Redis, you may lean on HA + AOF everysec for balance Practical recommendations - Production default (balanced): AOF everysec + periodic RDB snapshots - Minimizes data loss (~1s), acceptable write overhead, predictable recovery - Good starting point for most Prefect clusters - Performance-focused with some RPO tolerance: RDB-only - Use short snapshot intervals if you want tighter RPO; fastest loads and lowest continuous write overhead - Dev/test or tolerant to brief state loss and lag: No persistence - Fastest and simplest; expect reprocessing lag and potential loss of in-flight messages/leases on restart Suggested redis.conf for the “balanced” default

appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-use-rdb-preamble yes

# Optional RDB snapshots (tune to your RPO)
save 900 1
save 300 10
save 60 10000

rdbcompression yes

Operational tips for Prefect - Separate concerns: use different Redis databases or instances for background-service coordination (Docket) vs. messaging to limit blast radius and tune persistence independently. - Keep message streams trimmed aggressively to cap memory and AOF growth. - Monitor and adjust: - Memory:

used_memory

- AOF size and rewrite stats - Stream lengths (XLEN) and consumer lag - If you disable persistence, plan for brief lag after restarts while services rescan the database and rebuild state. Relevant docs - How to scale self-hosted Prefect (Redis for messaging and background services) - Run Prefect Server via Docker Compose (example with Redis and volumes) - Background tasks and services - Settings reference (messaging and Docket URL settings) - Redis persistence overview: Redis persistence (RDB vs AOF) If you can share your RPO/RTO targets, event volume, and whether this Redis is shared or dedicated, I can recommend a concrete mode and config.