Short answer: No—Redis does not need persistence for Prefect in an active-active (HA) OpenShift/Kubernetes setup. Details: - In Prefect 3.x, Redis is required for multi-replica API/server deployments to handle: - Messaging broker and dedup cache (event distribution) - Concurrency lease storage (global concurrency limits) - Docket (leader election/coordination for background services like the scheduler) - All of the above are ephemeral. Durable state (flow/task runs, history, configurations, automations, events DB) lives in PostgreSQL. - If Redis restarts or is lost, you may see brief delays or duplicate events, and leases will be reacquired, but there’s no data loss because critical state is in Postgres. Recommended approach: - Use HA Redis for availability (e.g., Sentinel/Cluster or multiple replicas) but disable persistence for durability (no PVCs needed). - Disable AOF and RDB snapshots:

REDIS_APPENDONLY=no
  REDIS_SAVE=""

- Ensure all API/server replicas share the same Redis and Postgres. Example Prefect 3.x config (self-hosted, HA):

# Durable state
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@postgres:5432/prefect>

# Use Redis-backed coordination for multi-replica/active-active
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_SERVER_DOCKET_URL=<redis://redis:6379/1>

# Connection details for the Redis integration
PREFECT_REDIS_MESSAGING_HOST=redis
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0

What you get with ephemeral Redis: - Concurrency leases auto-expire and are reacquired - Background services auto re-elect a leader - Events in transit are reprocessed - No durable data loss (Postgres holds it) Follow-up questions to tailor guidance: - Are you running multiple API/server replicas across namespaces/clusters? If so, ensure they share a single highly available Redis endpoint. - Do you already have a managed/operated Redis in OpenShift? If yes, you can usually turn off AOF/RDB and skip PVCs safely for Prefect’s use case.

Great question. Since you’re running multiple API pods with --no-services, you’ll want to run the Prefect “server services” as separate pods. Here’s the HA guidance: - How many service pods? Run at least 2 total for high availability. With two localities, running 1 services pod in each locality is a solid pattern. - Why this works: Prefect’s services (scheduler, automations, late-run detection, etc.) use Docket with Redis to elect a single active leader per service. Multiple service pods will all start; exactly one will hold the lock and run each service, and the others will sit idle as hot standbys. If the leader dies or loses the lock, another pod takes over automatically. - Critical requirement: All API pods and all service pods must point to the same Postgres and the same Redis. If each locality uses its own Redis/Postgres, you will get split-brain (duplicate scheduling, broken concurrency). Use a shared, highly available Postgres and Redis reachable from both localities. Recommended layout - Locality A: N API pods (--no-services), 1 services pod - Locality B: N API pods (--no-services), 1 services pod - Shared HA Postgres + shared HA Redis (persistence not required, but availability is) - All pods share the same Prefect config (database URL, Redis-backed messaging, docket URL) Example services pod command/env

# Same env as your API pods for DB + Redis:
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@postgres:5432/prefect>

# Multi-replica/HA requires Redis-backed coordination:
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_SERVER_DOCKET_URL=<redis://redis:6379/1>

# Start all enabled services in a pod:
prefect server services start

Notes and gotchas - You do not need Redis persistence, but you do want Redis to be highly available (Sentinel/Cluster or managed) and reachable from both localities. - Keep latencies to Redis/Postgres reasonable; services are sensitive to lock TTLs and scheduling loops. - If you prefer extra redundancy, you can run 2 services pods per locality (total 4). It won’t harm anything—only one pod per service will be active—but typically 1 per locality is sufficient. Questions to confirm so I can fine-tune: - Will both localities point to the same Postgres and Redis endpoints? - Any cross-locality latency constraints I should be aware of?

Great question. Here’s a practical way to size and scale Redis for Prefect 3.x in an HA setup. What drives Redis memory for Prefect - Messaging (Redis Streams): in-flight events and internal messages. Streams are trimmed based on consumer progress, so retention is bounded by how fast consumers keep up and your trim policy. - Dedup cache: short-lived keys (≈5 min TTL). Very small footprint. - Concurrency leases: small JSON blobs + indexes for active leases. Grows with the number of active leases, not history. - Event ordering/causal ordering: short-lived keys (seconds to minutes) for in-flight events. Rule-of-thumb sizing - Average event/message size in Redis is typically in the few hundred bytes range (payload + small metadata). Use 0.5–1.0 KB/message for estimation. - Memory for streams ≈ events_per_second × avg_message_size_bytes × retention_seconds × overhead_factor - Overhead factor: 1.5–2.0 to account for Redis internals, consumer group metadata, and misc keys. Quick estimates - Light workload (<=50 events/sec, short retention via trimming): - 50 eps × 700 B × 600 s (10 min) × 2 ≈ 42 MB - Add leases + ordering + headroom → 256–512 MB is sufficient - Medium (50–500 eps, 10–20 min retention): - 300 eps × 800 B × 1200 s × 2 ≈ 576 MB - Add overhead → target 1–2 GB - Heavy (500–2000 eps, 20–30 min retention): - 1000 eps × 1000 B × 1800 s × 2 ≈ 3.6 GB - Plan for 4–6 GB or reduce retention/trim more aggressively How to tune retention (keep streams small) - Ensure multi-replica API/services use Redis-backed messaging so trimming can consider all active consumer groups. - Prefer shorter trim intervals and mark idle groups as trim-eligible so old data isn’t retained for inactive consumers. - Keep dedup lookback small (≈5 minutes is typical). - Don’t let dead consumer groups linger; they inflate the “minimum delivered” watermark and prevent trims. Operational sizing loop (measure, don’t guess) 1) Baseline metrics

# Overall memory
redis-cli INFO memory | egrep 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'

# Stream sizes
redis-cli XLEN <your_event_stream_key>
redis-cli XINFO STREAM <your_event_stream_key>
redis-cli XINFO GROUPS <your_event_stream_key>
redis-cli XPENDING <your_event_stream_key> <group-name>

# Key counts by area
redis-cli --scan --pattern 'prefect:concurrency:*' | wc -l
redis-cli --scan --pattern '*:seen:*' | wc -l
redis-cli --scan --pattern 'message:*' | wc -l

2) Estimate average message size - Take a time window (e.g., 10 minutes), record XLEN delta, and watch used_memory delta. Divide to get bytes/message. Validate your 0.5–1.0 KB assumption. 3) Tune - If streams grow too large: increase trim frequency, reduce idle threshold for trimming, or add/accelerate consumers. - If PEL (pending entries) grows: consumers are falling behind—scale your API/services or lower per-message processing time. Redis configuration tips for Prefect - Persistence: keep it off (AOF off, RDB off) to reduce memory/IO overhead:

REDIS_APPENDONLY=no
  REDIS_SAVE=""

- maxmemory: set high enough to avoid evictions. If you must cap, use a policy that won’t silently drop streams (noeviction is safer operationally—you’ll see write errors rather than silent data loss). - HA: use Sentinel or a managed HA Redis. For very large volumes, Redis Cluster can help, but be mindful a single hot stream key will live on a single shard—shard streams by topic if needed. Concurrency/leases footprint - Leases are tiny (≈200–400 bytes/lease plus small index entries). Even 10k concurrent leases is typically just a few MB. Most memory is driven by streams. Practical starting points - Most deployments: start with 512 MB–1 GB, no persistence, and short stream retention (5–15 min) with consumers keeping up. - Higher throughput (hundreds of events/sec): 1–2 GB. - Thousands of events/sec or long retention: 4–6 GB, or reduce retention and increase consumer throughput.

If you can share your expected events/sec (flow/task state changes and automation events) and how long you want messages available in Redis, I can run the math and give you a concrete memory target and trim settings.