Short answer: don’t autoscale the Prefect “services” container, and definitely not on Redis queue depth. Here’s why and what to do instead: - In Prefect 3.x, the services process (started by

prefect server services start

) is designed to run as a single instance. Running multiple replicas can cause duplicate scheduling/triggers and other side effects. Use one replica with robust liveness/readiness probes and restart policy. - Redis in Prefect is not a traditional “work queue” for dispatching runs to workers. It’s used for events/messaging/leases. Redis stream length/queue depth is not a meaningful autoscaling signal for Prefect. - Scale the API servers horizontally instead (run multiple

prefect server start --no-services

pods behind a load balancer), and scale your worker deployments based on actual backlog from the Prefect API, not Redis. Recommended setup - API servers (scale out):

prefect server start --host 0.0.0.0 --no-services

- Services (singleton):

prefect server services start

- For HA of services (crash/restart safety), back Docket with Redis:

PREFECT_SERVER_DOCKET_URL=<redis://your-redis:6379/1>

Still 1 replica, but it can recover state after restarts. Autoscaling signals that do work If you want backlog-based scaling, target your workers (e.g., a Kubernetes worker deployment) using Prefect API endpoints: - Work queue health and late runs: - GET

/work_queues/{id}/status

→

late_runs_count

,

healthy

,

last_polled

curl -H "Authorization: Bearer $PREFECT_API_KEY" \
      "$PREFECT_SERVER_URL/api/work_queues/<QUEUE_ID>/status"

- Scheduled/late run counts (per queue, pool, or deployment): - POST

/flow_runs/count

with filters

curl -X POST "$PREFECT_SERVER_URL/api/flow_runs/count" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $PREFECT_API_KEY" \
      -d '{
        "flow_runs": { "state": { "name": { "any_": ["Scheduled","Late"] } } },
        "work_pool_queues": { "id": { "any_": ["<QUEUE_ID>"] } }
      }'

Use those counts in an HPA/KEDA strategy to scale your worker pods up/down. Common triggers: - Scale up when

late_runs_count

> 0 - Scale proportionally to number of Scheduled + Late runs - Scale down when no scheduled runs for N minutes and no active workers needed Docs and references - Scaling and self-hosted architecture (multiple API pods + single services): - Scale self-hosted Prefect - Server concepts - CLI references: - `prefect server start --no-services`:

prefect server start --help

- `prefect server services start`:

prefect server services --help

- REST API for autoscaling signals: - Work queue status (late_runs_count, health) - Count flow runs (filters for Scheduled/Late) If you can share: - Are you on Prefect 3.x and self-hosting? - Are you trying to increase scheduling throughput or just worker capacity for backlogs? I can suggest a concrete HPA/KEDA example for your worker deployment once I know your environment (Kubernetes worker vs. Docker worker, etc.).

Got it — you want to protect Redis from filling up when there’s a burst of Prefect events. The good news: when Prefect 3.x uses Redis for messaging, it uses Redis Streams and trims them automatically. If your event consumers are healthy (i.e., the Prefect services process is running), the stream should not grow unbounded. Key guardrails to keep Redis from filling up 1) Keep the event consumers healthy (singleton services) - Run exactly one services pod:

prefect server services start

- Back Docket with Redis so services recover cleanly after restarts:

PREFECT_SERVER_DOCKET_URL=<redis://your-redis:6379/1>

- Add liveness/readiness probes, strong restart policy, and sufficient CPU/memory to the services pod. If services stop for a while, streams will accumulate until they restart. 2) Trim Redis streams more aggressively Prefect trims acknowledged messages on a timer. You can make trimming more frequent and ignore stale consumer groups sooner:

# Trim every 5s instead of 60s (default)
PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_EVERY=5
# Consider consumer groups idle after 60s to avoid stale groups blocking trims (default is 5m)
PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_IDLE_THRESHOLD=60

Background: trimming uses XTRIM MINID and won’t remove entries that haven’t been acked by consumers; it also auto-ignores long-idle consumer groups so they don’t block trimming. 3) Configure Redis memory and alerts - Set a hard memory cap and choose an eviction policy you’re comfortable with. For safety, prefer backpressure over silent loss:

maxmemory 2gb
  maxmemory-policy noeviction

With noeviction, new XADDs will fail instead of evicting keys (you’ll see errors/DLQ instead of silent drops). If you choose an eviction policy like allkeys-lru, Redis may evict the stream key itself, losing events. - Monitor and alert on: - redis_memory_used_bytes / maxmemory - Stream sizes and pending (XINFO STREAM / XPENDING) for the Prefect streams - Publish errors and DLQ growth - Services pod health and logs 4) Reduce downstream pressure so events clear faster - Ensure the database is healthy and performant (event persister writes to DB; if DB is slow, ack is delayed and the stream grows). - Give the services pod enough CPU/memory to keep up with bursts. - Reduce event persistence volume by shortening DB retention (unrelated to Redis memory directly, but lowers DB write load so acks can happen sooner):

PREFECT_EVENTS_RETENTION_PERIOD=3d

5) Verify you’re using Redis for messaging (multi-server) If you’re running multiple API pods, switch to Redis-backed messaging:

PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

References (worth a quick skim) - Scaling/self-hosted (Redis + services design): Scale self-hosted Prefect - Server CLI (multi-worker API vs services): Server CLI guide - Database maintenance and event retention: DB maintenance - Redis messaging improvements and trimming fixes: prefect-redis release notes If you share: - Your expected peak event rate - Current Redis size/maxmemory and policy - Whether services ever get restarted or preempted …I can suggest concrete values for

TRIM_EVERY

,

TRIM_IDLE_THRESHOLD

, Redis

maxmemory

, and alert thresholds tailored to your load.