Short answer: Redis is great for small, short‑lived messages, but it will fill up fast if you’re putting large payloads or results in it. With ~30k items, you only need a few hundred KB per item to exhaust 12GB. The fix is to keep large data out of Redis—store it in object storage and only enqueue small references/IDs. A few concrete steps you can take (assuming Prefect 3.x): - Don’t pass large objects as task parameters or results - Put big inputs/outputs in S3/GCS/Azure (or a filesystem) and pass a URI or key instead. - For any tasks that only return a reference (e.g., an S3 path), set

persist_result=False

so Prefect doesn’t try to persist that result into your Redis storage. - If you must persist results, don’t use Redis for them - Point Prefect result storage to a file/object store (S3/GCS/Azure/Remote FS) and optionally enable compression to shrink on‑disk size. - Prefect supports a “result storage” per task/flow; Redis is memory‑backed and not ideal for large/long‑lived results. - Keep messages small and bounded - If you’re using Redis Streams, cap stream length (XADD ... MAXLEN ~ N). - Set TTLs on transient keys and use an eviction policy like

allkeys-lru

to avoid unbounded growth. - Inspect what’s actually big - Use

redis-cli --bigkeys

,

INFO memory

, and

MEMORY USAGE <key>

to find the offenders. Often it’s a few keys with very large values. Example patterns in Prefect 3.x 1) Return a reference and avoid persisting the result

from prefect import flow, task

@task(persist_result=False)
def compute_and_write() -> str:
    # Do work and write large output to object storage or a filesystem
    import fsspec, json, uuid
    uri = f"<s3://my-bucket/results/{uuid.uuid4().hex}.json>"
    with fsspec.open(uri, "w") as f:
        json.dump({"huge": "object"}, f)
    # Return only a small reference
    return uri

@flow
def my_flow():
    ref = compute_and_write()
    # Downstream tasks load from `ref` when needed
    return ref

2) If you do persist results, push them to object storage and (optionally) compress

from prefect import flow, task
from prefect.filesystems import RemoteFileSystem
from prefect.serializers import CompressedJSONSerializer

# Store results in S3 (via fsspec) instead of Redis
s3_store = RemoteFileSystem(basepath="<s3://my-bucket/prefect-results>")

@task(
    persist_result=True,
    result_storage=s3_store,
    result_serializer=CompressedJSONSerializer()
)
def compute_small_result():
    return {"small": "result"}  # persist small results only

Why this helps - 30k items with even ~400 KB each will exceed 12 GB. Large JSON payloads, pickled dataframes, or images can be much bigger than expected once serialized. - Prefect can persist results to a separate result store; use that for anything non‑trivial and keep the orchestration/backing system’s messages minimal. Helpful docs - Results and result storage: Prefect 3 Results - Blocks and storage concepts: Blocks - Caching and avoiding re‑computation: Caching A couple of quick questions to tailor the fix: - Are you using Redis specifically as Prefect result storage (e.g., a Redis block) or as a separate broker/queue? - Roughly how large are the payloads you’re enqueueing or the task results you’re persisting? - If you run

redis-cli --bigkeys

, what data types/keys are consuming the most memory? Share a snippet of how you’re enqueueing/returning data and I’ll suggest the minimal change to keep Redis slim.

Short answer: those are temporary Redis keys used by Prefect’s messaging/events system for de-duplication. The pattern

message:messaging-cache:<uuid>

marks a message as “seen” so Prefect won’t publish/process the same message twice within a short window. They auto-expire and are safe to delete. How to get rid of them: - Do nothing: they auto-expire after a short TTL. - Delete by pattern (safer with SCAN than KEYS in production):

redis-cli --scan --pattern 'message:messaging-cache:*' | xargs -L 100 redis-cli DEL

- If this Redis DB is dedicated to Prefect and you want a clean slate:

redis-cli FLUSHDB

(careful: this removes all keys in the current DB) If you’re seeing too many and want fewer created going forward: - Keep deduplication but shorten the TTL (if you control the messaging/dedup config in your setup). - Or disable deduplication in your Prefect Redis messaging/event publisher if duplicates are acceptable in your use case. This will stop creating these keys, but you may see duplicate events/messages. If you can share: - Are you using Prefect 3 with the Redis messaging/events integration? - Do you want a one-time cleanup, or to change behavior going forward? I can tailor exact config steps once I know your setup.

Short answer: that’s higher than I’d expect for just the

message:messaging-cache:*

de-dup keys. Those are tiny and auto-expire quickly. 87k keys / 4GB usually means you’ve also got Redis Streams data (events/logs) and/or other Prefect state accumulating (DLQ, causal-ordering state) in the same Redis. What to check first - Get a breakdown of where memory is going:

redis-cli INFO memory
redis-cli --bigkeys

- See how many of those de-dup keys you actually have and whether they’re expiring:

redis-cli --scan --pattern 'message:messaging-cache:*' | wc -l
redis-cli --scan --pattern 'message:messaging-cache:*' | head -n 5 | xargs -I{} redis-cli TTL {}

- Check your event/log streams length and consumer groups (streams can hold a lot of data):

redis-cli XLEN events
redis-cli XINFO STREAM events
redis-cli XINFO GROUPS events
redis-cli XLEN logs
redis-cli XINFO STREAM logs

- Look for dead-letter queue (DLQ) accumulation:

redis-cli SMEMBERS dlq | wc -l

- Check causal-ordering keys (can grow if followers/leaders aren’t getting reconciled):

redis-cli --scan --pattern '*:event:*' | wc -l
redis-cli --scan --pattern '*:followers:*' | wc -l
redis-cli --scan --pattern '*:waitlist*' | wc -l

How to reduce memory immediately - If the bulk is de-dup keys (safe to remove):

redis-cli --scan --pattern 'message:messaging-cache:*' | xargs -L 100 redis-cli DEL

- If the bulk is Redis Streams (events/logs), trim them. Ideally trim to the lowest delivered ID across active consumer groups, but for a quick reset you can clear them:

redis-cli XTRIM events MAXLEN 0
redis-cli XTRIM logs MAXLEN 0

Note: this only affects transient stream transport; Prefect still keeps event records in its database based on your server retention settings. - If DLQ is large and you’re okay discarding it:

redis-cli SMEMBERS dlq | xargs -L 100 redis-cli DEL
redis-cli DEL dlq

Prevent it from growing that large again - Ensure a consumer is actually running to process and trim streams. If no consumers are running, streams grow indefinitely. - If you’re using the Redis messaging/events integration, consider trimming more aggressively or verifying trimming is happening as expected (there’s periodic trimming logic that runs with consumers; if those aren’t active, nothing trims). - If you don’t need de-duplication, disable it in your publisher config to stop creating

message:messaging-cache:*

keys. If you do need it, it should auto-expire; a huge count there suggests very high publish rates or a misconfiguration. - Periodically clear DLQ if you don’t use it operationally. A few quick questions so I can tailor exact steps: - Are you using Prefect 3 with the Redis messaging/events integration for the API server? - Roughly how long has this instance been running and what’s your event/log volume? - Do you have any known inactive consumer groups or crashed consumers?

XLEN events = 1,507,966 means you have ~1.5M entries in the Redis Stream named “events”. In Prefect’s Redis messaging/events setup, that stream is just the transport for events. The canonical event records live in the Prefect database; the stream can be trimmed safely once all active consumers have read past the entries you’re trimming. Is it safe to trim? - Yes, if you trim up to the lowest last-delivered-id across your active consumer groups. That keeps all active consumers safe. - If you don’t care about replaying any backlog (e.g., you only use the DB and don’t need consumers to catch up), you can clear the stream entirely. Quick ways to trim now - Clear the stream (most aggressive):

redis-cli XTRIM events MAXLEN 0

- Safer trim to what active consumers have already delivered: 1) Inspect groups and note each group’s last-delivered-id:

redis-cli XINFO GROUPS events

2) Choose the minimum last-delivered-id among groups you consider active, then:

redis-cli XTRIM events MINID <that-id>

Tip: If you have stale groups you no longer use, destroy them first so they don’t hold back trimming:

redis-cli XGROUP DESTROY events <group_name>

Automate trimming Option A: Let Prefect do it. If you’re using Prefect 3 with the Redis messaging/events consumer running, it periodically trims the stream based on active consumers. You can tune it: - PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_EVERY (seconds, how often trimming runs) - PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_IDLE_THRESHOLD (e.g., 300s; groups where all consumers are idle beyond this are ignored when computing the safe trim point) Option B: Cron a safe trim script (Python using redis-py) that only trims to the minimum last-delivered-id of active groups:

import os, redis

r = redis.Redis(host=os.getenv("REDIS_HOST","localhost"),
                port=int(os.getenv("REDIS_PORT","6379")),
                db=int(os.getenv("REDIS_DB","0")))

stream = "events"
idle_threshold_ms = 5 * 60 * 1000  # 5 minutes

groups = r.xinfo_groups(stream)
candidate_ids = []

for g in groups:
    group = g["name"]
    consumers = r.xinfo_consumers(stream, group)
    # Skip groups where all consumers are very idle (treat as inactive)
    if consumers and all(c["idle"] > idle_threshold_ms for c in consumers):
        continue
    candidate_ids.append(g["last-delivered-id"])

if candidate_ids:
    min_id = sorted(candidate_ids)[0]
    trimmed = r.xtrim(stream, minid=min_id, approximate=False)
    print(f"Trimmed {trimmed} entries up to {min_id}")
else:
    print("No active groups to trim against.")

Run this every minute or five via cron/k8s Job. A few tips - 1.5M entries can easily account for multiple GB of memory. Trimming will drop memory usage quickly. - Ensure your consumers are running; if no consumers are active, streams won’t auto-trim. - Before aggressive trimming, verify which consumer groups are actually in use:

redis-cli XINFO GROUPS events
redis-cli XINFO CONSUMERS events <group_name>

If you share how you run Prefect (self-hosted vs Cloud, and whether you’ve enabled the Redis messaging/events integration), I can suggest exact settings and a right-sized retention strategy.

Those are Redis Stream consumer groups on the “events” stream that were created by older Prefect server instances. They don’t stop events from being published (“emitted”), but they can hold back stream trimming and contribute to memory growth if your trimmer isn’t running or isn’t configured to ignore idle groups. How to remove old groups - Identify active vs stale groups:

redis-cli XINFO GROUPS events
redis-cli XINFO CONSUMERS events <group_name>

Active groups will have consumers with low idle times. If a group has no consumers or all consumers have been idle for a long time, it’s likely stale. - Delete a stale group (safe if not used anymore):

redis-cli XGROUP DESTROY events <group_name>

- Optionally, just remove stale consumers in a group:

redis-cli XGROUP DELCONSUMER events <group_name> <consumer_name>

Will they cause events not being emitted? - No. Extra groups do not block publishing. If your “current” consumer is running, events will still be processed. - What they can do: keep the stream from being trimmed (so XLEN/memory keeps growing) in setups where trimming considers all groups. Prefect’s consumer typically ignores fully idle groups when computing the trim point, but if the trimmer isn’t running, old groups can effectively hold the stream forever. After cleanup, trim the stream - Quick trim to remove all backlog (if you don’t need replay):

redis-cli XTRIM events MAXLEN 0

- Safer trim to the minimum last-delivered-id among active groups: 1) Get each active group’s last-delivered-id:

redis-cli XINFO GROUPS events

2) Use the smallest ID:

redis-cli XTRIM events MINID <min_last_delivered_id> APPROXIMATE FALSE

Automating cleanup/trim - Ensure a Prefect consumer is running; it periodically trims. You can tune its schedule and idle threshold if you’re using the Redis messaging integration. - Or run a small job that destroys idle groups and trims safely. Example:

import os, redis

r = redis.Redis(host=os.getenv("REDIS_HOST","localhost"),
                port=int(os.getenv("REDIS_PORT","6379")),
                db=int(os.getenv("REDIS_DB","0")))

stream = "events"
idle_ms_threshold = 24*60*60*1000  # 24h

groups = r.xinfo_groups(stream)
active_ids = []
for g in groups:
    name = g["name"]
    consumers = r.xinfo_consumers(stream, name)
    if not consumers or all(c["idle"] > idle_ms_threshold for c in consumers):
        print("Destroying stale group:", name)
        r.xgroup_destroy(stream, name)
    else:
        active_ids.append(g["last-delivered-id"])

if active_ids:
    min_id = sorted(active_ids)[0]
    trimmed = r.xtrim(stream, minid=min_id, approximate=False)
    print("Trimmed up to", min_id, "removed:", trimmed)

If you can share which group your current server uses (and whether you have any consumers actively processing), I can point to exactly which groups are safe to remove.

Great questions. - What the Redis “events” stream is: it’s just the transport/buffer. Producers publish events to the Redis Stream; a Prefect consumer (part of the server) reads them, persists them to the database, triggers rules/actions, then ACKs them. The stream entry remains in Redis until it’s explicitly trimmed. - Why XLEN can be huge even if you “care about the backlog”: two common reasons 1) Your active consumer is up-to-date but old/stale consumer groups are still present. They don’t block publishing or DB writes, but they do hold back trimming. Result: events are already in the DB, but the stream keeps all history because trimming is conservative. 2) Your consumer isn’t running or can’t keep up. Then events aren’t being persisted fast enough and the stream accumulates real backlog. How to tell which case you have 1) Compare latest produced vs consumed IDs:

redis-cli XINFO STREAM events           # last-generated-id
redis-cli XINFO GROUPS events           # last-delivered-id per group

- If your “active” group’s last-delivered-id is close to the stream’s last-generated-id, the consumer is current and the big XLEN is historical untrimmed data (likely due to stale groups). - If it’s far behind, your consumer is lagging or not running. 2) Check pending/ack status for your active group:

redis-cli XPENDING events <active_group>

- Big “pending” with high idle times suggests stuck messages/consumers. Are those old groups causing events not to be emitted? - No, they don’t block publishing. - They can, however, prevent trimming, which makes XLEN and memory grow. If your consumer isn’t running, you also won’t see events in the DB/UI because they aren’t being consumed. What to do next - If consumer is current (it’s likely a trimming problem): 1) Remove stale groups:

redis-cli XINFO GROUPS events
  redis-cli XGROUP DESTROY events <stale_group_name>

2) Trim safely to the minimum last-delivered-id among the groups you actually use:

redis-cli XTRIM events MINID <min_last_delivered_id> APPROXIMATE FALSE

3) Ensure automatic trimming is happening (if you’re using Prefect’s Redis messaging consumer, it periodically trims; you can tune its frequency/idle-threshold). If you’d like, I can share a tiny cron script that trims based on active groups. - If consumer is behind (ingestion problem): 1) Make sure a Prefect events consumer is actually running with your server. 2) Scale out consumers (add API/server replicas) to increase throughput. 3) Check for a stuck/failed consumer (XPENDING with very old idle entries; XAUTOCLAIM/consumer restart can help). 4) After consumers catch up, remove stale groups and enable periodic trimming. If you can share: - How you deploy Prefect server (version 3.x?), number of replicas, and whether Redis messaging is enabled. - The output of:

redis-cli XINFO STREAM events
  redis-cli XINFO GROUPS events
  redis-cli XPENDING events <your_active_group>

I can tell you whether you’re just missing trimming or if your consumer needs scaling/fixing, and give you exact steps to automate trimming going forward.

Thanks for the details — that output confirms your consumers are significantly behind. What it means - The Redis “events” stream is the buffer. Your consumers (groups: event-persister, reactive-triggers, task-run-recorder) read from it and persist/process. - The large “lag” values show a real backlog;

pending=0

suggests messages are being ACKed as they’re read, but consumers aren’t keeping up with the publish rate. - The extra

ephemeral-...

group is almost certainly from an old server instance; it won’t block publishing, but it does hold back trimming and adds noise. Immediate steps to catch up 1) Make sure services are actually running - If you started the API with multiple workers, you must run services in a separate process. - Foreground (watch logs):

prefect server services start

- Background:

prefect server services start --background

If you accidentally ran the API with

--no-services

, start services separately. 2) Scale out the consumers - Each additional services process adds consumers to each group. Start more replicas/pods/containers of the “services” process to increase throughput. - If you want to scale specific groups independently (optional), run separate processes with only that service enabled (so they join the same consumer group and share the load). 3) Remove stale groups (cleanup, won’t fix throughput) - If

ephemeral-...

is not used anymore:

redis-cli XGROUP DESTROY events ephemeral-ca-prefect-svr-rl4-d1-usw3-001--d68dfd4-669d697cf8-hb4j8-7a18d5dbc8ac4060a7bf2bc32774ebf3

This will also allow trimming to be more effective once consumers catch up. 4) Validate consumer health and pressure points - Check consumer idleness and pending:

redis-cli XINFO CONSUMERS events event-persister
redis-cli XPENDING events event-persister

- Look at service logs for errors/timeouts (DB timeouts, retries, etc.). If DB is slow, the persister may be bottlenecked. Throughput tuning and automation - Scale services horizontally first — it has the biggest impact. - Keep an eye on stream length and lag:

redis-cli XINFO STREAM events
redis-cli XINFO GROUPS events

- Once you’re caught up, enable or verify periodic trimming (runs with the consumers). If you prefer, you can also run a small cron that trims to the minimum last-delivered-id across active groups. Why aren’t events getting into the DB faster? - Because the EventPersister (consumer group

event-persister

) is ~1.06M messages behind. The stream decouples producers from consumers; if the consumer can’t keep up (too few instances, slow DB, or it’s not running), backlog grows. Adding service instances and ensuring your DB is healthy will close that gap. If helpful, share: - How you run Prefect (single process vs API workers + separate services) - Whether you can add more service replicas - Any errors or slow query signs in your DB I can give you a concrete scaling plan (e.g., “add N service pods for M hours to drain backlog of K msgs”).

Short answer: Yes, you can run Prefect’s server services as separate containers in Azure Container Apps. That’s the right way to scale the Redis consumers that are behind. But don’t use

--background

in containers—run them in the foreground as the container’s main process. Also, don’t co-locate them with your flow execution “process workers” if you can avoid it; run a dedicated services app. Recommended setup - Create a separate ACA app (or deployment) for services and run:

prefect server services start

- Ensure your API pods run without services to avoid duplicates:

prefect server start --no-services

- Configure Redis-backed messaging (and install the integration package in the image): - Install:

pip install prefect-redis

- Env vars:

PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

PREFECT_REDIS_MESSAGING_HOST=...
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# optional:
PREFECT_REDIS_MESSAGING_USERNAME=...
PREFECT_REDIS_MESSAGING_PASSWORD=...
PREFECT_REDIS_MESSAGING_SSL=false

# point to the same DB your API uses
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@host:5432/db>

- Run multiple replicas of the services container to increase throughput; each replica adds consumers to the

event-persister

,

task-run-recorder

, and

reactive-triggers

groups and will help drain the backlog. Why not --background in containers? -

--background

daemonizes a child and returns. In containers, you generally need one foreground process to keep the container alive; backgrounding can lead to containers exiting or make lifecycle management and logs harder. Use the foreground command as the container’s entrypoint. Can I run services inside my “process worker” containers? - Possible, but not recommended. Mixing flow execution and server services in one container competes for CPU/memory and complicates restarts and scaling. Prefer a dedicated “prefect-services” app so you can scale it independently to drain backlog, then scale down. After services are running - Verify consumers are catching up:

redis-cli XINFO STREAM events
redis-cli XINFO GROUPS events

- Remove stale groups so they don’t affect trimming:

redis-cli XGROUP DESTROY events <stale_group_name>

- Once caught up, stream trimming will happen periodically with the services. You can tune:

PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_EVERY=60
PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_IDLE_THRESHOLD=300

If you share how your ACA apps are defined (image/command/env), I can suggest an exact command and env block to drop into your deployment and an initial replica count to drain the current lag.

It selects where the server stores “leases” for concurrency limits. In Prefect 3, a concurrency limit is enforced by issuing short-lived leases when a task/flow acquires slots. Those leases must be stored somewhere that all server/service instances can see. - Memory backend (default): in-process dict; fine for single-process dev, not safe for multi-worker. - Filesystem backend: JSON files on disk; survives restarts but not safe for distributed/multi-worker. - Redis backend: distributed, atomic operations; required for multi-worker/server replicas. So setting

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

tells Prefect to use the Redis implementation from prefect-redis to store and coordinate leases. That enables safe concurrency limits when your API/services run across multiple containers/replicas. What the lease lifecycle looks like - Acquire: server checks a limit, increments active slots, and writes a lease record to the lease storage with a TTL. - Renew: long-running holders periodically renew the lease. - Release: on completion, the lease is revoked and slots are decremented. - Expire: if a holder dies without releasing, the “repossessor” service finds expired leases in storage and frees their slots. When you need it - Any time you run multiple API workers or separate services processes/replicas. Without Redis-backed lease storage, different processes won’t see each other’s leases and your limits won’t be enforced correctly. To use it - Install the integration in the image:

pip install prefect-redis

- Set the env var (and your Redis connection settings):

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_REDIS_MESSAGING_HOST=...
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# optional auth/ssl vars as needed

If you share your deployment topology (number of API workers and services replicas), I can confirm whether Redis lease storage is required and suggest reasonable TTL/renewal settings.

Short answer: - Pausing work pools helps reduce new event production (fewer flow/task runs → fewer new events). - Increasing API workers does not drain the Redis “events” stream. Only the server “services” (event-persister, task-run-recorder, reactive-triggers) consume from that stream. If services aren’t running (or are under-provisioned), the backlog won’t drop. Important nuance about API workers: - In multi-worker mode, the API runs without services; you must run services in separate processes/containers to consume events. - If you previously had services running in a single-process server and then switched to multiple API workers without starting services separately, consumption may actually have stopped. What to do next: - Start dedicated services containers and scale them out until lag drops:

prefect server services start

(run in the foreground as the container’s main process; don’t use --background in containers) - Keep pools paused until the backlog (lag in XINFO GROUPS) catches up. - Remove stale consumer groups so trimming can work once you’re caught up:

redis-cli XGROUP DESTROY events <stale_group_name>

- Watch progress:

redis-cli XINFO GROUPS events
redis-cli XINFO STREAM events

If you share how many replicas you can run for services, I can suggest an initial replica count and a quick drain-time estimate based on your current lag.

Yes. You can run only the event consumer(s) that will drain the “events” stream. Two approaches depending on what you want to preserve: 1) Fastest drain to DB (may skip automations/task updates during the drain) - Run only the Event Persister. It reads from the Redis “events” stream, writes events to the DB, and participates in periodic trimming. With the other groups idle, the trimmer will ignore them after the idle threshold and trim based on the persister’s progress. - Use env vars to disable other services:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false

- Start the services process (in containers, run in the foreground):

prefect server services start

Trade-offs: automations (reactive triggers) and task-run recording will not process new events while they’re disabled; they won’t “catch up” for any entries trimmed while they were idle. 2) Drain while keeping full behavior (recommended if you don’t want to miss anything) - Run all three services and scale replicas to increase throughput: - Event Persister (persists to DB) - Task Run Recorder (updates task/flow states) - Reactive Triggers (automations) - Just start the services process (all enabled by default) and scale out replicas:

prefect server services start

- This way, each consumer group advances its last-delivered-id, and the built-in trimmer can safely trim without skipping groups. Prereqs and tips - Ensure you’re using Redis-backed messaging and have prefect-redis installed in the services image:

pip install prefect-redis
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
# plus your Redis connection/env

- Don’t use

--background

in containers; keep the process in the foreground. - Remove stale consumer groups to help trimming once you’re caught up:

redis-cli XGROUP DESTROY events <stale_group_name>

- Watch progress:

redis-cli XINFO GROUPS events
redis-cli XINFO STREAM events

If you tell me whether you’re okay skipping triggers/task updates during the drain, I’ll suggest the exact env/command set and a starting replica count to clear your current lag.

Great question. To drain the Redis “events” stream fastest, focus on two layers: the Redis consumer side (how quickly we pull from the stream) and the server’s EventPersister (how quickly we write to the DB). Here are the knobs that actually move the needle, plus a few caveats. High-impact settings - Scale out services replicas - Biggest win: run more replicas of the services process. Each replica adds consumers to: - event-persister (persists to DB) - task-run-recorder - reactive-triggers - More replicas = more parallel consumption. - EventPersister batching to DB - Increase batch size (bigger inserts, fewer transactions):

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=100

- Reduce flush interval so batches flush steadily:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1

- If DB is the bottleneck, raise batch size cautiously (100–500) and watch DB CPU/IO and lock contention. - Redis consumer cadence (prefect-redis) - Block time on reads:

PREFECT_REDIS_MESSAGING_CONSUMER_BLOCK=1

- Min idle reclaim time (faster retry of stuck messages):

PREFECT_REDIS_MESSAGING_CONSUMER_MIN_IDLE_TIME=5

- Retry limit (don’t spin endlessly on bad messages):

PREFECT_REDIS_MESSAGING_CONSUMER_MAX_RETRIES=3

- Trim cadence (keep memory in check while you drain):

PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_EVERY=30
  PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_IDLE_THRESHOLD=300

Important caveat about Redis consumer batch size - The prefect-redis consumer currently reads with COUNT=1 in XREADGROUP and processes one message at a time. There isn’t a configurable “consumer batch size” setting today. - Because of that, the best way to increase aggregate read throughput is to run more service replicas (horizontal scale), not to tweak a per-consumer COUNT. Other levers and checks - Remove stale consumer groups so trimming isn’t held back:

redis-cli XGROUP DESTROY events <stale_group_name>

- Ensure you’re not starving the services containers for CPU/memory; they’re async but benefit from CPU. - Watch for DB bottlenecks (slow inserts). If DB is slow, batch size helps up to a point; you may need DB tuning (indexes, connection pool size, instance size). Suggested starting config to drain backlog - Run N replicas of services (start with 3–5 if you can). - Set:

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=200
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1
PREFECT_REDIS_MESSAGING_CONSUMER_TRIM_EVERY=30

- Monitor:

redis-cli XINFO GROUPS events
redis-cli XINFO STREAM events

If you share your DB size/capacity and how many service replicas you can run, I can suggest a concrete replica count and batch size tuned to your setup.

Short answer: pausing your work pools/queues helps. Why it helps - Fewer flow/task runs = far fewer new events/logs being published to the Redis “events” stream and less DB write pressure. That gives your services a chance to catch up and shrink the backlog. - It doesn’t drain the queue by itself; it just slows/halts ingress so your consumers (event-persister, task-run-recorder, reactive-triggers) can make progress. What not to pause - Don’t pause/stop the server services; they are the consumers that drain the stream. Keep them running and scale them out if possible. Suggested approach - Keep pools/queues paused while you: - Start/scale dedicated services containers. - Remove stale consumer groups. - Monitor lag until near zero. - Then resume pools gradually to avoid a sudden spike. Monitor progress

redis-cli XINFO STREAM events
redis-cli XINFO GROUPS events

Watch the “lag” and “last-delivered-id” for your active groups come up to the stream’s last-generated-id. Caveat - If you disable certain services (e.g., triggers or task-run-recorder) to devote all bandwidth to persisting events, you’ll drain faster but may skip those services’ processing for trimmed entries. If you care about full behavior, keep all three services enabled and just scale them out.