Short answer: No. In Prefect 3.6.23 (self‑hosted), the events subsystem’s Redis backend supports a single standalone Redis instance (via Redis Streams). Redis Cluster and Redis Sentinel are not supported for events messaging. Details: - Default: Prefect Server uses in‑memory messaging by default (fine for single worker). - Multi‑worker: You must switch events messaging, ordering, and lease storage to the Redis integration. This backend connects to a single Redis host:port (Redis Streams), not Cluster or Sentinel. - Config uses discrete host/port/db settings (not a Redis URL for messaging), e.g.:

export PREFECT_REDIS_MESSAGING_HOST="redis-host"
  export PREFECT_REDIS_MESSAGING_PORT="6379"
  export PREFECT_REDIS_MESSAGING_DB="0"

  export PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
  export PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
  export PREFECT_SERVER_EVENTS_CAUSAL_ORDERING="prefect_redis.ordering"
  export PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE="prefect_redis.lease_storage"

Docs: - Run a local Prefect server - Scale self-hosted Prefect (multi-worker requirements) - Prefect Redis integration If you need HA/failover, common patterns are to use a managed Redis that presents a single endpoint or put a proxy in front of a clustered/sentinel setup so Prefect still sees one host:port. If that’s your goal, happy to discuss options.

Short version: - If Redis is down when the API tries to publish events, the POST /events call fails with a 5xx and the events are not buffered by Prefect Server. Data is lost unless the caller retries. - If Redis goes down after events have already been written to the Redis stream, there’s no data loss. The event consumer services back off and reconnect; pending stream entries remain and are processed when Redis returns. - There’s no automatic fallback from Redis to in-memory, and no local buffering of events when Redis is unavailable. What actually happens - Publish-time failure (Redis unavailable): - The API route /events batches and forwards events to the configured messaging broker. - With the Redis broker, a failed

XADD

re-raises an exception. The API does not swallow it, so the client gets an HTTP error and those events are not stored or queued by Prefect. - Consume-time failure (Redis becomes unavailable later): - Events already in the Redis Stream are durable. - The event persister/consumers use a reconnect loop with backoff and resume from the stream once Redis is back. - Message processing failures (handler errors) are retried up to a limit; after that they go to a DLQ (for Redis backend, the DLQ is stored in Redis). This DLQ is not used for publish-time outages. Implications and mitigations - To avoid data loss when Redis is down at publish time: - Add client-side retry with exponential backoff around POST /events (whatever is emitting your events should retry on non-2xx). - Run a highly-available Redis endpoint (managed Redis with automatic failover, or a proxy that presents a single stable host:port). Prefect does not natively support Redis Cluster or Sentinel, so HA needs to be provided behind one endpoint. - Monitor Redis and Prefect server health so you can remediate quickly. - For critical signals, consider dual-writing to your own durable store (e.g., object storage or a logging pipeline) so you have an audit trail even if Redis is briefly unavailable. Docs - Run a local Prefect server - Scale self-hosted Prefect (multi-worker + Redis requirements) - Prefect Redis integration If you want, I can share a minimal retry snippet you can wrap around event POSTs, or we can talk through HA options for your environment.

Short answer: No. If Redis is down at publish time, Prefect Server does not automatically retry or buffer those events. The POST /events call fails (5xx), and it’s up to the emitter/client to retry. There’s no fallback to in‑memory or local spool for publish failures. Once events are successfully written to Redis Streams, consumers will reconnect with backoff and retry processing, and failed message handling goes to a DLQ—but that’s after publish, not for publish-time outages. Recommended mitigations: - Add client-side retries with exponential backoff around POST /events. - Use a highly available Redis endpoint (managed Redis or a proxy that presents a single stable host:port). - Monitor Redis/Prefect health and alert on failures. Docs: - Run a local Prefect server - Scale self-hosted Prefect (multi-worker + Redis requirements) - Prefect Redis integration

Great idea. Here are simple, production-friendly client-side retry patterns with exponential backoff (and jitter) around POST /events. Notes - Endpoint (self-hosted default): http://127.0.0.1:4200/api/events - Body: a JSON array of events. Include a stable UUID in each event’s

id

so retries are idempotent. - Minimal event fields to include safely:

event

,

resource

,

occurred

,

id

. Python (requests, synchronous)

import time
import uuid
import random
import requests
from datetime import datetime, timezone

API_URL = "<http://127.0.0.1:4200/api/events>"

def now_iso():
    return datetime.now(timezone.utc).isoformat()

def post_events_with_retry(events, max_attempts=8, base_sleep=0.5, max_sleep=30, timeout=(3.0, 10.0)):
    """
    Exponential backoff with jitter. Retries on 5xx and connection errors.
    - max_attempts: total tries including the first
    - base_sleep: initial sleep in seconds
    - max_sleep: cap for backoff
    - timeout: (connect_timeout, read_timeout)
    """
    attempt = 0
    while True:
        attempt += 1
        try:
            resp = <http://requests.post|requests.post>(API_URL, json=events, timeout=timeout)
            # Retry on 5xx; 2xx/4xx return immediately
            if 500 <= resp.status_code < 600:
                raise RuntimeError(f"Server error {resp.status_code}: {resp.text}")
            resp.raise_for_status()
            return resp
        except (requests.exceptions.RequestException, RuntimeError) as e:
            if attempt >= max_attempts:
                raise
            # Exponential backoff with jitter
            sleep = min(max_sleep, base_sleep * (2 ** (attempt - 1)))
            sleep += random.uniform(0, sleep * 0.25)  # jitter up to 25%
            time.sleep(sleep)

# Build an idempotent example event
event = {
    "event": "example.event.occurred",
    "resource": {"prefect.resource.id": "example.resource:123"},
    "occurred": now_iso(),
    "id": str(uuid.uuid4())
}

# Send with retries
response = post_events_with_retry([event])
print("Status:", response.status_code)  # 204 on success

Python (httpx, async)

import asyncio
import uuid
import random
import httpx
from datetime import datetime, timezone

API_URL = "<http://127.0.0.1:4200/api/events>"

def now_iso():
    return datetime.now(timezone.utc).isoformat()

async def post_events_with_retry_async(events, max_attempts=8, base_sleep=0.5, max_sleep=30.0, timeout=10.0):
    attempt = 0
    async with httpx.AsyncClient(timeout=timeout) as client:
        while True:
            attempt += 1
            try:
                resp = await <http://client.post|client.post>(API_URL, json=events)
                if 500 <= resp.status_code < 600:
                    raise RuntimeError(f"Server error {resp.status_code}: {resp.text}")
                resp.raise_for_status()
                return resp
            except (httpx.RequestError, httpx.HTTPStatusError, RuntimeError):
                if attempt >= max_attempts:
                    raise
                sleep = min(max_sleep, base_sleep * (2 ** (attempt - 1)))
                sleep += random.uniform(0, sleep * 0.25)
                await asyncio.sleep(sleep)

event = {
    "event": "example.event.occurred",
    "resource": {"prefect.resource.id": "example.resource:123"},
    "occurred": now_iso(),
    "id": str(uuid.uuid4())
}

async def main():
    resp = await post_events_with_retry_async([event])
    print("Status:", resp.status_code)

asyncio.run(main())

Bash (curl) quick-and-dirty backoff loop ``` #!/usr/bin/env bash set -euo pipefail API_URL="http://127.0.0.1:4200/api/events" # Build a minimal JSON payload inline; replace UUID as needed PAYLOAD='[ { "event": "example.event.occurred", "resource": { "prefect.resource.id": "example.resource:123" }, "occurred": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'", "id": "550e8400-e29b-41d4-a716-446655440000" } ]' attempt=0 max_attempts=8 sleep_secs=0.5 max_sleep=30 while true; do attempt=$((attempt+1))

status=$(curl -s -o /tmp/resp.out -w "%{http_code}" -X POST \ -H "Content-Type: application/json" \ --data "$PAYLOAD" \ "$API_URL") if [[ "$status" =~ ^2 ]]; then echo "Success: $status" exit 0 elif [[ "$status" =~ ^5 ]]; then if [[ $attempt -ge $max_attempts ]]; then echo "Failed after $attempt attempts: HTTP $status" cat /tmp/resp.out exit 1 fi # exponential backoff with jitter (25%) jitter=$(python - <<'PY' import random, sys print(random.uniform(0, float(sys.argv[1])*0.25)) PY "$sleep_secs") sleep_time=$(python - <<'PY' import sys base=float(sys.argv[1]); jitter=float(sys.argv[2]); cap=float(sys.argv[3]) print(min(cap, base + jitter)) PY "$sleep_secs" "$jitter" "$max_sleep") echo "HTTP $status; retrying in $sleep_time s (attempt $attempt/$max_attempts)..." sleep "$sleep_time" sleep_secs=$(python - <<'PY' import sys print(float(sys.argv[1])*2) PY "$sleep_secs") else echo "Non-retryable status: $status" cat /tmp/resp.out exit 1 fi done ``` Tips - Use a stable UUID per event (

id

) so retries are idempotent server-side. - Consider retrying on connection errors/timeouts and 5xx; usually don’t retry 4xx. - Keep a reasonable cap on backoff (e.g., 30–60s) and a max attempts or total timeout. - Log failures and payload IDs so you can reconcile if retries ultimately fail. More on events: Prefect Events