<@ULVA73B9P> - Using self hosted prefect - 3.6.22 ...
# ask-community
c
@Marvin - Using self hosted prefect - 3.6.22 and using single node Redis. Do we have Redis Cluster support for Prefect?. If so, can you share the documentation.
m
thought for 258.8 seconds
Short answer: No—Prefect 3.6.22 (Prefect 3.x) does not officially support Redis Cluster or Redis Sentinel. Prefect’s self-hosted server expects a single Redis instance (with optional TLS/auth). There isn’t Prefect documentation for Redis Cluster because it isn’t supported. Relevant docs for configuring Redis in self-hosted Prefect: - How to scale self-hosted Prefect - Run the Prefect Server via Docker Compose - Run a local Prefect Server - Customize caching behavior (mentions Redis-based locking) Typical environment variables you’ll see in the self-hosted setup:
Copy code
# Use Redis for messaging/broker/cache
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging

# Event ordering and concurrency lease storage via Redis
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

# Redis connection details
PREFECT_REDIS_MESSAGING_HOST=redis-host
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# Optional auth/TLS
PREFECT_REDIS_MESSAGING_USERNAME=username
PREFECT_REDIS_MESSAGING_PASSWORD=password
PREFECT_REDIS_MESSAGING_SSL=true

# Docket (background services coordination)
PREFECT_SERVER_DOCKET_URL=<redis://redis-host:6379/0>
# or with TLS/auth:
# PREFECT_SERVER_DOCKET_URL=<rediss://username:password@redis-host:6379/0>
Why Cluster isn’t supported today: - Prefect relies on Redis features like Lua scripts (for leases) and Redis Streams-based messaging that aren’t cluster-transparent. - The integration uses standard single-node Redis clients and configuration (host/port/db/ssl), not cluster-aware clients. If your goal is high availability, common approaches are: - A managed Redis service that exposes a single endpoint (e.g., ElastiCache/Memorystore/Azure Cache with a primary endpoint). This works because Prefect just needs a stable single-node endpoint. - A Sentinel/proxy layer can sometimes be used to present a single endpoint, but it’s not officially supported or tested by Prefect—use with caution. If you can share your HA/throughput requirements, I’m happy to suggest a concrete architecture that fits within supported patterns.