Hi <@ULVA73B9P> I am considering using the <dragon...
# ask-community
p
Hi @Marvin I am considering using the dragonfly k8s operator as a replacement for bitnamilegacy/redis in Prefect server OSS. Has it ever been tested ? Will it work ?
m
thought for 578.8 seconds
Short answer: we haven’t officially tested DragonflyDB with Prefect Server OSS, and we don’t have an internal support statement for it. It will likely work if Dragonfly fully supports the Redis features Prefect uses, but you should validate before switching in production. What Prefect uses Redis for (Prefect 3.x assumed): - Redis is optional. Prefect Server runs out-of-the-box with an in-memory broker; Redis is typically added for durability and multi-replica/HA event + log distribution, distributed locks, and concurrency limits. - The Prefect Redis integration uses Redis Streams for messaging (publisher/consumer groups) and standard primitives for locks/leases. - Expect commands like XADD/XGROUP/XREADGROUP/XACK/XTRIM, and SET NX EX/PX for locks. DragonflyDB compatibility considerations: - Dragonfly markets Redis compatibility and (as of recent releases) supports Streams. If Dragonfly fully supports Streams and consumer groups semantics, it should work with Prefect’s Redis integration. - We do not have recorded tests or community reports confirming Dragonfly + Prefect. I couldn’t find issues/discussions referencing Dragonfly in our repos. - No issues found: GitHub issues search - No discussions found: GitHub discussions How to wire Prefect Server to a Redis-compatible broker (Dragonfly) in 3.x: 1) Ensure the server image has the Redis integration installed:
Copy code
pip install "prefect[redis]"
Docs: prefect-redis integration 2) Point Prefect to the Redis-backed broker and cache by setting these environment variables on your Prefect Server pods (API and services):
Copy code
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
Settings reference: settings reference 3) Provide Redis/Dragonfly connection details used by prefect-redis. The integration exposes settings via
RedisMessagingSettings
and related helpers; you can typically provide a standard Redis URL or host/port/auth via env vars. See the integration docs above for the exact options accepted. Notes for Helm/Kubernetes: - If you previously deployed Bitnami Redis via a subchart, you can remove that and point Prefect to your Dragonfly service instead by setting the env vars above on the Prefect Server workloads. Ensure the Prefect server image includes
prefect-redis
. - If you’re not running multiple replicas or don’t need cross-pod log/event distribution, you can skip Redis entirely and stay on the default in-memory broker. A quick validation plan: - Verify Dragonfly Streams end-to-end before switching Prefect: - Check that XADD/XGROUP/XREADGROUP/XACK work with consumer groups. - Stand up a test Prefect Server with:
Copy code
PREFECT_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_MESSAGING_CACHE=prefect_redis.messaging
- Run a few flows, watch that: - Flow/task logs appear reliably in the UI - Events render in the timeline - With multiple server replicas, logs/events are visible across pods - If you use concurrency limits or distributed locks, test those as well Open questions for you (to tailor guidance): - Which Prefect version and Helm chart are you on? - Do you plan to run the Server in HA (multiple replicas)? - Will you use Redis for just logs/events, or also for locks/concurrency? If you share your version/chart and whether you’re aiming for HA, I can suggest a concrete values.yaml/env setup for the Prefect pods. Links for reference: - Prefect Helm: Server on Helm and prefect-helm repo - Prefect Redis integration: prefect-redis
p
I have now prefect OSS using dragonfly through the operator, installed with helm charts. Everything seems to work fine: - Flow/task logs appear reliably in the UI - Events render in the timeline - With multiple server replicas, logs/events are visible I put two replicas for each component and set runAsSeparateDeployment: true. Thanks
Finally I removed my setup using dragonfly with 2 pod replicas. Because of probable connection hiccups, the leader pod was changed quite often, but Prefect sometime was not aware of it, which lead Prefect to stay connected to the slave/replica dragonfly DB. That gave the error
You can't write against a read only replica
. I could not find explanations due to a misconfiguration, but cannot totally rule it out.