Sebastian S
11/21/2025, 6:09 PMprefect-reseng-redis-svc.prefect.svc.cluster.local:6379> GET "message:messaging-cache:019aa797-c2c3-79cc-9412-00b96f218fe7"
"1"
prefect-reseng-redis-svc.prefect.svc.cluster.local:6379> GET "message:messaging-cache:019aa798-e8c8-77e7-8d67-6178aa1a1f1f"
"1"Marvin
11/21/2025, 6:09 PMMarvin
11/21/2025, 6:15 PMmessage:messaging-cache:<attribute_value>
- Value: "1" (just a marker)
- TTL: ~5 minutes, then Redis auto-expires it
Why they exist
- Prefect’s server uses a pluggable messaging system. When Redis is configured as the messaging cache, it writes a short-lived “seen” record per message to prevent duplicate processing.
- The cache topic is fixed as messaging-cache for these markers; the actual messages travel on separate Redis Streams by topic (e.g., events, actions, logs), not as these keys.
- For events, Prefect deduplicates by the event id, so you’ll see keys like message:messaging-cache:<event_uuid>.
How it works under the hood
- On publish, Prefect tries to set a key like message:messaging-cache:<dedup_attribute> with `SET NX EX <ttl>`; if it exists, the message is considered a duplicate and skipped. TTL defaults to 5 minutes.
- This is implemented in the Prefect Redis integration’s messaging cache:
- Source (dedup keys and TTL): prefect_redis/messaging.py
- Key format: message:{self.topic}:{m.attributes[attribute]}
- TTL constant: MESSAGE_DEDUPLICATION_LOOKBACK = timedelta(minutes=5)
- Server messaging factory (how the cache/broker are loaded): prefect/server/utilities/messaging/__init__.py
- Events publisher (deduplicate by id): prefect/server/events/messaging.py
How to verify in your environment
- Check your Prefect settings to confirm Redis is being used as the messaging cache/broker:
prefect config view --show-defaults | grep -E 'MESSAGING|REDIS'
Look for something like:
PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
PREFECT_REDIS_MESSAGING_HOST=...
- Inspect TTL on a key:
TTL "message:messaging-cache:019aa797-c2c3-79cc-9412-00b96f218fe7"
- See how many are around (use SCAN in prod):
SCAN 0 MATCH "message:messaging-cache:*" COUNT 100
Docs
- Self-hosted/Redis setup: Self-hosted Prefect (Redis setup)
- Settings reference (messaging_broker/cache): Settings reference
Notes
- Prefect uses Redis Streams (not Pub/Sub) for durable messaging; these message:messaging-cache:* keys are only for deduplication, not the messages themselves.
- Keys are expected and ephemeral; they’ll expire automatically.
If helpful, tell me your Prefect version and how your server is deployed, and I can map exactly which topics are deduplicated and where those keys originate in your setup.