<@ULVA73B9P>, can you explain how Prefect is likel...
# ask-marvin
s
@Marvin, can you explain how Prefect is likely using Redis in this case?
Copy code
prefect-reseng-redis-svc.prefect.svc.cluster.local:6379> GET "message:messaging-cache:019aa797-c2c3-79cc-9412-00b96f218fe7"
"1"
prefect-reseng-redis-svc.prefect.svc.cluster.local:6379> GET "message:messaging-cache:019aa798-e8c8-77e7-8d67-6178aa1a1f1f"
"1"
m
thought for 403.6 seconds
Short answer: those keys are Prefect’s Redis-backed message deduplication markers. What you’re seeing - Key pattern:
message:messaging-cache:<attribute_value>
- Value:
"1"
(just a marker) - TTL: ~5 minutes, then Redis auto-expires it Why they exist - Prefect’s server uses a pluggable messaging system. When Redis is configured as the messaging cache, it writes a short-lived “seen” record per message to prevent duplicate processing. - The cache topic is fixed as
messaging-cache
for these markers; the actual messages travel on separate Redis Streams by topic (e.g.,
events
,
actions
,
logs
), not as these keys. - For events, Prefect deduplicates by the event
id
, so you’ll see keys like
message:messaging-cache:<event_uuid>
. How it works under the hood - On publish, Prefect tries to set a key like
message:messaging-cache:<dedup_attribute>
with `SET NX EX <ttl>`; if it exists, the message is considered a duplicate and skipped. TTL defaults to 5 minutes. - This is implemented in the Prefect Redis integration’s messaging cache: - Source (dedup keys and TTL): prefect_redis/messaging.py - Key format:
message:{self.topic}:{m.attributes[attribute]}
- TTL constant:
MESSAGE_DEDUPLICATION_LOOKBACK = timedelta(minutes=5)
- Server messaging factory (how the cache/broker are loaded): prefect/server/utilities/messaging/__init__.py - Events publisher (deduplicate by
id
): prefect/server/events/messaging.py How to verify in your environment - Check your Prefect settings to confirm Redis is being used as the messaging cache/broker:
Copy code
prefect config view --show-defaults | grep -E 'MESSAGING|REDIS'
Look for something like:
Copy code
PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
PREFECT_REDIS_MESSAGING_HOST=...
- Inspect TTL on a key:
Copy code
TTL "message:messaging-cache:019aa797-c2c3-79cc-9412-00b96f218fe7"
- See how many are around (use SCAN in prod):
Copy code
SCAN 0 MATCH "message:messaging-cache:*" COUNT 100
Docs - Self-hosted/Redis setup: Self-hosted Prefect (Redis setup) - Settings reference (messaging_broker/cache): Settings reference Notes - Prefect uses Redis Streams (not Pub/Sub) for durable messaging; these
message:messaging-cache:*
keys are only for deduplication, not the messages themselves. - Keys are expected and ephemeral; they’ll expire automatically. If helpful, tell me your Prefect version and how your server is deployed, and I can map exactly which topics are deduplicated and where those keys originate in your setup.