Yes — if you’re running multiple Prefect Server background-service replicas in HA, this is a known failure mode unless you’ve configured shared Docket coordination via Redis. Most likely root cause: both replicas are running the scheduler loop independently and each is creating scheduled flow runs. Prefect’s self-hosted HA docs explicitly warn about this: Self-hosted Prefect docs Specifically, in Prefect 3.x: - background services need shared coordination - the default Docket backend is

memory://

-

memory://

is process-local, so 2 replicas = 2 independent schedulers - result: duplicate scheduled runs, duplicate automation actions, and other correctness issues What to check first 1. Are you running more than 1

prefect server services

replica? If yes, this is the first thing I’d suspect. 2. Do all service replicas have

PREFECT_SERVER_DOCKET_URL

set to Redis? You want something like:

bash
   PREFECT_SERVER_DOCKET_URL=redis://<redis-host>:6379/1

3. Are both replicas emitting scheduler logs at the same time? If both are scheduling every interval, that strongly confirms the issue. How to diagnose 1. Inspect your deployment config / env vars Make sure all background service pods share the same:

bash
PREFECT_SERVER_DOCKET_URL=redis://...

If it is unset, Prefect uses:

bash
memory://

which is not safe for multi-replica background services. 2. Check logs from both service pods Look for scheduler activity from multiple pods in the same window. For example, if both pods are logging scheduling work every ~10s, that’s the smoking gun. 3. Check the database for duplicate scheduled runs A useful query is to group by deployment + scheduled start time:

sql
SELECT
    deployment_id,
    next_scheduled_start_time,
    COUNT(*) AS run_count,
    MIN(created_at) AS first_created,
    MAX(created_at) AS last_created
FROM flow_run
WHERE auto_scheduled = true
  AND created_at > NOW() - INTERVAL '4 hours'
GROUP BY deployment_id, next_scheduled_start_time
HAVING COUNT(*) > 1
ORDER BY first_created DESC;

If you see duplicates created within seconds, that’s consistent with two schedulers racing. Recommended fix If you want HA for Prefect Server background services, configure shared Redis-backed Docket for all replicas. Example:

bash
PREFECT_SERVER_DOCKET_URL=<redis://redis-service:6379/1>

All service replicas must use the same value. Important nuance This is separate from using Karpenter for worker/job scheduling. Karpenter may affect where work runs, but duplicate scheduled flow runs are typically a server-side scheduling coordination problem, not a work-pool autoscaling problem. Practical recommendations - If you do not have Redis-backed Docket configured: - reduce background services to 1 replica immediately, or - add Redis-backed Docket before scaling back to 2+ - verify only one scheduler loop is effectively active per interval - monitor for duplicate

flow_run

creation after the change If you want, I can help you: - verify whether your current Helm/EKS setup is vulnerable - identify which Prefect pods should be singleton vs multi-replica - draft the exact Kubernetes env/config changes for your deployment

I guess the continuous training on Marvin is pretty good: <https://linen.prefect.io/t/33252122/ulva73b9p-i-am-experiencing-the-following-in-prefect-3-since%7C< Marvin> I am experiencing the following in Prefect 3 Since Prefect Community #ask-marvin>

@Marvin 1. what happens when Redis goes down when running Prefect with 2 replicas? 2. What key spaces does Prefect consume

Short version: 1. If Redis goes down, your multi-replica Prefect background services generally stop coordinating and stop making progress until Redis comes back. 2. Prefect uses Redis for a few internal namespaces, mainly Docket/service coordination and optionally event messaging / automations. 1) What happens if Redis goes down with 2 replicas? Assuming your 2 replicas are using:

bash
PREFECT_SERVER_DOCKET_URL=redis://...

then Redis is the coordination layer for background services. Expected behavior - background services fail closed - schedulers/automations/cleanup loops pause or repeatedly retry - API/database-backed parts of Server may still respond - once Redis recovers, services typically reconnect and resume So in practice, during a Redis outage you should expect: - no new scheduled runs being created while Redis is unavailable - automation/event processing to stall if you’re also using Redis-backed messaging - late-run detection / cleanup / other perpetual services to pause - workers already running flow runs may continue executing, but orchestration-side updates can lag depending on what they need to talk to This is usually preferable to the alternative: duplicate schedulers both acting independently. Operationally: with 2 replicas, Redis becomes part of the control plane. If Redis is a single instance, it can become a single point of failure. Recommendation - use HA Redis if you need HA Prefect background services - at minimum, use a managed Redis or Sentinel-backed deployment - monitor for scheduler lag and Redis connection errors 2) What key spaces does Prefect consume? There are two main buckets to think about. A. Docket keys Used for background service coordination. Typical prefixes look like:

text
prefect-server:queue
prefect-server:stream
prefect-server:runs:*
prefect-server:cancel:*

These are used for things like: - scheduling/coordinating service work - tracking in-flight service tasks - cancellation signaling B. Event / messaging keys If you’re using Redis-backed messaging for events/automations, keys look more like:

text
{message:events}:stream
{message:events}:dedupe:*
{message:events}:dlq

These are used for: - event streams - deduplication - dead-letter queue behavior Important note The exact prefix can depend on configuration and implementation details, but the above patterns are the important ones to expect operationally. What I’d recommend for your setup If you’re on EKS and already using HA Prefect Server: - keep Postgres HA as well - make Redis HA too, otherwise you’ve just shifted the SPOF from scheduler coordination into Redis - collect: - Redis availability metrics - Prefect server service logs - scheduler lag / duplicate run counts - stream lengths if using Redis events A practical way to inspect keys From a Redis shell:

bash
SCAN 0 MATCH "prefect-server:*" COUNT 100
SCAN 0 MATCH "{message:events}:*" COUNT 100

And to inspect type/cardinality:

bash
TYPE prefect-server:stream
TYPE prefect-server:queue
XLEN {message:events}:stream

If you want, I can also help with: - a recommended HA topology for Prefect Server + Redis on EKS - which Prefect components depend on Docket vs Postgres only - a Redis monitoring checklist for duplicate-run diagnosis