What you describe is most consistent with the scheduler service getting stuck, not with your deployment schedule being disabled. In self-hosted Prefect 3.x, scheduling is performed by background services, not by the deployment itself. So if: - the schedule still shows as active - no new flow runs are being created - restarting the server/background services makes scheduling resume then the likely failure mode is: the scheduler loop stopped making progress while the deployment metadata remained intact. What likely happened On self-hosted Prefect, scheduled runs are created by background services that run in a perpetual loop. In HA/self-hosted setups, this depends on: - the background services pod - Redis for coordination / service state - the database for reading deployments and inserting scheduled runs A few likely causes fit your symptoms: 1. A scheduler/background-service task got stuck in Redis state - If Redis had a transient disconnect/failover/network blip, the scheduling task can end up effectively “running” forever from the coordinator’s perspective. - The deployment still appears active, but no future runs are generated. - Restarting the pods resets that state and scheduling resumes. 2. The background worker loop died or stopped progressing - A timeout or uncaught exception during Redis or DB communication can cause the scheduling worker to stop doing useful work without producing obvious logs. - The pod may still look healthy from Kubernetes’ point of view. 3. Database pressure / connection pool exhaustion - If the scheduler can’t reliably query deployments or insert flow runs, it may silently fail or stall. - This is more likely if your instance has many deployments/runs over time. 4. Scheduler query degradation over time - There have been scheduler-related fixes in newer Prefect versions around scheduling reliability and performance. - On a busy server, this can show up as “works for days, then stops until restart.” Important point: this does not sound like “your cron/schedule turned off.” It sounds like “the service responsible for materializing scheduled runs stopped running correctly.” Why restart fixed it Restarting the

prefect server

and background services pods likely: - reinitialized the scheduler loop - reacquired coordination state in Redis - cleared stale in-memory or Redis-backed service state - reestablished fresh DB/Redis connections That’s why new runs started being scheduled again immediately. Most likely practical root cause Given your topology: - 1 background services pod - 2 API/server pods - 1 Redis - Kubernetes the strongest suspicion is a scheduler/background-service coordination issue involving Redis, possibly triggered by a transient Redis/network interruption or a stuck perpetual service state. What to inspect next time before restarting If it happens again, check these before bouncing pods: 1. Background services logs Look for scheduler-related messages: -

schedule_deployments

-

schedule_recent_deployments

- Redis timeout / connection errors - DB timeout / pool errors 2. Whether scheduled runs are still being created in the DB If no new

auto_scheduled

flow runs are appearing, the scheduler is stalled. 3. Redis health / restarts / failovers Check: - pod restarts - OOMKills - brief disconnects - latency spikes 4. Postgres health Check: - connection saturation - slow queries - pool exhaustion - lock contention 5. Pod liveness vs actual scheduler activity A pod can be “healthy” to Kubernetes while the scheduler loop is effectively dead. What I’d recommend 1. Upgrade Prefect You’re on

3.6.12

, and there have been scheduler/perpetual-service reliability fixes in later 3.x releases. If possible, upgrade to a newer 3.7.x or later release. 2. Add monitoring for “last scheduled run created”

Alert if no new scheduled flow runs are created for, say, 15–20 minutes for a deployment that should schedule every 10 minutes. 3. Watch Redis closely Since Redis is part of the coordination path, make sure it is stable and not restarting or timing out. 4. Check DB connection settings / capacity Especially if your server has been accumulating lots of runs and deployments. 5. Capture logs before restart next time The most valuable evidence will be logs from the background services pod right before recovery. Bottom line The most likely answer is: Prefect’s self-hosted scheduler/background service stopped progressing—probably due to a Redis- or DB-related coordination/stall issue—so active schedules remained visible but no new flow runs were materialized until the scheduler services were restarted. If you want, I can help you build a targeted troubleshooting checklist for your Kubernetes deployment, including: - which pod logs to inspect - which environment variables/settings matter - what Redis/Postgres signals to monitor - what version to upgrade to next Relevant docs: - Prefect docs - Self-hosted Prefect guides - Schedules overview

Yes — you can have two background services pods online at the same time. In Prefect 3.x, that is a supported HA pattern as long as they share the same PostgreSQL and Redis-backed coordination settings. How it works If you run multiple

prefect server services start

processes/pods: - they can all be up at once - Prefect uses Redis + Docket to coordinate periodic background work - for services like scheduling, only one pod should execute a given periodic task per interval - the others are effectively available to take over if one pod dies or loses coordination So this is not “both pods schedule everything independently” if configured correctly. Critical caveat This only works correctly if your background service pods are using a shared Redis docket backend. You should have this configured consistently across the service pods:

bash
PREFECT_SERVER_DOCKET_URL=redis://<redis-host>:6379/<db>

If

PREFECT_SERVER_DOCKET_URL

is left at the default

memory://

, then each pod coordinates only with itself, which can cause duplicate or inconsistent background work. Recommended deployment pattern - API/server pods:

bash
  prefect server start --no-services

- Background service pods:

bash
  prefect server services start

And all of them should point at the same: - Postgres database - Redis instance - Docket URL In your case Given your earlier symptom, running 2 background services pods could improve availability if the issue was one pod dying/stalling. However, if the underlying problem is: - Redis instability - a stuck scheduler state in Redis - a version-specific bug in 3.6.12 then adding a second pod may reduce impact but may not fully eliminate the issue. So my recommendation would be: 1. Yes, you can run 2 background services pods 2. Verify

PREFECT_SERVER_DOCKET_URL

is set to Redis 3. Upgrade from 3.6.12 if possible, since newer 3.x releases include scheduler/background-service reliability improvements Relevant docs: - Scale self-hosted Prefect - Prefect server Helm deployment If you want, I can also help you validate whether your current Kubernetes manifests are actually set up for safe multi-background-service HA.