<@ULVA73B9P> we have a deployment with a cron defi...
# ask-marvin
j
@Marvin we have a deployment with a cron defined to run every 6 hours, but we are seeing duplicates flow runs each time, with a difference of one minute or in the same minute. What could be the cause?
m
thought for 475.4 seconds
Short answer: this almost always comes from two things creating runs instead of one. Most common causes to check: - Multiple schedules on the same deployment (e.g., a cron schedule added twice by repeated deploys) - An automation that also “runs a deployment” at the same cadence as the schedule - A second deployment of the same flow with its own schedule - Timezone mismatch (two schedules with different timezones) — less common, but can look like :00 and :01 duplicates - Self‑hosted only: multiple API scheduler instances running (not applicable if you’re on Prefect Cloud) Notes: - Work pools (including Vertex AI) do not create runs; they only pick them up. Duplicates are created by the scheduler or automations. How to confirm quickly 1) List the deployment’s schedules
Copy code
prefect deployment schedule ls "your-flow-name/your-deployment-name"
If you see more than one schedule, that’s the cause. Remove extras:
Copy code
prefect deployment schedule delete "your-flow-name/your-deployment-name" <SCHEDULE_ID> -y
Or clear all and re-add just one:
Copy code
prefect deployment schedule clear "your-flow-name/your-deployment-name" -y
prefect deployment schedule create "your-flow-name/your-deployment-name" --cron "0 */6 * * *" --timezone "UTC"
Docs: Schedules 2) Check for automations that also run this deployment
Copy code
prefect automation ls
In the UI, look for automations with an action “Run deployment …” pointing at the same deployment. Disable or adjust them if they mirror your schedule. Docs: Automations and event triggers 3) Verify there isn’t a second deployment of the same flow with its own schedule
Copy code
prefect deployment ls
Look for another deployment targeting the same flow that’s also scheduled. 4) If you’re self-hosting Prefect Server Ensure only one scheduler service is active. The scheduler is an API service; running multiple API instances with the scheduler enabled can create duplicate runs. The environment setting to control this is typically managed at server start; if you suspect this, let me know how you’re running the API and I can advise the exact toggle for your setup. A couple of tips - In the Flow Run details (UI), check the event that created the run: it will say whether it was created by a schedule (with a schedule id) or by an automation. That usually pinpoints the duplicator immediately. - If you’ve been running CI that re-runs
prefect deploy
, it can accidentally accumulate multiple schedules on the same deployment unless you clear or update them deliberately. If you can share: - Cloud vs self-hosted - The deployment name, and whether both duplicates show the same deployment in the UI - Output of
prefect deployment schedule ls "flow/deployment"
…I can pinpoint which of the above is happening and give the exact fix.
j
@Marvin I'm running Prefect self-hosted. Basically I have two services, one that runs the API only with this command
"prefect", "server", "start", "--no-services"
, and another service that run only the prefect services with this command
"prefect", "server", "services", "start"
, both with multiple instances. The API should be able to run multiple instsances, but maybe the services are not supposed to?
m
thought for 167.8 seconds
Yes — that setup will cause duplicates. What’s happening - Each instance of
prefect server services start
runs the Scheduler service. In self‑hosted Prefect (OSS), the Scheduler does not use leader election, so multiple schedulers will independently create the same scheduled flow runs. - The “off by ~1 minute” you’re seeing lines up with the scheduler’s default loop interval (60s). Two schedulers race to “top up” the future runs for the same deployment, so you get duplicates at the same or adjacent minute. Recommended pattern - Scale API horizontally: multiple instances of
prefect server start --no-services
behind a load balancer is fine. - Run a single instance of the services process: exactly one
prefect server services start
for the whole deployment. - If you must run more than one copy of the services pod/process for failover, disable the Scheduler (and Automations/Triggers) on all but one via env vars so only a single instance actually schedules. How to fix quickly - Keep one replica of the services process, or disable the scheduler in the extra replicas:
Copy code
# Disable the scheduler in this instance
PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED=false

# (Optional) If you use Automations, also ensure only one instance runs triggers:
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false
- Then restart those extra services processes. Leave one services instance with scheduler enabled. Why this matches your symptoms - The scheduler loop runs every 60s by default: -
loop_seconds=60
in the server service settings. - With two schedulers, both will attempt to create the next run for your “every 6 hours” cron, yielding two runs at the same or adjacent minute. Docs for reference - Schedules: Schedules - Self-hosted/Server overview: Self-hosted Prefect - Settings reference (for env vars): Settings reference If you’d like, I can help sanity-check by: - Verifying you only have one services instance with the scheduler enabled - Confirming in the UI that the duplicate runs both show “created by schedule” around the same minute - Cleaning up any extra runs and ensuring only one future run is created per window