<@ULVA73B9P> There are ghosts runs that are runnin...
# marvin-ai
m
@Marvin There are ghosts runs that are running at random times for a deployment we have. What could be causing this?
m
thought for 934.9 seconds
Short answer: “Ghost” runs almost always come from a trigger you’re not expecting (a schedule, an automation, or an API call) or from backlogged scheduled runs being picked up later by a worker. Here’s how to track it down and fix it. Assuming you’re on Prefect 3.x: What typically causes unexpected runs - A deployment schedule you forgot about or misconfigured - Multiple schedules attached to the same deployment - Timezone mismatch (default is UTC if not set) - Interval schedules without a clear anchor date - Overlapping rules (cron + rrule) - Automations that create flow runs - e.g., “on failure of X, run Y” or webhook/event-driven automations - Programmatic/API triggers - Other code calling
create_flow_run_from_deployment
at runtime (or CI/CD scripts) - Backlog being picked up after downtime or low capacity - Runs were scheduled earlier but only start when a worker comes online or capacity frees up - Another deployment or subflow calling this deployment unexpectedly How to diagnose quickly 1) Inspect a “ghost” run’s source - In the UI: open the flow run details and check “Triggered by” and compare “Scheduled for” vs “Start time”. - CLI:
Copy code
prefect flow-run inspect <run_id>
Look for who/what created it and when it was scheduled versus when it actually started. 2) List and verify schedules on the deployment - See if there are multiple or overlapping schedules:
Copy code
prefect deployment schedule ls "<flow_name>/<deployment_name>"
- Inspect full deployment details:
Copy code
prefect deployment inspect "<flow_name>/<deployment_name>"
- If you suspect the schedule, pause it to confirm:
Copy code
prefect deployment schedule pause "<flow_name>/<deployment_name>"
If runs stop, the schedule is the source. You can also remove/clear schedules:
Copy code
prefect deployment schedule clear "<flow_name>/<deployment_name>"
3) Check Automations - In the UI, open Automations and look for any with an action like “Create flow run” targeting this deployment/flow. Review recent automation runs to see if they align with your “ghost” run times. 4) Check for API/Code triggers - Search your codebase/other flows for calls to create this deployment’s runs, e.g.
create_flow_run_from_deployment
or orchestration flows that trigger this deployment. 5) Validate worker/backlog behavior - If “Scheduled for” is earlier than “Start time,” it’s not a ghost—it’s a scheduled run being picked up later. Check worker logs and work pool capacity/concurrency; runs will start as capacity frees up. 6) Timezone and interval alignment - For cron/rrule schedules, confirm the timezone is what you expect. - For interval schedules, set an anchor date if you need runs aligned to a specific boundary to avoid odd offsets. Helpful docs - Schedules: Prefect 3.x schedules If you can share: - One or two example run IDs - The deployment name - Whether you’re on Prefect Cloud or OSS …I can help pinpoint the exact trigger (schedule, automation, or API) and suggest the precise fix.