Thanks for the log snippet — that warning means your API’s scheduling loop is running much slower than intended, and it often shows up alongside “Internal Server Error” when the DB/API is under load. Separately, since you upgraded from Prefect 2.x to 3.x, the “agent” command is gone in 3.x — flows will sit in Scheduled → Late/Pending unless a 3.x worker is polling a work pool that your deployments target. Quick path to green 1) Start a 3.x worker (replaces agents) - Create or confirm a work pool and start a worker on it:

prefect work-pool ls
prefect work-pool create "default" --type process   # if you need one
prefect worker start --pool default                 # add -q default if you use queues

- If you previously ran

prefect agent start -q default

, the equivalent is:

prefect worker start --pool default -q default

2) Make sure your deployments point at that pool - Inspect a deployment and confirm the work pool name:

prefect deployment ls
prefect deployment inspect <flow-name>/<deployment-name>

- If it doesn’t point at your pool, re-deploy using the 3.x pattern: - Python:

from prefect import flow

@flow
def my_flow(): ...

my_flow.deploy(
  name="my-deployment",
  work_pool_name="default",
)

- Or via CLI (interactive):

prefect deploy

3) Retry a run and check the worker logs - You should see the worker polling and picking up the run within seconds. If runs are still going Late: - Lower polling intervals temporarily and retry:

export PREFECT_WORKER_QUERY_SECONDS=10
export PREFECT_WORKER_PREFETCH_SECONDS=20
prefect worker start --pool default

- Check for deployment concurrency limits or filters that could be blocking execution:

prefect deployment inspect <flow-name>/<deployment-name> | grep -i concurrency

About the “RecentDeploymentsScheduler took 27s” and Internal Server Error - What it means: The server’s fast scheduler loop took longer to finish than its interval. Common causes: - Database slowness or low resources (CPU, memory, disk IOPS) - Large backlog of deployments/runs to schedule - Network latency to the DB or API - What to do: - Ensure you’re on Postgres (SQLite will struggle under load) - Check DB health (connections, CPU, slow queries) and right-size resources - If needed, relax the scheduler loop frequency to reduce churn (use

prefect config view --show-defaults

to find the exact setting names under server.services.scheduler, then set them via env and restart your server) - Reduce batch sizes if you have memory/IO constraints (same config area) - Once the DB/API are responsive again, those Internal Server Errors typically stop. Useful commands to verify health - Confirm worker visibility/polling:

PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool default

- Check deployments and runs:

prefect deployment ls
prefect flow-run ls --state Late -a | head -20

- Inspect current settings and find tunables:

prefect config view --show-defaults

Docs and guides - Agent → Worker migration: Upgrade agents to workers - Worker health checks (recommended for production): Worker health checks A couple of quick clarifications to help me tailor next steps: - Are you running self-hosted Prefect Server with Postgres, or Prefect Cloud? - Do you currently have any 3.x workers running? If so, what work pool and (optionally) queue are they polling? - Approximately how many deployments/runs are you scheduling, and what DB are you using?

Got it — thanks for clarifying. The symptoms you’re seeing (runs stuck Late/Pending + “Internal Server Error” + the RecentDeploymentsScheduler warning) point to two separate things: - Nothing is actively picking up your scheduled runs (agent in 2.x vs worker in 3.x) - Your API/server is running slowly (DB or resource bottleneck), which can surface as “Internal Server Error” Before we prescribe commands, can you share: - The output of:

prefect version

- Are you on Prefect Cloud or self-hosted Prefect Server? - If self-hosted: which database (Postgres vs SQLite) and roughly what instance size? Once we confirm the version, here’s how to fix quickly: If you are on Prefect 3.x - Make sure a worker is running and polling the correct work pool/queue:

prefect work-pool ls
prefect worker start --pool <your-pool>          # add -q <your-queue> if you use queues

- Ensure your deployments point at that work pool (and queue, if used). If they don’t, re-deploy with the pool set and retry a run. - If runs still go Late, temporarily increase polling:

export PREFECT_WORKER_QUERY_SECONDS=10
export PREFECT_WORKER_PREFETCH_SECONDS=20
prefect worker start --pool <your-pool>

- The scheduler warning means the API/DB is slow. If you’re self-hosting: - Prefer Postgres over SQLite - Check CPU/memory/disk IOPS on your DB and API - If needed, relax the recent-scheduler loop to reduce pressure (set env and restart server):

PREFECT_SERVER_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=15
PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS=120

- For context on workers and healthchecks: - Upgrade agents to workers: Upgrade agents to workers - Worker health checks: Worker health checks If you are on Prefect 2.x - Confirm an agent is running and polling the exact work queue your deployments target (e.g.,

default

). If no agent is up, scheduled runs will sit in Scheduled → Late/Pending. - Check that the work queue is not paused and that any concurrency limits/filters aren’t blocking execution. - The “RecentDeploymentsScheduler took X seconds” warning still indicates API/DB slowness: - Use Postgres (SQLite will struggle under load) - Right-size resources for the API and DB - Inspect API logs around “Internal Server Error” for stack traces and DB timeouts Helpful checks (either version) - See what’s Late:

prefect flow-run ls --state Late -a | head -20

- Inspect a deployment and confirm the target pool/queue (3.x) or queue (2.x):

prefect deployment inspect <flow-name>/<deployment-name>

- View current settings to discover tunables:

prefect config view --show-defaults

If you can paste: -

prefect version

- Whether you’re on Cloud or self-hosted (and DB details) - The exact command you use to run your agent/worker (if any) - A snippet of the API/server stack trace around “Internal Server Error” …I can give you precise next steps and the exact commands to run.

Thanks for confirming — you’re on Prefect 2.20.16 (self‑hosted, Postgres). The two issues you’re seeing are usually independent: 1) No one is picking up your scheduled runs → they sit in Scheduled and then become Late/Pending 2) Your API is slow under load → “Internal Server Error” and the “RecentDeploymentsScheduler took X seconds” warning Do these to get unstuck quickly 1) Make sure a 2.x Agent is running and polling the exact work queue your deployments target - On the machine that should execute, set the API URL (if not already):

export PREFECT_API_URL="http://<your-server-host>:4200/api"

- Start the agent on the correct queue (example: default):

prefect agent start -q default

- In the agent logs, look for lines like “Polling work queue 'default'”. If you don’t see that, it’s not connected to the right place. 2) Confirm your deployments target that same work queue and it isn’t paused - In the UI, open a deployment and confirm its

work_queue_name

matches the queue your agent polls. - Also check the Work Queue in the UI to ensure it’s not paused and that any filters/tags match the runs you expect. 3) If runs still go Late, temporarily make pickup more forgiving while you triage - Increase the “Late” threshold (reduces Late noise while you fix the root cause):

export PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS=60

- Then restart your API/server so the setting takes effect. Fix the scheduler/API slowness (the warning and 500s) The warning means your scheduling loop is taking longer than its configured interval — typically due to DB or resource pressure. Actions: - Check and right-size Postgres and API resources - Ensure Postgres CPU/memory/IO aren’t saturated; increase instance size or IOPS if needed - Confirm connections aren’t maxed out; if you use a pooler (e.g., pgbouncer), check its limits - Make sure regular VACUUM/ANALYZE is running - Relax the scheduler loop and reduce batch sizes (to lower burst pressure) - Set these on the API/server container/environment, then restart the server:

# Slow the loops a bit
export PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=15
export PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS=120

# Process smaller batches if you have memory/IO constraints
export PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
export PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250

- Capture a stack trace for “Internal Server Error” - The API logs around the 500 should show a traceback (often a DB timeout). If you can paste a snippet, I can help pinpoint the exact cause. Optional: consider 2.x Workers (instead of agents) - If you prefer Work Pools/Workers in 2.20.16, you can:

prefect work-pool create "process-pool" --type process
prefect worker start --pool process-pool

- Then update deployments to target that work pool. But if you already have agents, it’s fine to stick with them. If you can share: - The command you’re using to start the agent (and its logs on startup) - The deployment’s

work_queue_name

- Whether the queue is paused or has filters/tags - A snippet of the API log around an “Internal Server Error” …I’ll give you exact next steps and any targeted config changes.