Short answer: “Late” just means “didn’t start within ~15s of its scheduled time.” If you scheduled 90 at once but the system only had capacity for a few, the rest will flip to Late quickly. The surprising bit in your case is the capacity looks like ~3, not 50 — that’s the part to track down. What to check - Work queue concurrency - If any work queue in your Cloud Run v2 pool has a concurrency limit of 3, the server will only hand out 3 runs at a time, regardless of the pool’s 50. - Check with:

prefect work-pool ls -v
    prefect work-queue ls -p <your-pool> -v

- Deployment-level concurrency - If your deployment has a concurrency limit, runs beyond that will stay Scheduled and quickly become Late. - Check with:

prefect deployment inspect "<flow-name>/<deployment-name>"

- Worker local limit - If your worker was started with a limit (e.g., 3), it will submit only that many concurrent runs to Cloud Run even if the pool allows 50. - How did you start your Cloud Run v2 worker? If you used the CLI, look for a flag like:

prefect worker start -p <your-pool> --limit 3 ...

- If you’re running the worker in a container (e.g., Cloud Run service or VM), confirm the start command/envs don’t set a low limit. - Cloud Run Jobs constraints - If Cloud Run is rate-limiting or blocking overlapping executions (per job name), you’ll effectively see much lower concurrency. Look for 409/429/5xx responses in worker logs when it tries to create executions. - Also check org/project quotas for Cloud Run Jobs concurrent executions in your region. Why so many Late states - In Prefect 3.x, a Scheduled run turns Late if it hasn’t started roughly 15 seconds after its scheduled_time. When you launch a big burst, anything that isn’t picked up immediately will flip to Late; they will still run when capacity opens. This is expected labeling, not a failure state. About “Concurrency lease renewal failed” after 3.6.5 - That message indicates the engine’s lease maintainer could not renew a concurrency lease and cancels the run to avoid over-allocation. If you’re seeing these crashes after upgrading, it’s often due to network/auth interruptions during long-running runs or an issue with the lease configuration. - Related reports for context: - Issue: Concurrency lease renewal failed - slots no longer reserved - Issue: Infra stays running after Crash due to failed lease renewal - Issue: Lease duration cap feedback - If you have logs around the time of the crash (worker + flow), I can help pinpoint whether this is a transient API/egress issue or something else. Next steps I suggest 1) Share the output of:

prefect work-pool ls -v
   prefect work-queue ls -p <your-pool> -v
   prefect deployment inspect "<flow-name>/<deployment-name>"

plus how you start your worker(s) for this Cloud Run v2 pool (command or container args). 2) If possible, paste any worker log lines around when you launched the 90-run burst — especially lines about capacity, queue limits, or Cloud Run API errors. 3) Confirm whether all 90 runs were for the same deployment; if yes, note that some infrastructures (e.g., if a single Cloud Run Job name is reused) can constrain overlapping executions. Once we see those, we can identify the exact bottleneck (queue limit, deployment limit, worker limit, or Cloud Run constraints) and get you closer to the expected ~50 concurrent.