Short version: your flow is running under a concurrency limit whose lease could not be renewed. When the server replied 410 Gone (“Lease not found — it may have expired or been revoked”), Prefect treated the run as no longer holding a valid concurrency slot and cancelled the run to prevent overrunning the limit. That cancellation is what surfaces as “Crash detected! Execution was cancelled by the runtime environment.” What’s happening under the hood (Prefect 3.x): - When a flow acquires a concurrency slot, Prefect starts a background renewal loop. It renews every ~75% of the lease duration (default lease is 300s, renew ~225s). - If renewal fails repeatedly or the server no longer recognizes the lease (expired or revoked/reset), the next renew call returns 410 Gone with “Lease not found…”. - For deployment-level concurrency limits, Prefect runs in strict mode: losing the lease triggers an immediate cancellation of the run, which yields the “Crash detected! Execution was cancelled by the runtime environment.” log. - Common triggers: - The lease actually expired because renewals were missed (network hiccups, API unreachable, worker/pod paused/restarted, heavy CPU pressure). - The limit was reset/deleted in the UI or via API, which revokes existing leases immediately. - Long blocking operations (esp. in async flows) that prevent the renewal task from running in time. Why your dbt runs are crashing mid-execution - The 410 indicates the server no longer had your lease at renew time (expired or revoked). In strict mode (used for deployment concurrency), Prefect cancels the run to avoid violating the limit; that’s the “Crash detected…” you’re seeing. How to mitigate - Operational - Check for worker/pod restarts or node pauses around failure times (Kubernetes evictions, autoscaler events, spot interruptions, etc.). - Ensure stable egress to api.prefect.cloud (no intermittent DNS/firewall/proxy issues). Monitor for

HTTPStatusError 410

and connection errors in worker logs. - Avoid deleting/resetting the relevant concurrency limit while runs are active. - If using async flows, avoid long blocking calls in the event loop; offload to threads/processes or use non-blocking helpers. - For Kubernetes, consider PodDisruptionBudgets, graceful termination, and resource requests that keep the runtime from being paused/starved. - Keep an eye on Cloud status for incidents: Prefect Cloud status. - Tuning - For user-managed concurrency contexts (not deployment concurrency), you can increase the lease duration so there’s more tolerance for transient blips:

from prefect.concurrency.sync import concurrency

    with concurrency("dbt-prod", lease_duration=900, strict=True):
        run_dbt_build()  # your dbt step

-

lease_duration

defaults to 300s; try 600–1800s for long dbt operations. - If you prefer the run to continue even if the lease can’t be renewed (accepting the risk of temporarily exceeding the limit), set

strict=False

. - Note: Deployment-level concurrency always operates in strict mode and uses a fixed lease strategy; if you need custom lease durations for long-running dbt steps, consider moving the constraint to an explicit

concurrency(...)

block around the critical section instead of using deployment concurrency. What I’d check next - Are you using deployment-level concurrency limits in the Prefect Cloud UI for these flows? If yes, the strict cancellation explains the behavior. - Did anyone reset/delete the relevant concurrency limit around the failure timestamps? - What worker type/environment are you using (Kubernetes, Docker, VM)? Any restarts or pauses recorded at those times? - Rough duration of your dbt runs, and how frequently the failures occur. - A couple of failed flow run IDs and timestamps so we can correlate with logs.

If you share those details, I can help pinpoint whether this is expiry from missed renewals, explicit revocation, or something environmental, and suggest the most appropriate fix for your setup. References - Concurrency limits overview: Prefect docs: Concurrency limits - Flow/task states and crash vs cancelled: Prefect docs: States

Thanks for the detailed context — that helps a lot. I’ll answer each question directly and then suggest a path that meets your “one run per region, no fragile crashes” requirement. First, an important correction (Prefect 3.x): deployment-level concurrency leases are not short (e.g., “30s, renew every ~10s”). In 3.x they use a 300s (5 min) lease by default and renew every 75% of that (~225s). During a 30-minute run you’ll see ~8 renewals, not ~180. Each renewal has built-in retries (up to 3) before failing. 1) What could cause a 410 Gone during lease renewal? - What 410 means: The server no longer recognizes the lease id you’re trying to renew — “Lease not found — it may have expired or been revoked.” - Common causes - Lease expired because the runtime missed renewals (transient network loss, API temporarily unreachable, process paused, pod/VM restart, heavy CPU pressure blocking the renewal task) - Lease was revoked because the concurrency limit was reset or deleted in the UI/API - Less common: Cloud incident on the concurrency lease endpoint - This is generally not a configuration mistake on your end. It’s almost always environmental (network/process) or an intentional reset/delete of the limit. - References - Concurrency limits overview: docs: Concurrency limits - Renew endpoint: API: renew concurrency lease - Cloud status: Prefect Cloud status 2) Is deployment-level concurrency appropriate for long-running flows? - Yes. With the correct 3.x timing (300s lease, renew ~every 225s), long runs are common and supported. - Important behavior: deployment-level concurrency runs in strict mode — if the lease cannot be renewed after retry, the run is cancelled to prevent overrunning the limit. That’s where “Crash detected! Execution was cancelled by the runtime environment.” comes from. - So it’s appropriate, but your environment must be stable enough to avoid missed renewals. 3) Best practices to prevent concurrent runs without fragile lease renewals You have a few solid options. Pick based on how strongly you want enforcement vs. resiliency: A) Keep deployment-level concurrency and harden the environment - Ensure worker/pod stability (no frequent evictions/restarts), stable egress to Prefect Cloud API, and avoid resetting/deleting the relevant concurrency limits while runs are active. - Add an Automation to retry on Crashed/Cancelled so transient 410s recover quickly. - Automations: docs: Automations B) Use flow-level concurrency context with tolerant behavior (most resilient to transient blips) - Remove deployment-level concurrency, then wrap your flow’s critical section with a concurrency context using a longer lease_duration and strict=False. If a renewal fails, the run continues (concurrency enforcement may temporarily be lost but will not crash). - Example:

from prefect import flow
  from prefect.concurrency.sync import concurrency

  @flow
  def dbt_flow_ca():
      # Create a concurrency limit named "dbt-internal-ca" (limit=1) in the UI first
      with concurrency("dbt-internal-ca", occupy=1, lease_duration=1800, strict=False):
          run_dbt_build()  # your dbt invocation

- Trade-off: stronger resiliency, but during a transient failure the lease may lapse and a second run could start if something else acquires the slot. In practice this is rare if outages are brief and lease_duration is generous. C) Gate starts with per-region work pools (no mid-run lease to renew) - Create a work pool per region and set its pool-level concurrency to 1; point the region’s deployments to the region’s pool. - This enforces “only one run can start per region” at the scheduling/worker layer and does not rely on an in-run lease — so you won’t see mid-run cancellations from lease renewal.

- Keep in mind this constrains all deployments in that pool; that’s generally what you want per region. 4) Can we configure lease duration or renewal behavior? - Deployment-level concurrency - The in-run lease renewal cadence is fixed in 3.x (300s lease, renew at ~225s). There isn’t a public setting today to change the renewal TTL or retry policy for deployment concurrency. - You can configure an initial “grace period” on a deployment via concurrency options, but that affects the initial server-held lease before the client starts renewing — it does not change the ongoing renewal cadence. - Deployments + concurrency docs: docs: Deployment concurrency - Flow-level concurrency contexts - Fully configurable: set

lease_duration

(e.g., 900–1800s for long runs) and

strict

to control behavior on renewal failure. - Concurrency concepts: docs: Concurrency limits Recommended path for your requirement - If you want zero tolerance for overlap and are okay retrying on rare blips: - Keep deployment-level concurrency per region (what you have). - Add an Automation to retry on Crashed with a small backoff (e.g., up to 1–2 retries). - Harden the runtime: check for pod/VM restarts, ensure stable egress to Cloud, avoid limit resets. - If you want “never crash mid-run” and can accept a tiny risk of temporary over-enforcement loss: - Remove deployment-level concurrency and guard runs with a flow-level concurrency context per region, using a long

lease_duration

(e.g., 1800s) and

strict=False

as shown above. - If you prefer to enforce “one-at-a-time per region” without any in-run lease at all: - Create one managed work pool per region and set each pool’s concurrency to 1, then point each region’s deployment at its region’s pool. If you can share a couple of failing flow run IDs and timestamps, I can help you correlate with worker restarts or network logs to pinpoint whether these were expiry due to missed renewals vs. a limit reset. Useful links - Concurrency limits: docs: Concurrency limits - Deployment concurrency: docs: Deployment concurrency - Lease renew endpoint: API: renew concurrency lease - Automations (for retries): docs: Automations - Cloud status: Prefect Cloud status