Thanks for sharing the logs. In Prefect 3.x, this specific error means the runtime could not renew the concurrency lease it was holding. To prevent exceeding your concurrency limit, Prefect cancels the run, which is why you see repeated “Crash detected! Execution was cancelled by the runtime environment.” What it usually means - You’re likely using a concurrency limit (global or deployment-level). When a run starts, it acquires a lease. Prefect must periodically renew that lease; if renewal fails, Prefect terminates the run to avoid over-allocation. - Renewal failures are most commonly caused by: - Network/transient connectivity issues between your worker and the Prefect API - Worker restarts/evictions (K8s pod reschedules, OOM, SIGTERM) - The event loop being blocked for too long (long GC pauses, heavy synchronous CPU work) - Brief Prefect API downtime or reverse proxy timeouts Why it cancels immediately for deployments - For deployment concurrency limits, Prefect runs in “strict” mode by design: if a renewal fails, the flow run is cancelled to keep the limit accurate. Quick triage checklist 1) Does the crash happen ~3–5 minutes (or in multiples of that) after the run starts? - If yes, that strongly indicates a lease renewal failure timing out. 2) Check worker logs at the crash time for: - Network/connect errors, timeouts, 5xx, or “410 Gone” - Pod/node restarts, reschedules, OOMs 3) Verify stable network path to the Prefect API (Cloud or Server). Look for load balancer/proxy idle timeouts or DNS hiccups. 4) Make sure your run isn’t blocking the event loop for long periods. If you have heavy CPU-bound or blocking I/O, push it to threads or processes so the renewal coroutine can run. Mitigations - If you’re using deployment concurrency: focus on stabilizing renewals (network reliability, avoid worker restarts, avoid long event-loop blocks). This mode intentionally cancels on failure. - If you’re using a

concurrency(...)

context in your own code and cancellations are undesirable on transient blips, run it in non-strict mode:

from prefect.concurrency.asyncio import concurrency

  async with concurrency(
      "my-limit",
      occupy=1,
      strict=False  # continue execution if a renewal blips
  ):
      await do_work()

Note: This allows execution to continue if a renewal fails temporarily, which can momentarily exceed the limit. - If you suspect event loop blocking: - Offload blocking work:

import anyio

    async def run_blocking(fn, *args, **kwargs):
        return await anyio.to_thread.run_sync(fn, *args, **kwargs)

- Or ensure CPU-heavy steps run in a separate process pool. - Ensure your workers aren’t being killed or rescheduled mid-run (K8s eviction, HPA scale-down, node rotations). - Upgrade to the latest 3.x release and enable DEBUG logs to surface renewal errors:

PREFECT_LOGGING_LEVEL=DEBUG

Helpful docs - Global Concurrency Limits (concepts) - How to use Global Concurrency and Rate Limits A few questions to pinpoint root cause - Are you using deployment concurrency limits on this deployment, or global/user-defined limits via a

concurrency(...)

context? - Prefect version, worker type (Kubernetes, Docker, process), and where it’s running? - Roughly how long after the run starts does the crash occur? - Do your worker logs show connection errors or worker restarts around that time? If you can share a short snippet of your deployment configuration and any worker log lines near the failure, I can help narrow this down further and recommend a targeted fix.