Thanks for the report — that log is coming from Prefect’s concurrency lease maintainer. When a deployment (or code-level concurrency context) holds a concurrency slot, the client periodically renews a lease with the server. If renewal fails, the engine assumes the slot is no longer reserved and cancels the run to avoid over-allocation. Common triggers after an upgrade - Lease expired before renewal could complete (default lease is 300s, renewal at ~75% of that) - Temporary network or API timeouts between your worker and the server (especially relevant for Vertex AI work pools reaching your Prefect server) - Server restart or loss of in-memory lease storage (the default memory backend drops leases on restart) - Lease was revoked/404/410 from the server (e.g., race around flow completion; this was improved in 3.6.5) Related reports and context - Concurrency lease renewal failures/crash: #19068, #18893 - Infra/pods keep running after crash: #18839 - Max lease duration (API caps at 1 day): #19218 What changed in 3.6.5 - Flows that have already reached a final state should no longer be marked crashed if the server has already cleaned up their lease (a guard was added so we only cancel if the run is still active). Quick mitigations to try - Increase the lease duration to give more margin for renewal - Server setting (affects deployment concurrency leases): - Env var:

PREFECT_SERVER_CONCURRENCY__INITIAL_DEPLOYMENT_LEASE_DURATION=600

(10m) - Or in prefect.yaml:

prefect:
        server:
          concurrency:
            initial_deployment_lease_duration: 600.0

- Code-level concurrency (if you’re using

prefect.concurrency.*

in your flow code), pass a longer `lease_duration`:

from prefect.concurrency.asyncio import concurrency

    async with concurrency(["my-limit"], occupy=1, lease_duration=600):
        ...

- Make lease storage persistent so server restarts don’t drop leases - Memory is the default; consider filesystem or Redis: - Filesystem:

PREFECT_SERVER_CONCURRENCY__LEASE_STORAGE=prefect.server.concurrency.lease_storage.filesystem

- Redis (requires prefect-redis):

PREFECT_SERVER_CONCURRENCY__LEASE_STORAGE=prefect_redis.lease_storage.RedisLeaseStorage

- Verify connectivity from Vertex AI workers to your Prefect server (DNS, TLS, firewalls, proxies). Even brief blips during renewal can cause this. - If this is not a deployment-level limit but a code-level limit that’s non-critical, consider continuing on renewal failure:

from prefect.concurrency.sync import concurrency

  with concurrency(["non-critical"], occupy=1, lease_duration=600, strict=False):
      ...

Note: deployment-level concurrency in the engine is strict by design to prevent over-allocation. A few clarifying questions to help pinpoint it - You wrote “but the …” — could you finish that thought? For example, is the flow marked Crashed even though all tasks finished, or does your Vertex AI job keep running after Prefect marks the run Crashed? - Are you using deployment-level concurrency limits, code-level concurrency contexts, or both? - Roughly how long do these runs take vs the default 5-minute lease? Does increasing the lease help? - Did your Prefect server restart around the time of these failures? Which lease storage backend are you using? - Since you’re on Vertex AI work pools: is outbound HTTPS from the Vertex jobs to your Prefect server ever blocked or proxied? If you can share a run id and a few minutes of logs around the error from both the flow run and the server, I can help zero in on whether this is lease expiration, connectivity, or storage-related.

Got it — thanks for clarifying. Since your runs are ~6–7 minutes, you’re likely hitting the first renewal around 3.75 minutes (the default lease is 300s and the client renews at ~75% of that). If that renewal fails even briefly (network hiccup from Cloud Run to your Prefect server, server restart, or in-memory lease loss), the engine cancels the run to avoid over-allocation — which matches what you’re seeing. What I recommend - Bump the deployment lease duration so renewal happens after your job finishes - On your Prefect server, set:

PREFECT_SERVER_CONCURRENCY__INITIAL_DEPLOYMENT_LEASE_DURATION=900

or in prefect.yaml:

prefect:
      server:
        concurrency:
          initial_deployment_lease_duration: 900.0

900–1200 seconds is a good starting point for 6–7 minute runs. This pushes the first renewal to 11.25–15 minutes, so it won’t happen mid-run. Note: deployment lease duration has an upper cap (server-enforced); 15–20 minutes is well within limits. - Make lease storage persistent (avoid memory, which loses leases on server restart) - Filesystem:

PREFECT_SERVER_CONCURRENCY__LEASE_STORAGE=prefect.server.concurrency.lease_storage.filesystem

- Redis (requires prefect-redis):

PREFECT_SERVER_CONCURRENCY__LEASE_STORAGE=prefect_redis.lease_storage.RedisLeaseStorage

- Double-check Cloud Run → Prefect server connectivity - Any brief egress blip or TLS/DNS issue during renewal will cancel the run. If you use a VPC connector/proxy, verify it’s stable and that outbound HTTPS to the Prefect API is consistently allowed. - Quick health check from the worker container during a run:

curl -s -o /dev/null -w "%{http_code}\n" $PREFECT_API_URL/api/health

- If you don’t need deployment concurrency limits, temporarily remove them to confirm the diagnosis (the error stops when no lease is maintained). Relevant reports for context - Lease renewal failures causing crashes: #19068, #18893 - Infra continuing after crash: #18839 - Max lease duration discussion: #19218 A couple quick questions to confirm - Do you have a deployment-level concurrency limit set on these deployments? - Do the crashes happen ~3.5–4 minutes into the run? If yes, that aligns with the renewal timing. - Are you running OSS server or Prefect Cloud? If OSS: which lease storage backend are you using? If you can share one flow run ID and a snippet of server logs around the time of the failure, I can help confirm whether this is an expiry, a 404/410 from the server, or a network timeout.

Thanks — that 410 Gone on renew nails it. With deployment-level concurrency and the OSS server using the default in-memory lease storage, a 410 during renewal means the server no longer has that lease record. On Cloud Run, that typically happens when: - The server scaled to zero or restarted (memory wiped), or - Requests are hitting different Cloud Run instances (each instance has its own in-memory store), so a renewal lands on an instance that never saw the original lease. Because deployment-level leases are “strict”, a failed renewal cancels the run — which explains your crash ~5 minutes in. What to change 1) Switch concurrency lease storage to a shared, persistent backend (recommended) - Since you already have Redis, use the Redis lease storage so renewals are consistent across instances: - Build your server image with prefect-redis installed, then set:

PREFECT_SERVER_CONCURRENCY__LEASE_STORAGE=prefect_redis.lease_storage.RedisLeaseStorage

- Configure it to point at your Redis endpoint (same one you use for messaging is fine). - This removes the per-instance memory split and survives restarts. 2) Increase the deployment lease duration to push renewals after your runs finish - Your runs are 6–7 minutes; set 900–1200 seconds so the first renewal happens well after completion:

PREFECT_SERVER_CONCURRENCY__INITIAL_DEPLOYMENT_LEASE_DURATION=900

or in prefect.yaml:

prefect:
    server:
      concurrency:
        initial_deployment_lease_duration: 900.0

3) Optional Cloud Run guardrails (short-term mitigation if you can’t change storage yet) - Keep the server from scaling away or splitting across instances: - Set min instances = 1 (avoid scale-to-zero) - Set max instances = 1 (avoid cross-instance routing) - This is a stop-gap; Redis storage is the robust fix. Why this matches your symptoms - Default lease is 300s and renewals happen periodically; if the renewal hits an instance without the lease (or after a restart), the server returns 410 and the engine cancels the run. - With 6–7 minute runs, you’re seeing the renewal window mid-run; hence the crash around ~5 minutes. References - Concurrency lease renewal failures and crashes: #19068, #18893 - Infra left running after crash: #18839 - Max lease duration discussion: #19218 If you’d like, I can share a minimal Dockerfile snippet for the OSS server that adds prefect-redis and the env vars needed for Redis lease storage. Also, can you confirm your Cloud Run min/max instance settings for the server? That’ll help validate whether instance turnover played a role.

Great question — here’s exactly how to point your Prefect OSS server at Redis for concurrency lease storage. Prereqs - Your server image needs the Redis integration. Easiest is to install the extra:

pip install "prefect[redis]"

or bake it into your server image:

RUN pip install "prefect[redis]"

Configure the server to use Redis leases - Set the lease storage module to the Redis backend:

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

- Point it at your Redis endpoint (these are the same settings used for Prefect’s Redis messaging, so you can reuse them):

PREFECT_REDIS_MESSAGING_HOST=<your-redis-host>
  PREFECT_REDIS_MESSAGING_PORT=6379
  PREFECT_REDIS_MESSAGING_DB=0
  # If auth/TLS:
  PREFECT_REDIS_MESSAGING_USERNAME=default
  PREFECT_REDIS_MESSAGING_PASSWORD=<your-password>
  PREFECT_REDIS_MESSAGING_SSL=false

- Since your runs are ~6–7 minutes, also bump the deployment lease duration to avoid mid-run renewals:

PREFECT_SERVER_CONCURRENCY__INITIAL_DEPLOYMENT_LEASE_DURATION=900

Cloud Run example (server) - Add these as environment variables on your Prefect server service (API/background). If you run separate services, set them on both. - Short-term hardening (optional if you can’t switch storage immediately): - Min instances = 1 (avoid scale-to-zero) - Max instances = 1 (avoid cross-instance lease loss) prefect.toml alternative - If you prefer config files instead of env vars:

[server.concurrency]
  lease_storage = "prefect_redis.lease_storage"
  initial_deployment_lease_duration = 900.0

  [redis.messaging]
  host = "your-redis-host"
  port = 6379
  db = 0
  username = "default"
  password = "your-password"
  ssl = false

Verify it’s working - After deploying, check that settings are loaded:

prefect config view | grep -E "CONCURRENCY|REDIS_MESSAGING"

- Run a test deployment with a concurrency limit; you should no longer see 410 Gone on renew, and runs should not crash ~5 minutes in. If you want, share how you currently deploy the server to Cloud Run (Dockerfile or gcloud command), and I can tailor exact snippets. For reference on the lease-renewal crash you were hitting: #19068, #18893, #18839.

Thanks for catching that — my first value included a class name, which is not what the setting expects. - Correct env var name:

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

- Correct value for Redis:

prefect_redis.lease_storage

(module path, not class path) Why: the server loads this with

importlib.import_module(<value>)

and then instantiates

ConcurrencyLeaseStorage

from that module. So you must provide the module, not

...RedisLeaseStorage

. Example

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_REDIS_MESSAGING_HOST=<your-redis-host>
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# auth/TLS if needed:
# PREFECT_REDIS_MESSAGING_USERNAME=default
# PREFECT_REDIS_MESSAGING_PASSWORD=...
# PREFECT_REDIS_MESSAGING_SSL=true

And ensure the server image has the integration installed:

pip install "prefect[redis]"