Short answer: yes—upgrading from 3.4.20 to 3.6.5 is generally safe. There are no breaking changes to flows, deployments, work pools, or the Vertex AI worker path. The main thing to watch is the Python version requirement bump in 3.5.0. What changed that could affect you - Python 3.10+ required (3.5.0). If any agents/workers are still on Python 3.9, upgrade them first. - Background task system was upgraded (3.6.0) to improve reliability of things like deployment pull steps; this is transparent to users. - Concurrency limits got more robust (lease-based slots, caching). If you use Global Concurrency Limits, the behavior is more reliable; no action needed. - Workers/work pools: several additive improvements; nothing breaking. Dot-delimited job_variables syntax was added (3.6.5) but is optional. - No core breaking changes for Vertex AI work pools. If you use GCS result storage, consider updating

prefect-gcp

to the latest for bugfixes. Recommended upgrade checklist - Verify Python >= 3.10 on any machines running agents/workers. - If self-hosting Prefect Server, back up your database. Migrations will run automatically on startup. - Update packages: - Upgrade Prefect to 3.6.5. - If you use GCP/Vertex AI, also upgrade

prefect-gcp

to the latest. - Smoke test: - Start a worker for your Vertex AI work pool and run one deployment end-to-end. - If you use concurrency limits, trigger a run that should be limited and confirm behavior. - If you use event-driven triggers/automations, trigger one and confirm it fires. Known safe areas for your setup - Vertex AI workers: no breaking API changes in this range. - Docker image architecture: nothing in 3.6.x changes architectures; keep your base image/worker platform consistent (amd64 vs arm64) to avoid the mismatch you hit previously. Release notes for reference - OSS 3.x release notes: Prefect 3.x release notes - 3.6.5 diff: 3.6.5 - 3.6.0 diff: 3.6.0 - 3.5.0 (Python 3.10+): 3.5.0 If you share your Python version and whether you’re self-hosting Server or using Cloud, I can flag any remaining gotchas and tailor a quick rollout plan.

That’s normal. In Prefect 3.x OSS, migrations run on server startup but there isn’t an explicit “Running migrations…” banner. By default the server logs at WARNING; even at DEBUG you may not see a distinct migration message because Prefect calls Alembic directly without extra logging. If migrations were needed and failed, you’d see errors and the API wouldn’t start cleanly—so your observation is consistent with a successful migration. Quick ways to confirm - Show what would run (non-destructive):

prefect server database upgrade --dry-run

- Verify the migrate-on-start setting is enabled (it is by default):

prefect config view --show-defaults | grep PREFECT_SERVER_DATABASE_MIGRATE_ON_START

- If you prefer a definitive check against the DB, look at the

alembic_version

table in your database and ensure it’s at

head

. If you want more visibility next time - Start the server with higher log level:

PREFECT_SERVER_LOGGING_LEVEL=INFO prefect server start

- To see SQL emitted during migrations:

PREFECT_SERVER_DATABASE_SQLALCHEMY__ECHO=true prefect server database upgrade --dry-run

If you run the dry run and see no pending SQL/migrations, you’re fully up to date.

Thanks for the details — that error message is coming from Prefect’s lease maintainer. It means the lease backing your reserved slots could not be renewed and Prefect is running in strict mode for that lease, so it terminates the run to avoid over-allocation. A few key points about how this works in Prefect 3.x: - Leases are renewed proactively at about 75% of their duration. If renewal fails (e.g., 410 Gone from the API, network issue, or the lease has already expired), Prefect logs “Concurrency lease renewal failed…” and: - if strict=True: terminates the run - if strict=False: logs a warning and continues - Deployment-level concurrency uses a strict lease by default. Global concurrency you acquire in code is strict only if you set

strict=True

. - There aren’t any Cloud Run–specific issues reported around this in 3.6.5; this is most often caused by renewal timing out or the lease expiring before renewal runs. Given what you shared (Cloud Run worker pool, jobs ~6–7 minutes, you set lease timeout to 900s), a couple things don’t quite add up yet: with a 900s lease, the first renewal would occur around 675s (~11.25 minutes). If your jobs consistently finish in 6–7 minutes, they shouldn’t hit the first renewal at all. That suggests either: - The error is not from the global concurrency limit you configured but from a different strict lease (commonly: deployment concurrency), or - The lease duration in effect is not actually 900s for the lease that’s failing (e.g., default 300s), or - The process missed the renewal window (e.g., long blocking work, event loop starved, or a transient network/clock issue) and the server returned 410 Gone. A few targeted questions to narrow this down: 1) Where did you set the 900s “LEASE TIMEOUT”? In the Prefect Cloud UI for the global concurrency limit, or in code, or elsewhere? 2) Are you also using deployment-level concurrency on this deployment? 3) Do you set

strict=True

in any

prefect.concurrency.*.concurrency(...)

blocks in your flow? 4) Roughly when does the failure occur relative to run start? Around 3.75 minutes (75% of 300s), around 11.25 minutes (75% of 900s), or right at ~15 minutes? Immediate diagnostics and mitigations: - Turn up logging to DEBUG in the flow run to capture more context around the renewal attempt (any 4xx from the renew endpoint will be relevant). - If you’re using a code-level global concurrency block and can tolerate overages temporarily, try

strict=False

to validate that it’s a strict-termination path:

from prefect.concurrency.sync import concurrency

  with concurrency("your-limit-name", occupy=1, strict=False):
      # work

If the error becomes a warning and the run completes, we’ve confirmed it’s that lease. - If you suspect deployment-level concurrency is the strict lease that’s failing, temporarily disable deployment concurrency or inspect the deployment to confirm its concurrency settings:

prefect deployment inspect "<flow-name>/<deployment-name>"

- Double-check your Cloud Run job’s max runtime and that it’s not being throttled or paused in a way that would delay the renewal loop. Also ensure outbound connectivity to Prefect Cloud is stable throughout the run. - If you do need strict enforcement and long leases, increase the lease timeout further as a test and see if the failure shifts accordingly. That will tell us whether this is a simple expiry/renewal timing issue. Notes on versions: - I didn’t find any open issues specific to Cloud Run and lease renewal failures in 3.6.5, and there are no related GitHub Discussions at the moment. - There have been small concurrency-related improvements after 3.6.5; if you can upgrade, consider testing on the latest 3.6.x to pick up improvements. Full changelog compare: 3.6.5 → 3.6.6 If you can share: - A link to one failing run, - Where/how you set the 900s lease timeout,

- Whether you use deployment concurrency or

strict=True

in code, - The timestamp when the error logs relative to run start, …I can pinpoint which lease is failing (deployment vs global) and suggest the exact fix (e.g., adjust lease duration source, switch strict mode, or tune deployment concurrency).

Short answer: it’s not a network blip. An HTTP 410 Gone from the renew endpoint means the server no longer has a record of that lease — typically because it expired or was lost/revoked on the server side. That’s why the client immediately logs “Concurrency lease renewal failed …” and (in strict mode) terminates the run. Given your setup (Prefect Server on Cloud Run + Cloud Run worker pool), the most common causes are: - The server instance restarted or scaled to zero between acquisition and renewal while using in-memory (default) or filesystem lease storage. In Cloud Run, either will lose leases on restart/scale. - Multiple server instances without a shared lease backend (each instance has its own memory), so renew hits an instance that doesn’t know the lease. - The lease actually expired before the renewal was processed (renewals run at ~75% of the lease duration; if the process was paused, CPU-throttled, or blocked, the renewal could be late). - The concurrency limit was deleted or reset, revoking the lease. What to do: 1) If you’re self-hosting Prefect Server on Cloud Run, switch concurrency lease storage to Redis so leases survive restarts and are shared across instances:

# Use Redis-backed lease storage
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage

# Redis connection (shared with Prefect messaging)
PREFECT_REDIS_MESSAGING_HOST=your-redis-host
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
PREFECT_REDIS_MESSAGING_USERNAME=default
PREFECT_REDIS_MESSAGING_PASSWORD=your-password
PREFECT_REDIS_MESSAGING_SSL=false

If you run multiple server workers, Redis is required for messaging and leases. 2) Confirm which lease is failing by checking the timing: - ~225–240s after run start: likely deployment concurrency lease (defaults to 300s, strict). - ~75% of your configured global concurrency lease (e.g., ~675s for 900s): then it’s your global limit’s lease. 3) If it’s deployment concurrency and you truly need longer leases, adjust the server’s initial deployment lease duration:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=900

Note: deployment concurrency leases are strict by default; a failed renewal will terminate the run. 4) For code-level global concurrency, ensure you’re actually passing the desired lease duration and set strict mode according to your tolerance:

from prefect.concurrency.sync import concurrency

with concurrency("your-limit", occupy=1, lease_duration=900, strict=True):
    # your work

5) Check for restarts around the 410 in your Prefect Server logs and verify Cloud Run isn’t scaling your server instance to zero during runs. 6) If you can, test on the latest 3.6.x to pick up concurrency improvements: 3.6.5 → 3.6.6 If you can share: - Whether your Prefect Server is self-hosted on Cloud Run and which lease storage backend it’s using, - The timestamp of the 410 relative to run start, - Whether you’re using deployment concurrency for this deployment, …I can confirm which lease is failing (deployment vs global) and give you the exact configuration change to stop the 410s.

Thanks — that 410 is definitive signal from the server that the lease no longer exists. It’s not a transient network error; the renew reached the server and the server replied “lease is gone.” Given your details: - Self‑hosted Prefect Server with Redis lease storage - 410 happened ~16 minutes after run start - You’re using a Global concurrency limit (defined in the UI) What this timing strongly suggests - A global concurrency lease with a 900s (15 min) TTL expired, then the next renew attempt happened around minute 16 and received 410 because the lease had already been cleaned up. That typically happens when the renewal loop did not run on time (e.g., event loop blocked or process paused) so the lease hit its TTL. Important nuance about global concurrency TTLs - In Prefect 3.x, the TTL for a global concurrency lease comes from the client that acquires it (the

concurrency()

context), not from a per‑limit setting in the UI. The defaults are: - Public concurrency context:

lease_duration=300

unless you set it - Task tag concurrency (automatic, per task tag):

lease_duration=60

internally - If you intended 900s, make sure your flow code is acquiring the lease with that duration, e.g.:

from prefect.concurrency.sync import concurrency

  with concurrency("your-limit-name", occupy=1, lease_duration=900, strict=True):
      # your long-running work

If you’re relying solely on task tags for concurrency, note those leases are 60s by default — long tasks can easily outlive that unless the renew loop runs reliably. Why a renew could miss and let the lease expire - Async flows: long synchronous/blocking work can block the event loop, delaying the renewal task. If the planned renewal at 75% of the TTL (for 900s → ~11.25m) doesn’t run, the lease can expire at 15m; the next attempt (e.g., near 16m) will 410. - Heavy CPU-bound sections in a single process can starve scheduling. - The limit was deleted/reset (rare, but would revoke leases). - Redis eviction or restart removed the keys (check evictions/flushes; you said Redis is in use, but memory pressure or eviction policy could still be a factor). Concrete next steps 1) Confirm how you’re acquiring the limit: - Are you using

prefect.concurrency.concurrency(..., lease_duration=900, strict=?)

around the long-running section? If yes, please share that snippet. - Or are you relying on task tags only? If so, consider wrapping the long-lived section in an explicit

concurrency()

block with a larger

lease_duration

. 2) If your flow is async, don’t block the event loop: - Offload blocking work with a thread/process executor:

from prefect import flow
     from prefect.concurrency.asyncio import concurrency
     import anyio

     @flow
     async def run():
         async with concurrency("your-limit-name", lease_duration=900, strict=True):
             await anyio.to_thread.run_sync(long_blocking_function)

- Or keep the flow sync so the renewal runs in a background thread. 3) Validate Redis is not evicting leases: - Check

maxmemory

and

maxmemory-policy

(avoid allkeys/volatile evictions for lease keys). - Inspect

INFO stats

for

evicted_keys

and Redis logs around the failure time. - Ensure there were no Redis restarts or FLUSH events. 4) Quick confirmation test: - Temporarily increase

lease_duration

(e.g., 1800s) in your

concurrency()

call. If the 410 shifts correspondingly (~30 min), that confirms lease expiry due to missed renewal. - While diagnosing, you can set

strict=False

to avoid crashing on renewal failure and just log a warning. 5) Server logs: - Check Prefect Server logs around 15–16 minutes after start for the repossessor expiring that lease id or any errors touching concurrency leases. If you can share: - How you’re acquiring the global concurrency lease (code snippet), - Whether your flow is async and has long blocking sections,

- Redis

maxmemory-policy

and any eviction stats around the timestamp, …I can give you a precise fix. The most common resolution is to explicitly acquire with

lease_duration=900

and ensure the renewal loop isn’t blocked (or avoid async event loop blocking).

Thanks for the detail — this helps a lot. What your logs show - The server returned 410 Gone on renew. That’s a definitive “the lease no longer exists” from the server, not a network/transient issue. With Redis-backed leases, that typically means the lease expired (hit its TTL) and Redis cleaned it up; the next renew attempt gets 410. Why this matches your timing - You set the global concurrency lease timeout to 900s (15 minutes) in the UI. The maintainer renews at ~75% of the TTL. If that renewal loop doesn’t run on time (e.g., the process was busy/blocked), the lease can expire at 900s; the next renew attempt (your log shows at ~16 minutes) hits 410 because the lease is already gone. On your questions 1–2) You’re not acquiring in code; you’re using a global concurrency limit configured for the deployment in the UI. That path uses a server-acquired lease and the flow engine maintains it for the run. Those server-acquired leases are treated as strict by the engine: if renewal fails, the run terminates to avoid over-allocation. 3) Redis having “expired” keys is expected; that’s exactly how leases end. The key point is there were no restarts/flushes — good. 4) How to set strict=False via configuration? - There isn’t a Prefect-wide or deployment YAML knob to make server-acquired leases non‑strict. Server-issued leases (deployment concurrency and UI-configured global limits applied to the run) are enforced strictly by design. - If you want non‑strict behavior, you need to own the acquisition in code with the concurrency context and set

strict=False

, then remove the UI-managed global limit for that run. Example:

from prefect import flow
  from prefect.concurrency.sync import concurrency  # or prefect.concurrency.asyncio for async flows

  @flow
  def my_flow():
      with concurrency("your-global-limit-name", occupy=1, lease_duration=900, strict=False):
          # your long-running work
          ...

That will log a warning if renewal fails but will not terminate the run. Make sure to remove the UI-configured global limit for this deployment to avoid double-acquisition. - Note: the

concurrency_limit

block you showed in prefect.yaml:

concurrency_limit: { "limit": 50, "collision_strategy": "ENQUEUE" }

is a separate feature: per-deployment concurrency (max parallel runs for the deployment). That also uses a strict lease under the hood. If you don’t intend to use per‑deployment concurrency, remove this from your deployment config to avoid maintaining an extra lease. 5) The server log you shared confirms a clean, low-latency 410 on renew — consistent with “lease already expired,” not connectivity. What I recommend Option A (keep UI-managed global limit): - Increase the lease timeout in the UI (e.g., 1800s) and see if the 410 shifts accordingly. If yes, we’ve confirmed expiry due to missed renewal. - Ensure the renewal loop is not starved: - If your flow is async, don’t block the event loop; offload blocking work with

anyio.to_thread.run_sync

or similar. - If your flow is sync and CPU-heavy on a single core, the background renewal thread should still run, but severe CPU contention can delay scheduling; consider slightly higher TTL for safety. Option B (own it in code for non‑strict): - Remove the UI-attached global limit for this deployment and wrap the critical section in a

concurrency(...)

block with

lease_duration=900

and

strict=False

(or

True

if you still want strict). - This gives you full control over TTL, strictness, and renew behavior. Useful references - Global concurrency limits overview: Docs: Concurrency limits - Deployments and deployment concurrency: Docs: Deployments If you can confirm: - Whether your flow is async or sync, - Whether it does long blocking/CPU-bound work, - Whether you want strict enforcement or just best‑effort,

…I can suggest the exact wiring (either the code snippet to move concurrency into your flow, or how far to increase the TTL to avoid the rare late-renewal window).

Got it — thanks for clarifying. What’s happening - You’re using deployment-level concurrency via the deployment spec:

concurrency_limit: { limit: 50, collision_strategy: ENQUEUE }

. - That concurrency uses a strict lease under the hood. If the server says the lease no longer exists (410 Gone), Prefect terminates the run to prevent over-allocation. - Your 410 at ~16 minutes means the lease expired (hit its TTL) and when the engine tried to renew, the server had already cleaned it up. With Redis, that’s normal behavior for an expired lease — not a network issue. Key implications - Deployment concurrency strictness is not configurable via YAML or a global setting. There’s no “strict=False” switch for deployment concurrency; the engine always treats it as strict. - If a run can exceed the lease TTL, you must increase the TTL for deployment concurrency. How to increase the deployment concurrency lease duration You control it with the per-deployment “grace period” (initial lease duration). Add it to your deployment’s YAML:

deployments:
  - name: your-deployment
    entrypoint: path/to/flow.py:your_flow
    work_pool:
      name: your-pool
    concurrency_limit:
      limit: 50
      collision_strategy: ENQUEUE
      grace_period_seconds: 1200   # e.g., 20 minutes

Notes: -

grace_period_seconds

valid range is 60–86400. - This only affects deployment concurrency; it does not change strictness (still strict). Optional server-wide default (if you don’t want to set it per deployment) Set this on your Prefect Server:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=1200

This sets the default initial deployment concurrency lease duration for all deployments unless they override with

grace_period_seconds

. Double-check you aren’t also using a Global Concurrency Limit Since you only want deployment concurrency, please verify in the UI that there isn’t a Global Concurrency Limit also applied to this deployment (e.g., a limit named for the deployment or a tag). If there is, remove it — otherwise you end up with two leases to maintain, and either one expiring will terminate the run. Verify the setting Run:

prefect deployment inspect "<flow-name>/<deployment-name>"

Look for

concurrency_options.grace_period_seconds

in the output to confirm it’s applied. If you ever need non-strict behavior The only way is to manage concurrency in code: - Remove the deployment’s

concurrency_limit

- Wrap the critical section with a concurrency block and set `strict=False`:

from prefect import flow
from prefect.concurrency.sync import concurrency

@flow
def your_flow():
    with concurrency("your-limit-name", occupy=1, lease_duration=1200, strict=False):
        # work

But based on your goal (“only deployment concurrency”), the better path is to keep deployment concurrency and set

grace_period_seconds

high enough for your longest runs. Next steps I recommend - Add

grace_period_seconds

to your deployment YAML (try 1200s to start). - Ensure no Global Concurrency Limits are attached to this deployment in the UI. - Re-deploy and re-run. If a run occasionally exceeds even that window, bump it again. Docs for reference: - Deployment concurrency options: Deployments: Concurrency limiting - Concurrency limits concept: Global concurrency limits

You’re exactly right — the Global Concurrency Limit you see named like:

deployment:d1eed5c8-5415-40fe-8a65-4f71f86e002a

is the system-managed limit that backs your deployment-level concurrency. When you set

concurrency_limit

in your deployment spec, Prefect creates and manages that limit under the hood using the same global concurrency mechanism, so it shows up in the Global Concurrency Limits UI. You don’t need to create a separate limit for it. Why you’re seeing 410s - Deployment concurrency uses a strict lease. If the lease expires (TTL reached) before the renewal loop successfully renews it, the next renewal call returns 410 Gone and the run is terminated. - The fix is to increase the initial lease duration for deployment concurrency — called the grace period — so it comfortably covers your longest runs. How to fix it in your deployment YAML Add

grace_period_seconds

to your deployment’s

concurrency_limit

block. For 6–7 minute runs, give yourself buffer (e.g., 1200 seconds = 20 minutes):

deployments:
  - name: your-deployment
    entrypoint: path/to/flow.py:your_flow
    work_pool:
      name: your-pool
    concurrency_limit:
      limit: 50
      collision_strategy: ENQUEUE
      grace_period_seconds: 1200

Notes - Deployment concurrency strictness cannot be turned off via config; there is no

strict=False

knob for this path. If you ever need non-strict behavior, you’d have to manage concurrency in code with

concurrency(..., strict=False)

and remove the deployment-level concurrency — but based on your intent, stick with deployment concurrency and increase the grace period. - You can verify the applied value with:

prefect deployment inspect "<flow-name>/<deployment-name>"

Look for

concurrency_options.grace_period_seconds

. Optional server-wide default If you want a new default for all deployments (unless they override it), set this on your Prefect Server:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=1200

Docs - Deployments: Concurrency limiting: Docs - Concurrency limits concept: Docs If you bump

grace_period_seconds

and still see a 410 around the new TTL, share the new timestamp relative to run start and we can dig deeper into why the renewal loop might be missing its window.

Short answer: yes — a 410 at ~16 minutes almost certainly means the deployment-concurrency lease expired at 900s and the client’s renewal loop didn’t run in time, so the next renew hit “lease not found”. Why that happens in your setup - Deployment concurrency uses a strict lease that the flow engine renews on a schedule (~75% of the TTL). With a 900s TTL, the first renewal should run around 11m15s. - If that renewal loop is delayed (e.g., the process is CPU-bound and a background thread can’t get the GIL, or the process is otherwise paused/starved), the lease can hit its 900s TTL. Once expired, the next renew attempt returns 410 Gone — which matches your ~16 minute timestamp. - Redis “expired” keys are expected — that’s how expired leases are cleaned up. What you can do - Increase the initial lease/grace period for deployment concurrency to give more headroom: - Per deployment (recommended): add grace_period_seconds to your deployment spec:

deployments:
      - name: your-deployment
        entrypoint: path/to/flow.py:your_flow
        work_pool:
          name: your-pool
        concurrency_limit:
          limit: 50
          collision_strategy: ENQUEUE
          grace_period_seconds: 1200   # e.g., 20 minutes

- Or increase the server default further:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=1200

Then redeploy and verify with:

prefect deployment inspect "<flow-name>/<deployment-name>"

Look for concurrency_options.grace_period_seconds. - Make sure the renewal loop can run: - Since your flows are sync, a background renewal thread should be able to run — unless the work is CPU-bound pure Python that holds the GIL for long stretches. - If you have long, CPU-bound sections in pure Python, consider: - Moving those sections to multiprocessing (bypasses GIL), or - Using libraries that release the GIL (NumPy, Pandas C-ops usually do), or - Breaking long loops with brief yields (even tiny sleeps) so other threads can run. - Also confirm your Cloud Run job isn’t being CPU-throttled; Jobs generally have CPU while running, but double-check CPU allocation and limits. - Ensure there’s only one lease in play: - You’re using deployment concurrency only — good. Just make sure there isn’t an additional global limit attached to this deployment in the UI. The “deployment:<uuid>” you see is the system-managed one for deployment concurrency; you do not need a separate custom limit. If you bump grace_period_seconds to, say, 1200 and still get a 410 around ~20 minutes, that confirms the renewal loop is being delayed by several minutes. In that case, I’d focus on the CPU/GIL angle or further increase the grace period to match worst-case runtime plus buffer. If you can share whether your flow has long CPU-heavy pure-Python sections, I can suggest a concrete pattern (e.g., multiprocessing) to keep renewals timely.

The GIL (Global Interpreter Lock) is a mutex in CPython that allows only one thread to execute Python bytecode at a time. Why it matters: - CPU-bound Python code in multiple threads does not run in true parallel on multiple cores; only one thread runs Python code at a time. - I/O-bound threads usually “release” the GIL while waiting on I/O, so they can overlap nicely. - Many C extensions (NumPy, Pandas, some compression libs) release the GIL during heavy work, enabling parallelism across cores. But C code that holds the GIL (or long pure-Python loops) can block other threads. In your context: - Prefect’s lease-renewal runs in a background thread. If your flow spends long stretches in CPU-bound code that holds the GIL, that background thread may not get scheduled in time, the lease can expire, and a 410 shows up on the next renew attempt. Common workarounds: - Use multiprocessing for CPU-bound work to bypass the GIL. - Prefer libraries that release the GIL for heavy computation (NumPy/Pandas vectorized ops). - Break up long tight loops (even tiny sleeps/yields) so other threads can run. - Keep in mind there are emerging “free-threaded” Python builds without the GIL, but most production environments still use standard CPython with the GIL.

Great question. Short answer: deployment concurrency became strict earlier — in Prefect 3.4.15 — so both 3.4.20 and 3.6.5 enforce “raise on lease renewal failure” for deployment concurrency. Why you saw a warning on 3.4.20 but now get termination on 3.6.5: - In 3.4.20, the strict behavior already applied to the deployment-concurrency lease. If you saw the WARNING “Execution will continue…” it was likely from a different lease path (e.g., a user/global concurrency block in code or tag-based task concurrency) or it occurred after the run had already reached a final state. Later releases tightened behavior around when to raise vs. ignore after completion. - In 3.6.5, you’re definitely hitting the deployment-concurrency lease renewal failure, which is strict by design — hence termination. What to do: - Make sure the lease window comfortably covers the longest run: - Per-deployment override in your YAML (recommended):

deployments:
      - name: your-deployment
        entrypoint: path/to/flow.py:your_flow
        work_pool:
          name: your-pool
        concurrency_limit:
          limit: 50
          collision_strategy: ENQUEUE
          grace_period_seconds: 1200   # try 20 minutes

- Or increase the server default further:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=1200

- Verify the value applied:

prefect deployment inspect "<flow-name>/<deployment-name>"

Look for

concurrency_options.grace_period_seconds

. - Since your flows are sync, the renewal runs in a background thread; if you have long, CPU-bound pure-Python loops, they can still starve other threads. If you still see expirations near the TTL after increasing it: - Add tiny sleeps/yields in long tight loops or move heavy CPU work to multiprocessing. - Or bump

grace_period_seconds

further to match worst-case runtime. References: - Deployments: concurrency limiting Docs - Concurrency limits concept Docs If you want non-strict behavior, that’s only possible by managing concurrency in code (remove deployment concurrency and wrap with

concurrency(..., strict=False)

), but if your intent is to use deployment concurrency, increasing

grace_period_seconds

is the right fix.

Thanks — for async flows, the most common reason leases expire is the event loop being blocked long enough that the renewal task can’t run before the TTL. Since deployment concurrency leases are strict, that leads to termination once the server returns 410. What to check and change for async flows: - Increase the deployment concurrency lease window (grace period) so you have plenty of buffer beyond the longest possible blocking interval:

deployments:
    - name: your-deployment
      entrypoint: path/to/flow.py:your_flow
      work_pool:
        name: your-pool
      concurrency_limit:
        limit: 50
        collision_strategy: ENQUEUE
        grace_period_seconds: 1800   # try 30 minutes (valid range 60–86400)

Then redeploy and verify with:

prefect deployment inspect "<flow-name>/<deployment-name>"

Look for

concurrency_options.grace_period_seconds

. - Avoid blocking the event loop in async flows: - Don’t call sync/blocking APIs directly (e.g.,

requests

, most Google Cloud Python clients, boto3, pandas IO, time.sleep, heavy pure-Python loops) from async code. - Offload sync or CPU-bound work so the event loop stays free for the renewal task:

import anyio

    # Blocking I/O or sync client
    result = await anyio.to_thread.run_sync(sync_fn, *args, **kwargs)

    # CPU-bound work: prefer a process to bypass the GIL and keep the loop free
    from anyio import to_process
    result = await to_process.run_sync(cpu_heavy_fn, *args, **kwargs)

    # If you have long async loops, add periodic yields
    await asyncio.sleep(0)  # inside long-running loops

- For HTTP, prefer async clients (e.g.,

httpx.AsyncClient

,

aiohttp

) where possible; otherwise wrap the sync calls with

to_thread.run_sync

. - Sanity-check timing: - Renewals run at ~75% of the lease duration. With 900s TTL, the renewal should happen ~11m15s in. If you’re consistently seeing 410s around TTL+1 minute, the loop was blocked past expiry. - If your async jobs sometimes spend many minutes in blocking sections, push

grace_period_seconds

higher (e.g., 3600s) to give a big buffer while you refactor hotspots. - Optional test to confirm cause: - Temporarily raise

grace_period_seconds

substantially (e.g., 3600). If failures shift to ~1h, it’s purely a missed renewal window. - Enable more logging to correlate timings:

PREFECT_LOGGING_LEVEL=DEBUG

on your worker to capture renewal attempts and failures. Docs for reference: - Deployments: concurrency limiting Docs - Concurrency limits concept Docs If you can share a small snippet of one failing async flow (especially the parts doing network IO, pandas/BigQuery work, or heavy computation), I can point to the exact spots to offload to threads/processes so the lease renewal task stays on schedule.

Short answer: No — your Cloud Run worker (and the Cloud Run job containers that run your flows) do not need Redis env vars for concurrency leases. Why: - Concurrency leases are stored/managed by the Prefect Server. The worker and flow runtime only talk to the server over HTTP (renewals call the server’s /concurrency_limits/... endpoints). They never talk to Redis directly. - Only the Prefect Server process needs the Redis settings (e.g., PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage and the PREFECT_REDIS_MESSAGING_* connection vars). What the worker/flow runtimes do need: - PREFECT_API_URL pointing to your self-hosted server endpoint - Any GCP credentials required for Cloud Run Job submission (unrelated to Redis) Docs for reference: - Concurrency limits: Docs: Concurrency limits - Deployments and deployment concurrency: Docs: Deployments concurrency So your server should have the Redis env vars; the worker/jobs do not.

Great question. Here’s a practical overview of tag-based concurrency limits in Prefect 3.x, how to use them, and what “strict” means in this context. High level - Tag-based limits control how many task runs with a given tag can run at the same time. - They’re configured centrally (CLI/UI/API) and enforced by Prefect’s orchestration layer — you don’t add any code in your tasks to “acquire” them. - By default they are “soft” during execution: if the lease renewal for a running task fails, the task continues and you’ll see a warning/debug log. They do not cancel the task by default. How to use tag-based concurrency limits 1) Tag your tasks

from prefect import task, flow

@task(tags=["db"])
def do_work(i):
    ...

@flow
def my_flow():
    for i in range(20):
        do_work.submit(i)

2) Create and manage the limit (CLI)

# Create a limit of 5 for the "db" tag
prefect concurrency-limit create db 5

# List all tag limits
prefect concurrency-limit ls

# Inspect a specific tag limit (shows active task run IDs)
prefect concurrency-limit inspect db

# Reset slots (e.g., if they got wedged)
prefect concurrency-limit reset db

# Delete the limit
prefect concurrency-limit delete db

3) Behavior and queuing - When a task is scheduled, Prefect acquires a slot for each of its tags. If any tag is at capacity, the task waits; the scheduler retries with backoff (no explicit “queue” object). - If a task has multiple tags, it must have capacity for all tags (logical AND). - The slots are tracked via timed leases; for tag-based limits, Prefect renews the lease periodically during execution. Are tag-based limits “strict”? - Default: Not strict during execution. If the lease renewal fails mid-run, Prefect logs a message and the task keeps running. Concurrency may be exceeded temporarily. - Strict cancellation on lease renewal failure is used for deployment-level concurrency, not tag-based task limits. If you’re seeing runs cancel with messages like “Terminating execution to prevent over-allocation,” that’s usually a deployment concurrency lease, or explicit global concurrency usage with strict=True — not tag tags. Where to configure/monitor - CLI: see above. - UI: Concurrency section. You’ll see “tag:your-tag” entries (tag limits are backed by global concurrency limits under the hood). You can enable/disable, update, and observe slot usage there. - Docs: - Tag-based concurrency concepts: Concepts - How-to (tags): How-to - CLI (tags): CLI reference - Global concurrency concepts: Concepts - CLI (global): CLI reference Cloud Run specifics (since your jobs run ~6–7 minutes) - Tag-based task leases renew on a short interval (~60s lease with periodic renewal). Temporary network blips in Cloud Run can cause “Concurrency lease renewal failed” logs. For tag-based limits this should not cancel the task by default; it should log and continue. - If you’re seeing cancellations/crashes, it’s likely one of: - Deployment concurrency is enabled for the deployment (that one is strict by default and will cancel the flow if renewal fails). - You’re using the global concurrency context manager in code with strict=True. - Recommendation: - Confirm whether your deployment has “Max concurrent runs” or a deployment concurrency limit enabled. - Check the exact log line: does it say “Execution will continue…” (soft) or “Terminating execution…” (strict)? - If you do use global concurrency in code on Cloud Run, consider a longer lease_duration (e.g., 300–600s) and avoid strict=True unless you need hard guarantees. Quick checklist to improve usage - Keep tag sets minimal; remember multi-tag is AND’ed.

- Size the limit close to real capacity (DB connections, API quotas). - Use the UI to monitor active and denied slots; adjust limits if you see persistent blocking. - Reset a limit only if you’re sure slots are wedged. - If you need per-flow-run caps, consider deployment concurrency — but be aware it is strict on lease renewal by default. - If you need strict behavior inside code, use global concurrency contexts with strict=True (not tag-based). Otherwise, prefer the default soft behavior. A couple of clarifying questions to help pin down your crashes: - Are the failures happening on flow runs (entire flow cancels) or just specific task runs? - Do you have a deployment-level “Max concurrent runs” set, or any explicit global concurrency contexts in code? - Could you share the exact log line around “Concurrency lease renewal failed” — does it include “Execution will continue” or “Terminating execution”? If you can share those details, I can recommend a targeted fix for your Cloud Run setup.

You’ve got a few good “non‑strict” options to cap concurrency per deployment/flow without using deployment concurrency (which is strict and can cancel runs on lease renewal failure). Option A — Per‑deployment via a dedicated work queue (recommended) - Create a work queue just for that deployment and set a queue‑level limit. This gates new flow runs at submission time and never cancels running ones. - Commands:

# Create a queue in your existing work pool
prefect work-queue create my-deploy-q --pool <your-pool>

# Cap concurrency for that queue
prefect work-queue set-concurrency-limit my-deploy-q 2

# See queues and limits
prefect work-queue ls

- Route your deployment to that queue: - Easiest in UI: Deployment > Edit > Work queue = my-deploy-q - Or set it at deploy time if you use CLI (UI path is simplest). - Notes: - Queue limit <= pool limit (if pool also has a limit). - This approach avoids concurrency lease renewal at the flow level, so you won’t see “lease renewal failed” logs. It’s a “soft” cap on starting new runs. Option B — Per‑pool cap (broad brush) - If you can isolate a deployment into its own work pool, set the pool’s concurrency limit:

prefect work-pool set-concurrency-limit <your-pool> 2
prefect work-pool ls

- This also gates submission (no cancellations), but it applies to all runs in that pool. Use this if you don’t need per‑deployment granularity. Option C — Per‑flow (soft) using Global Concurrency Limits in code - Use a Global Concurrency Limit (GCL) with a name per flow/deployment and wrap your flow body. With strict=False (default), failure to renew the lease logs but does not cancel. - Create the GCL:

prefect gcl create flow:my-flow --limit 2

- Wrap your flow:

from prefect import flow
from prefect.concurrency.sync import concurrency

@flow
def my_flow():
    with concurrency("flow:my-flow", strict=False):
        # flow body
        ...

- For async flows:

from prefect.concurrency.asyncio import concurrency

@flow
async def my_flow():
    async with concurrency("flow:my-flow", strict=False):
        ...

- Tradeoffs: - Pro: Easy, truly per‑flow/deployment, remains “soft” (no cancellations). - Con: Uses leases under the hood; in Cloud Run you may still see occasional “Concurrency lease renewal failed … execution will continue” logs during network blips. Key points for your environment (Cloud Run v2, 6–7 min jobs) - If you want to avoid lease-related logs entirely, prefer Option A (queue limit) or B (pool limit). They control submission and do not rely on leases during execution. - If you do use Option C, keep

strict=False

(default). That way, even if a lease renewal fails, the flow keeps running. Docs and references - Work pools: Concepts and How-to - Work queue CLI: - CLI reference - Verified commands:

prefect work-queue set-concurrency-limit

,

prefect work-queue clear-concurrency-limit

,

prefect work-queue ls

- Global concurrency limits: Concepts and CLI reference Quick suggestion for you - For “per deployment, non‑strict” caps with Cloud Run, a dedicated work queue per deployment with a queue limit is usually the cleanest and most robust. Do you prefer to manage this via UI or CLI? If you share the deployment name and pool, I can provide exact commands/steps for your setup.

Yes — a work queue concurrency limit caps the number of flow runs that can be started from that queue at the same time. It does not affect tasks inside those flows. Think “how many flows from this queue can run concurrently,” not “how many tasks.” How to set it up (Option A) - Create a dedicated queue for your deployment in your Cloud Run work pool and give it a limit:

# Create a queue and set its limit in one go
prefect work-queue create my-deploy-q --pool <your-pool> --limit 2

# Or create first, then set the limit
prefect work-queue create my-deploy-q --pool <your-pool>
prefect work-queue set-concurrency-limit my-deploy-q 2

- Point your deployment at that queue: - UI: Deployment > Edit > Work queue = my-deploy-q (save) - Verify/monitor:

prefect work-queue ls
prefect work-queue inspect my-deploy-q

Notes - If your work pool also has a pool-level concurrency limit, the effective concurrency is constrained by both; a queue cannot exceed the pool’s capacity. - Multiple deployments can share a queue and will share its limit. Use one queue per deployment if you want strict isolation. - This gates new flow runs at submission time (no mid-run cancellations and no lease renewal noise). Tasks inside those flows are unaffected; if you also need task-level caps, add tag-based concurrency to those tasks. Docs and references - Work pools (concepts): Concepts - Work queue CLI: CLI reference If you share your work pool name and desired queue/limit, I can give you the exact commands tailored to your setup.

Perfect. Here are the exact commands to create the queue and set its limit:

# Create the 'api' work queue in your 'cloud-run-v2' pool with a limit of 50
prefect work-queue create api --pool cloud-run-v2 --limit 50

# Verify it was created and the limit is applied
prefect work-queue inspect api
prefect work-queue ls

Next, point your deployment(s) to the

api

queue: - UI: Deployment > Edit > Work queue =

api

> Save Notes: - This caps concurrent flow runs launched from the

api

queue at 50. It won’t cancel in-flight runs; it just gates new starts. - If the

cloud-run-v2

pool has its own pool-level limit set lower than 50, that lower pool limit will be the effective cap. - Multiple deployments can share

api

and will share the 50‑run limit. Create more queues if you want isolation per deployment. If you want me to double-check your pool’s current limit to ensure it won’t constrain the queue, I can share the command for that too.

Almost. With a work queue limit: - Excess flow runs do not move into AwaitingConcurrencySlot. They simply remain in Scheduled until the worker can pick them up. - AwaitingConcurrencySlot is used by deployment concurrency limits (server-side orchestration). Work queue limits are enforced at retrieval time: when a worker asks for work, the API only returns up to the number of open slots for that queue. So with your queue limit = 50: - Up to 50 flow runs from queue api can be Running at once. - Any additional scheduled runs for that queue will just stay Scheduled until one of the 50 finishes and a worker polls again. This is why queue-level limits are nice and “soft”: they gate starts but don’t introduce lease-renewal or mid-run cancellations.

Short answer: yes — the queue limit is enforced at “work retrieval” time. Extra runs stay Scheduled until capacity opens; they do not move to AwaitingConcurrencySlot. Here’s the technical flow with a queue limit of 50 on queue `api`: 1) Worker polls the server for ready flow runs from queue

api

- “Ready” = Scheduled with scheduled_time <= now and eligible to start. - Server computes open slots: open_slots = 50 - active_runs_for_queue - active_runs_for_queue counts currently-active flow runs for that queue across all workers (e.g., Running/Submitted). 2) Server returns up to open_slots runs - If open_slots is 0 (because 50 are already active), the server returns an empty list. The remaining Scheduled runs are untouched and remain Scheduled. 3) Worker submits the returned runs - Those runs transition through Pending → Submitted → Running as usual. 4) Next poll while 50 are still active - open_slots is still 0 → server returns nothing → Scheduled runs remain Scheduled. 5) When one active run finishes - active_runs_for_queue drops below 50 → on the next poll, open_slots > 0 and the server returns up to that many Scheduled runs. Key properties - Global per-queue cap: enforced across all workers attached to the same queue (not per worker). - No special state: runs exceeding the cap remain in Scheduled; AwaitingConcurrencySlot is only used by deployment-level concurrency limits. - Selection order: the server returns eligible runs by earliest scheduled time (and a stable tiebreaker). Queue priority only matters when a worker is polling multiple queues; within a single queue, it’s schedule-time-based. - Pool-level limit (if set) also applies: the effective throughput is min(queue_limit, pool_limit). Helpful commands

# See which runs would be returned next from the queue
prefect work-queue preview api

# Inspect queue and confirm the limit
prefect work-queue inspect api

# List all queues and limits
prefect work-queue ls

Docs - Work pools and queues: Concepts - Work queue CLI: CLI reference

Great question. The short version: a Prefect “work queue” is not a message broker; it’s a persisted object in the Prefect API’s database that defines routing and limits. There is no in‑memory queue to lose on restart. Flow runs themselves are rows in the database with states. Workers repeatedly query for eligible runs from a queue; the API returns up to the number of open slots based on the queue’s limit and how many runs are already active for that queue. More technical detail - Where the queue “lives”: - It’s a database-backed resource stored by the Prefect API (Prefect Cloud or your self-hosted Prefect Server). The queue definition (name, pool, limit, priority) is durable. - Flow runs are also stored in the same database with their current state (Scheduled, Pending, Running, etc.). - How the limit is enforced: - When a worker polls, the API computes open_slots = queue_limit − active_runs_for_that_queue and returns at most open_slots scheduled runs. - active_runs_for_that_queue is computed by counting matching flow runs in “running-ish” states (e.g., Pending/Submitted/Running) for that queue across all workers. - If open_slots is 0 (e.g., 50 already active), the API returns nothing; the additional runs simply remain Scheduled. - What happens on restarts: - Prefect API restart: All definitions (work pools/queues) and flow runs are persisted. When the API comes back, workers resume polling; the concurrency calculation still works because it’s derived from persisted flow run states. - Worker restart: Workers are stateless. On restart they just resume polling. Runs that were already submitted to Cloud Run continue independently. If a worker fetched runs but crashed before submitting, those runs remained Scheduled and will be picked up later. - There are no queue “leases” to get stuck for work-queue concurrency; it’s computed on demand by counting active runs. - No special state transitions: - Exceeding the queue limit does not move runs into AwaitingConcurrencySlot. They stay Scheduled until capacity opens. AwaitingConcurrencySlot is specific to deployment concurrency limits. Useful references - Work pools and queues (concepts): Concepts - Work queue CLI: CLI reference - REST API (work pools/queues): Work Pools API and Work Queues API If you want, I can also show you how to preview exactly which runs the API would return from

api

at any moment using:

prefect work-queue preview api