https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> ```Opening process... 12:39:53 PM pre...
# marvin-ai
k
@Marvin
Copy code
Opening process...
12:39:53 PM
prefect.flow_runs.runner
Completed submission of flow run '019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9'
12:39:55 PM
prefect.flow_runs.worker
 > Running git_clone step...
12:40:38 PM
prefect.flow_runs
Beginning flow run 'cautious-albatross' for flow 'clickstream-dbt-run'
12:41:35 PM
prefect.flow_runs
Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.
12:41:36 PM
prefect.flow_runs
Crash detected! Execution was cancelled by the runtime environment.
12:41:37 PM
prefect.flow_runs
Finished in state Crashed('Execution was cancelled by the runtime environment.')
12:41:38 PM
prefect.flow_runs
Engine execution exited with unexpected exception
Traceback (most recent call last):
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flow_engine.py", line 1552, in run_flow
    ret_val = run_flow_sync(**kwargs)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flow_engine.py", line 1395, in run_flow_sync
    engine.call_flow_fn()
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flow_engine.py", line 801, in call_flow_fn
    result = call_with_parameters(self.flow.fn, self.parameters)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/utilities/callables.py", line 210, in call_with_parameters
    return fn(*args, **kwargs)
  File "flows/clickstream/clickstream.py", line 152, in clickstream_dbt_run
    stack.enter_context(
  File "/usr/lib64/python3.9/contextlib.py", line 448, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/concurrency/sync.py", line 110, in concurrency
    acquisition_response = _acquire_concurrency_slots_with_lease(
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/concurrency/sync.py", line 57, in _acquire_concurrency_slots_with_lease
    result = run_coro_as_sync(
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 185, in result
    self._condition.wait(timeout)
  File "/usr/lib64/python3.9/threading.py", line 312, in wait
    waiter.acquire()
prefect._internal.concurrency.cancellation.CancelledError
My setup: deployed 6 deployments via prefect.yaml each deployment has its own schedule, and are separated with appropriate offset of intervals to avoid collisions due to my RDS bottleneck had to define global concurrency limit (which is set as 2) also there is per deployment level concurrency limit which is 1 and the straegy is CANCEL_NEW i was using this from quite sometime but recently i have started getting these errors, what is the real cause, also this is the entrypoint code for all the deployments def function(...params): with ExitStack() as stack: try: stack.enter_context( concurrency( PREFECT_ALL_FLOWS_GCL, # name of the gcl timeout_seconds=PREFECT_ALL_FLOWS_GCL_SLOT_TIMEOUT, # short timeout for the clickstream as when the slot is not available, dont wait in the queue and exit strict=True, # fail if limit is missing or cannot be enforced lease_duration=PREFECT_ALL_FLOWS_GCL_LEASE_DURATION ) ) except AcquireConcurrencySlotTimeoutError: return Cancelled( message=f"No dbt-runs slots available within {PREFECT_ALL_FLOWS_GCL_SLOT_TIMEOUT}s; exiting early" ) except ConcurrencySlotAcquisitionError as e: logger.error(f"Failed to acquire dbt-runs slot: {e}") return Failed(message=str(e)) some tasks...
m
✅ thought for 437.0 seconds
Thanks for sharing the logs and your setup — this points clearly to a lost concurrency lease. What’s happening - The log “Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.” means the workflow concurrency lease held by your 
concurrency(...)
context could not be renewed. Because you’re using 
strict=True
, Prefect cancels the run to avoid exceeding your global limit — hence the 
CancelledError
and Crashed state. - This is unrelated to your deployment-level concurrency strategy (CANCEL_NEW). That strategy only affects scheduling of new runs; the error is from the workflow-level global limit enforced by your 
concurrency(...)
context. Common triggers - Lease duration too short for the time you hold the concurrency context; a renewal happens at ~75% of 
lease_duration
. If the renewal call fails (even briefly) the run is cancelled when 
strict=True
. - Temporary network/API hiccup at the renewal moment (Prefect Cloud/Server unreachable). - The concurrency limit was deleted/renamed while the run was in-flight. - Less commonly: server-side cleanup of the lease due to TTL expiration before renewal. - There are also a few open/related reports around renewals and cancellation: - #19068 - #18839 - #20251 (flows in AwaitingRetry) Why it started “recently” - If 
PREFECT_ALL_FLOWS_GCL_LEASE_DURATION
was reduced (e.g., ~60s), your renewal happens very quickly (~45s). Any brief API/network hiccup will cause an immediate cancel with 
strict=True
. - If there were recent network/proxy changes, Prefect version upgrades, or changes to the concurrency limit (rename/reset/delete), those can surface during the next renewal. What to change 1) Set a safer lease duration - Make 
lease_duration
comfortably longer than the total time you hold the concurrency lock (include your dbt run). A practical starting point is 10–30 minutes for medium jobs. - Example:
Copy code
with concurrency(
      PREFECT_ALL_FLOWS_GCL,
      timeout_seconds=PREFECT_ALL_FLOWS_GCL_SLOT_TIMEOUT,
      strict=True,
      lease_duration=1800  # 30 minutes
  ):
      ...
- If you want to prove this is the root cause, temporarily increase 
lease_duration
(e.g., 900–1800s) and see if the errors disappear. 2) Verify the limit exists and is stable - Double-check the limit name is correct and present:
Copy code
prefect concurrency-limit ls
  prefect concurrency-limit inspect <YOUR_LIMIT_NAME>
If you’re programmatically deleting/recreating this limit, avoid doing so while runs are active. 3) Consider the trade-off of strict enforcement - If brief over-allocation is acceptable, you can set 
strict=False
to continue execution when a renewal blips. That trades hard guarantees for resiliency. - If you must strictly protect RDS, keep 
strict=True
but favor a larger 
lease_duration
to ride out short API blips. 4) Check for environmental causes - Look at worker logs around the renewal time for HTTP errors/timeouts to the Prefect API. - Confirm whether you recently upgraded Prefect or changed network/proxy/firewall settings. A few confirmations - The 
concurrency
signature in Prefect 3.x is:
Copy code
concurrency(
    names, occupy=1, timeout_seconds=None, max_retries=None,
    lease_duration=300, strict=False, holder=None, suppress_warnings=False
  )
- The CLI to inspect limits exists:
Copy code
prefect concurrency-limit --help
with subcommands 
ls
, 
inspect
, 
create
, 
reset
, 
delete
. Questions to help pinpoint the cause - What is your current 
PREFECT_ALL_FLOWS_GCL_LEASE_DURATION
? - Are you on Prefect Cloud or Prefect Server, and which Prefect version? - Did these failures start after a version upgrade or network change? - Do your flows use long `retries`/`retry_delay_seconds` that could park a run for a while?
If you share those details, I can suggest exact values and next steps tailored to your environment.
4 Views
Open in Slack
PreviousNext
Powered by