@Marvin

Worker 'ProcessWorker b738eab3-5c9a-4e52-b72c-d5285e7541b3' submitting flow run '019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9'
12:39:50 PM
prefect.flow_runs.worker
Opening process...
12:39:53 PM
prefect.flow_runs.runner
Completed submission of flow run '019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9'
12:39:55 PM
prefect.flow_runs.worker
 > Running git_clone step...
12:40:38 PM
prefect.flow_runs
Beginning flow run 'cautious-albatross' for flow 'clickstream-dbt-run'
12:41:35 PM
prefect.flow_runs
Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.
12:41:36 PM
prefect.flow_runs
Crash detected! Execution was cancelled by the runtime environment.
12:41:37 PM
prefect.flow_runs
Finished in state Crashed('Execution was cancelled by the runtime environment.')
12:41:38 PM
prefect.flow_runs
Engine execution exited with unexpected exception
Traceback (most recent call last):
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flow_engine.py", line 1552, in run_flow
    ret_val = run_flow_sync(**kwargs)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flow_engine.py", line 1395, in run_flow_sync
    engine.call_flow_fn()
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flow_engine.py", line 801, in call_flow_fn
    result = call_with_parameters(self.flow.fn, self.parameters)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/utilities/callables.py", line 210, in call_with_parameters
    return fn(*args, **kwargs)
  File "flows/clickstream/clickstream.py", line 152, in clickstream_dbt_run
    stack.enter_context(
  File "/usr/lib64/python3.9/contextlib.py", line 448, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/concurrency/sync.py", line 110, in concurrency
    acquisition_response = _acquire_concurrency_slots_with_lease(
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/concurrency/sync.py", line 57, in _acquire_concurrency_slots_with_lease
    result = run_coro_as_sync(
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result
    return self.future.result(timeout=timeout)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 185, in result
    self._condition.wait(timeout)
  File "/usr/lib64/python3.9/threading.py", line 312, in wait
    waiter.acquire()
prefect._internal.concurrency.cancellation.CancelledError
12:41:38 PM
prefect.flow_runs
Process for flow run 'cautious-albatross' exited with status code: 1
12:41:50 PM
prefect.flow_runs.runner
Loading flow to check for on_crashed hooks
12:41:51 PM
prefect.flow_runs.runner
 > Running git_clone step...
12:41:52 PM
prefect.flow_runs
[Errno 2] No such file or directory
12:41:55 PM
prefect.flow_runs
Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9').
Traceback (most recent call last):
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 136, in run_step
    step_func = _get_function_for_step(fqn, requires=keywords.get("requires"))
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 78, in _get_function_for_step
    step_func = import_object(fully_qualified_name)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 168, in import_object
    module = load_module(module_name)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 133, in load_module
    working_directory = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/runner/runner.py", line 1476, in _run_on_crashed_hooks
    flow = await load_flow_from_flow_run(
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/flows.py", line 2810, in load_flow_from_flow_run
    output = await run_steps(
  File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone

My setup: deployed 6 deployments via prefect.yaml each deployment has its own schedule, and are separated with appropriate offset of intervals to avoid collisions due to my RDS bottleneck had to define global concurrency limit (which is set as 2) also there is per deployment level concurrency limit which is 1 and the straegy is CANCEL_NEW i was using this from quite sometime but recently i have started getting these errors, what is the real cause, also this is the entrypoint code for all the deployments def function(...params): with ExitStack() as stack: try: stack.enter_context( concurrency( PREFECT_ALL_FLOWS_GCL, # name of the gcl timeout_seconds=PREFECT_ALL_FLOWS_GCL_SLOT_TIMEOUT, # short timeout for the clickstream as when the slot is not available, dont wait in the queue and exit strict=True, # fail if limit is missing or cannot be enforced lease_duration=PREFECT_ALL_FLOWS_GCL_LEASE_DURATION ) ) except AcquireConcurrencySlotTimeoutError: return Cancelled( message=f"No dbt-runs slots available within {PREFECT_ALL_FLOWS_GCL_SLOT_TIMEOUT}s; exiting early" ) except ConcurrencySlotAcquisitionError as e: logger.error(f"Failed to acquire dbt-runs slot: {e}") return Failed(message=str(e)) some tasks...

Your message was too long, here's your message at the allowed limit: ```Worker 'ProcessWorker b738eab3-5c9a-4e52-b72c-d5285e7541b3' submitting flow run '019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9' 12 39 50 PM prefect flow_runs worker Opening process 12 39 53 PM prefect flow_runs runner Completed submission of flow run '019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9' 12 39 55 PM prefect flow_runs worker > Running git_clone step 12 40 38 PM prefect flow_runs Beginning flow run 'cautious-albatross' for flow 'clickstream-dbt-run' 12 41 35 PM prefect flow_runs Concurrency lease renewal failed - slots are no longer reserved Terminating execution to prevent over-allocation 12 41 36 PM prefect flow_runs Crash detected! Execution was cancelled by the runtime environment 12 41 37 PM prefect flow_runs Finished in state Crashed('Execution was cancelled by the runtime environment ') 12 41 38 PM prefect flow_runs Engine execution exited with unexpected exception Traceback (most recent call last) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/flow_engine py line 1552 in run_flow ret_val = run_flow_sync(*kwargs) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/flow_engine py line 1395 in run_flow_sync engine call_flow_fn() File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/flow_engine py line 801 in call_flow_fn result = call_with_parameters(self flow fn self parameters) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/utilities/callables py line 210 in call_with_parameters return fn(*args *kwargs) File flows/clickstream/clickstream py line 152 in clickstream_dbt_run stack enter_context( File /usr/lib64/python3 9/contextlib py line 448 in enter_context result = _cm_type __enter__(cm) File /usr/lib64/python3 9/contextlib py line 119 in enter return next(self gen) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/concurrency/sync py line 110 in concurrency acquisition_response = _acquire_concurrency_slots_with_lease( File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/concurrency/sync py line 57 in _acquire_concurrency_slots_with_lease result = run_coro_as_sync( File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/utilities/asyncutils py line 207 in run_coro_as_sync return call result() File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/_internal/concurrency/calls py line 365 in result return self future result(timeout=timeout) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/_internal/concurrency/calls py line 185 in result self _condition wait(timeout) File /usr/lib64/python3 9/threading py line 312 in wait waiter acquire() prefect _internal concurrency cancellation CancelledError 12 41 38 PM prefect flow_runs Process for flow run 'cautious-albatross' exited with status code 1 12 41 50 PM prefect flow_runs runner Loading flow to check for on_crashed hooks 12 41 51 PM prefect flow_runs runner > Running git_clone step 12 41 52 PM prefect flow_runs [Errno 2] No such file or directory 12 41 55 PM prefect flow_runs Runner failed to retrieve flow to execute on_crashed hooks for flow run UUID('019c7469-b4b1-7a6a-ab42-a8b5d1e1b9b9') Traceback (most recent call last) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/deployments/steps/core py line 166 in run_steps step_output = await run_step(step upstream_outputs) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/deployments/steps/core py line 136 in run_step step_func = _get_function_for_step(fqn requires=keywords get( requires )) File /home/ec2-user/prefect-dbt/prefect-dbt-env/lib/python3 9/site-packages/prefect/deployments/steps/core py line 78 in _get_function_for_step step_func = import_object(fully_qualified_name) File /home/ec2-user/p