https://prefect.io logo
g

Gregory Hunt

09/06/2023, 2:17 PM
I am getting this error in my process workers and it is cause the worker to crash and hang flow runs in a pending or running state
Copy code
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
  File "/opt/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 383, in __call__
    return self.result()
  File "/opt/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
  File "/opt/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
  File "/opt/venv/lib/python3.9/site-packages/prefect/cli/worker.py", line 178, in start
    server_thread.start()
  File "/opt/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/opt/venv/lib/python3.9/site-packages/prefect/utilities/services.py", line 53, in critical_service_loop
    await workload()
  File "/opt/venv/lib/python3.9/site-packages/prefect/workers/base.py", line 529, in get_and_submit_flow_runs
    return await self._submit_scheduled_flow_runs(flow_run_response=runs_response)
  File "/opt/venv/lib/python3.9/site-packages/prefect/workers/base.py", line 738, in _submit_scheduled_flow_runs
    self._limiter.acquire_on_behalf_of_nowait(flow_run.id)
  File "/opt/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1846, in acquire_on_behalf_of_nowait
    raise RuntimeError(
RuntimeError: this borrower is already holding one of this CapacityLimiter's tokens
b

Bianca Hoch

09/06/2023, 3:57 PM
Hi Gregory, thanks for reporting this. Another user reported a similar error this morning and created an issue here. Would you be willing to create a separate issue, or comment on the original detailing the behavior that you're seeing?
g

Gregory Hunt

09/06/2023, 4:52 PM
I commented on the issue