has anyone see this kind of error before..?

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 2467, in <module>
    enter_flow_run_engine_from_subprocess(flow_run_id)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 297, in enter_flow_run_engine_from_subprocess
    state = from_sync.wait_for_call_in_loop_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/api.py", line 242, in wait_for_call_in_loop_thread
    waiter.wait()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/waiters.py", line 152, in wait
    self._handle_waiting_callbacks()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/waiters.py", line 126, in _handle_waiting_callbacks
    callback.run()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 273, in run
    return self.context.run(asyncio.run, self._run_async(coro))
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1871, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/lib/python3.10/selectors.py", line 469, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1866, in cancel_flow_run
    raise TerminationSignal(signal=signal.SIGTERM)
prefect.exceptions.TerminationSignal
rpc error: code = NotFound desc = an error occurred when try to find container "5742094a832ae44a72aef45eaccf8421c33577d21e0631ae96cdd58acbbe96bb": not found

What version of Prefect are you using?

pip freeze | grep prefect prefect==2.14.9 prefect-airbyte==0.2.0 prefect-aws==0.2.5 prefect-dask==0.2.6 prefect-dbt==0.3.1 prefect-github==0.1.5 prefect-shell==0.2.2 prefect-snowflake==0.26.1

What type of environment is your flow running in?

we are using k8s in AWS

The only thing that stands out to me here is that where you're calling

wait

, I usually see people use

gather

. I did some reading on the differences between them, and it seems like how they handle exceptions is a little different. Maybe try

gather

and see what happens? I'm not familiar with the content of the error itself.

I have done that too

are you using an agent or a worker? if agent, what infra block? if worker, what's your work pool type?

i am using an worker, k8s infra block

Hi Joish!

@Kevin Grismore @Bianca Hoch I've identified the underlying issue, and it turns out it wasn't related to Prefect. The root cause was due to Karpenter's node consolidation process. The pods were evicted because Karpenter de-provisioned the nodes. Thank you soo much!

Glad you found the issue, Joish!