Hi everyone, Last night we upgraded prefect version to 2.8.4 We keep getting these errors in prefect

Flow run infrastructure exited with non-zero status code -1.

But when I check the logs of kubernetes pod, tasks keep running. Sometime even the subsequent tasks are triggered but at time it fails and kills the pod. Anyone knows what can it be? Thanks.

can you paste the full error stack trace

this is what I am getting, above shared was state message

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 665, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 156, in run_sync_in_interruptible_worker_thread
    tg.start_soon(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 135, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "/app/feanor/ecommerce_data_lake/forward/ecom_sales_mis_forward_flow.py", line 51, in ecom_sales_mis_forward
    end_time = get_time()
  File "/usr/local/lib/python3.8/dist-packages/prefect/tasks.py", line 469, in __call__
    return enter_task_run_engine(
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 965, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.8/dist-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 1114, in get_task_call_return_value
    return await future._result()
  File "/usr/local/lib/python3.8/dist-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/usr/local/lib/python3.8/dist-packages/prefect/states.py", line 103, in _get_state_result
    raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.

are they long running?

yes, takes about 10 mins to complete

10 mins could be the issue

flows which takes over 30 mins were also working fine with 2.7.12, probably some new settings

yeah, are you using a k8s infra block

yes

if so maybe try recreating it

thanks @redsquare, I will try recreating the block.