https://prefect.io logo
Title
a

Ankit

03/21/2023, 6:54 AM
Hi everyone, Last night we upgraded prefect version to 2.8.4 We keep getting these errors in prefect
Flow run infrastructure exited with non-zero status code -1.
But when I check the logs of kubernetes pod, tasks keep running. Sometime even the subsequent tasks are triggered but at time it fails and kills the pod. Anyone knows what can it be? Thanks.
👀 1
r

redsquare

03/21/2023, 7:01 AM
can you paste the full error stack trace
a

Ankit

03/21/2023, 7:28 AM
this is what I am getting, above shared was state message
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 665, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 156, in run_sync_in_interruptible_worker_thread
    tg.start_soon(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 135, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "/app/feanor/ecommerce_data_lake/forward/ecom_sales_mis_forward_flow.py", line 51, in ecom_sales_mis_forward
    end_time = get_time()
  File "/usr/local/lib/python3.8/dist-packages/prefect/tasks.py", line 469, in __call__
    return enter_task_run_engine(
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 965, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.8/dist-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 1114, in get_task_call_return_value
    return await future._result()
  File "/usr/local/lib/python3.8/dist-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/usr/local/lib/python3.8/dist-packages/prefect/states.py", line 103, in _get_state_result
    raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
subflows ran but flow got into Crashed state, last subflow ran after parent flow was in Crashed state and that's when the above error got raised
r

redsquare

03/21/2023, 7:38 AM
are they long running?
a

Ankit

03/21/2023, 7:38 AM
yes, takes about 10 mins to complete
r

redsquare

03/21/2023, 7:42 AM
10 mins could be the issue
a

Ankit

03/21/2023, 7:43 AM
flows which takes over 30 mins were also working fine with 2.7.12, probably some new settings
r

redsquare

03/21/2023, 7:47 AM
yeah, are you using a k8s infra block
a

Ankit

03/21/2023, 7:47 AM
yes
r

redsquare

03/21/2023, 7:47 AM
if so maybe try recreating it
they added a bunch of stuff from late 2.7 to now to do with timeouts etc
a

Ankit

03/21/2023, 7:51 AM
thanks @redsquare, I will try recreating the block.
👍 1