Hello we are experiencing intermittent errors on flows that

Question

Hello we are experiencing intermittent errors on flows that are submitting a large number of tasks to DaskTaskRunner No individual tasks seem to fail but the flow will periodically err out with the following message full traceback in thread ```Flow run encountered an exception MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ``` We are on Prefect version 2 8 2 but this has been occurring for a few versions now Other users seem to have experienced this behavior as well see thread <https prefect community slack com archives CL09KU1K7 p1675390311559929|here> Is this being worked on and if so where can we track it

Ilya Galperin · Answer

```Encountered exception during execution Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 651 in orchestrate flow run result = await run sync flow call File usr local lib python3 10 site packages prefect utilities asyncutils py line 154 in run sync in interruptible worker thread async with anyio create task group as tg File usr local lib python3 10 site packages anyio backends asyncio py line 662 in aexit raise exceptions 0 File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File usr local lib python3 10 site packages prefect utilities asyncutils py line 135 in capture worker thread and result result = fn args kwargs File opt prefect flow py line 128 in entrypoint if future result is not None File usr local lib python3 10 site packages prefect futures py line 226 in result return sync File usr local lib python3 10 site packages prefect utilities asyncutils py line 267 in sync return run async from worker thread async fn args kwargs File usr local lib python3 10 site packages prefect utilities asyncutils py line 177 in run async from worker thread return anyio from thread run call File usr local lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File usr local lib python3 10 site packages prefect futures py line 237 in result return await final state result raise on failure=raise on failure fetch=True File usr local lib python3 10 site packages prefect states py line 101 in get state result raise MissingResult prefect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ```

Zanie · Answer

Generally this is because of an upstream problem

Zanie · Answer

Do you have additional logs for the flow run

Zanie · Answer

<https github com PrefectHQ prefect issues 8228 issuecomment 1448366073> is likely helpful

Ilya Galperin · Answer

Thank you Zaine I looked over this Github issue and we are having similar behavior as describes The flow fails after a short amount of time in the most recent instance lt 5 minutes I ve attached the logs from the pod around the time of the crash

Zanie · Answer

What is your kubernetes job timeout set to

Ilya Galperin · Answer

180 secs

Zanie · Answer

If the timeout is 180s it ll fail at 180s

Zanie · Answer

So if it runs longer i e approaching 5 minutes it ll be marked as failed by the agent then exit when the next task runs

Ilya Galperin · Answer

Sorry I m a little confused what would need to run longer than gt 180 seconds for this to crash This flow typically runs gt 2 hours and does not have this issue this does seem to be intermittent

Ilya Galperin · Answer

Happy to set our job watch timeout to None but I m not sure how this would address the issue

Zanie · Answer

If your job watch timeout is less than the runtime of the flow the flow should fail

Zanie · Answer

If the issue is intermittent it is likely because we were not enforcing job watch timeouts correctly until 2 8 3

Zanie · Answer

It s not clear to me if your issue is something else yet the logs did not seem to include the full run

Ilya Galperin · Answer

I ll try to set our timeout to None then thank you I was under the impression `job watch timeout seconds` listens for events and will timeout if no events are received in the span passed here has this changed I can definitely provide a longer log if that would be helpful here but they might contain some semi sensitive output is there somewhere I can forward them privately through DM