Hello We had four deployments fail last night with the same

Question

Hello We had four deployments fail last night with the same error a few hours into their runs see below All four deployments were long running 2+ hours and had a large number of tasks 3 5k 3 4k 1 1k and 675 respectively These deployments were part of two different flows with two very different sets of code One of these flows is running Prefect 2 7 5 the other is running 2 6 7 Our agent is running 2 7 5 and we are using Prefect Cloud Any insights as to what may have happened and how to fix this issue `prefect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ` full traceback in thread

Ilya Galperin · Answer

```Encountered exception during execution Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 637 in orchestrate flow run result = await run sync flow call File usr local lib python3 10 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File opt prefect flow py line 160 in entrypoint results append future result File usr local lib python3 10 site packages prefect futures py line 226 in result return sync File usr local lib python3 10 site packages prefect utilities asyncutils py line 267 in sync return run async from worker thread async fn args kwargs File usr local lib python3 10 site packages prefect utilities asyncutils py line 177 in run async from worker thread return anyio from thread run call File usr local lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File usr local lib python3 10 site packages prefect futures py line 237 in result return await final state result raise on failure=raise on failure fetch=True File usr local lib python3 10 site packages prefect states py line 100 in get state result raise MissingResult prefect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ```

Kalise Richmond · Answer

Hi < Ilya Galperin> There was a in 2 7 5 that causes this error I recommend updating immediately to 2 7 6

Ilya Galperin · Answer

Hi < Kalise Richmond> this is only happening intermittently on flows and we are not using Prefect Orion using SQLite we are on Prefect Cloud adding clarification to my original message It doesn t seem like the bug linked to 2 7 5 is related

Zanie · Answer

This error occurs after the flow run finishes

Zanie · Answer

These deployments usually run without problems

Ilya Galperin · Answer

It looks like they were surfaced at the end of the flow runs One of flows flow a a ~2 5 hour one usually runs without problems and the other flow flow b with 3 deployments is expected to run gt 4 hours per deployment Flow b runs fine on other deployments using the same code when it has shorter runtimes we re re trying these long running deployments now

Ilya Galperin · Answer

For flow a though it is using dask and the flow will fail in the middle of the flow before it reaches its final set of tasks After all the dask tasks finish during which the error happened the flow re surfaces the error and doesn t trigger any remaining downstream tasks

Zanie · Answer

Can you share timestamps for the failures Are you using result persistence

Ilya Galperin · Answer

For flow b 11 06 00 PM 11 05 45 PM 11 06 50 PM Flow a 10 51 27

Ilya Galperin · Answer

I don t think we re persisting any results and we do not have retries enabled

Ilya Galperin · Answer

Unless result persistence is enabled somewhere by default that I m not sure of

Zanie · Answer

It s not often automatically enabled at the flow level

Zanie · Answer

What timezone are those

Ilya Galperin · Answer

Sorry these are PST from last night

Ilya Galperin · Answer

Jan 5th

Zanie · Answer

slightly smiling face thanks

Ilya Galperin · Answer

Happy to give you run IDs if that helps at all as well

Zanie · Answer

Yeah that d be helpful for identifiying if it s a server sie issue

Zanie · Answer

I suspect this is a client side bug though and it s going to be hard to find without a minimal example that reproduces the problem

Ilya Galperin · Answer

```bad5b6dd 8f0f 428c a045 e95cf72da4cd 82b0ca57 1501 47a2 91c8 ed621f184e82 646d0c66 320a 4435 9fb2 e0f4d03bb547 012ae57c 9296 46d0 af83 16eafc1218e0```

Zanie · Answer

Thanks I ll forward this to a Cloud engineer and we ll rule out server errors

Zanie · Answer

Can you explain more about what s going on here ``` File opt prefect flow py line 160 in entrypoint results append future result ```

Ilya Galperin · Answer

Sure this is doing something like ```futures = for e x in enumerate mylist futures append sometask with options name=task name submit x=x y=y z=z results = for future in futures results append future result ```

Ilya Galperin · Answer

So it is submitting a number of tasks to dask then waiting for them to complete

Ilya Galperin · Answer

Before moving to the next step

Ilya Galperin · Answer

This is only happening in flow B flow A is not using Dask the traceback might actually be slightly different let me check

Ilya Galperin · Answer

Here is the traceback for flow A different line in flow py ```Encountered exception during execution Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 637 in orchestrate flow run result = await run sync flow call File usr local lib python3 10 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File opt prefect flow py line 228 in entrypoint temp table created = create temp table File usr local lib python3 10 site packages prefect tasks py line 436 in call return enter task run engine File usr local lib python3 10 site packages prefect engine py line 927 in enter task run engine return run async from worker thread begin run File usr local lib python3 10 site packages prefect utilities asyncutils py line 177 in run async from worker thread return anyio from thread run call File usr local lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File usr local lib python3 10 site packages prefect engine py line 1068 in get task call return value return await future result File usr local lib python3 10 site packages prefect futures py line 237 in result return await final state result raise on failure=raise on failure fetch=True File usr local lib python3 10 site packages prefect states py line 100 in get state result raise MissingResult prefect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ```

Ilya Galperin · Answer

And another run in Flow A also a different line in flow py ```Encountered exception during execution Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 637 in orchestrate flow run result = await run sync flow call File usr local lib python3 10 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File opt prefect flow py line 279 in entrypoint cols = get table schema File usr local lib python3 10 site packages prefect tasks py line 436 in call return enter task run engine File usr local lib python3 10 site packages prefect engine py line 927 in enter task run engine return run async from worker thread begin run File usr local lib python3 10 site packages prefect utilities asyncutils py line 177 in run async from worker thread return anyio from thread run call File usr local lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File usr local lib python3 10 site packages prefect engine py line 1068 in get task call return value return await future result File usr local lib python3 10 site packages prefect futures py line 237 in result return await final state result raise on failure=raise on failure fetch=True File usr local lib python3 10 site packages prefect states py line 100 in get state result raise MissingResult prefect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ```

Zanie · Answer

Do you have in memory storage of task results disabled

Ilya Galperin · Answer

Hmm I am not sure how do I check that

Zanie · Answer

It d be ```cache result in memory=False```

Ilya Galperin · Answer

I wouldn t think so since we re definitely storing some results in memory between tasks

Zanie · Answer

Are you using cache keys

Ilya Galperin · Answer

We are not

Zanie · Answer

smile

Zanie · Answer

So I suspect the problem is that the results are being dropped from memory for some reason Perhaps on the way back from a Dask worker

Ilya Galperin · Answer

Maybe only one of the flows is using Dask though 3 of the failures are just running the SequentialTaskRunner

Ilya Galperin · Answer

Not sure if it s relevant but the Dask one has never failed with this error before last night when we had all of these deployments running simultaneously could it maybe have something to do with high load against the agent or API

Ilya Galperin · Answer

All of these deployments are running on different k8s pods as well so 4 pods total between these 4 runs

Zanie · Answer

Is that latest traceback from the sequential task runner

Ilya Galperin · Answer

Ya the first log was from the flow running DaskTaskRunner

Ilya Galperin · Answer

The later 2 are from the default task runner not defined in the flow which I believe is just sequential

Zanie · Answer

If you re using ` submit` the default is concurrent

Zanie · Answer

Or are you just calling the tasks

Ilya Galperin · Answer

The first one we have explicitly called dasktaskrunner

Ilya Galperin · Answer

the other flow doens t use submit

Ilya Galperin · Answer

it just calls them directly

Zanie · Answer

exploding head okay that s really weird

Zanie · Answer

Are your dependencies pinned or could a version of a package changed

Zanie · Answer

i e could pydantic have changed versions or something

Ilya Galperin · Answer

They are all pinned

Zanie · Answer

I m out of questions I think This is very peculiar

Zanie · Answer

If you can create a minimal example that reproduces it we can debug further but I m not sure what would cause this

Ilya Galperin · Answer

Hmm I can try but it might be tough This behavior only seems to be happening when we re running thousands of tasks simultaneously possibly across flows in a long running manner

Ilya Galperin · Answer

Does it sound like there were no issues with the API cloud services around this time last night

Zanie · Answer

We don t see anything suspicious

Zanie · Answer

Can you take a look at these three task runs

Zanie · Answer

```9bf44e45 bcc6 413b a67e 0ee8a1b06c77 at 02 06 30 633 EST 0424b8f8 06ff 4f78 808d 13543167632f at 02 06 00 680 EST 3ae5277e 61f1 4a10 9ff3 cf89af740aa7 at 02 05 45 138 EST```

Zanie · Answer

Are they in the affected flow runs What state transitions did they go through Are these the task runs your sequential flows crashed on

Ilya Galperin · Answer

Let me check

Ilya Galperin · Answer

Yep all of them are in the affected flow runs

Ilya Galperin · Answer

These are all the sequential flows and all the tasks are still in pending

Ilya Galperin · Answer

The actual error isn t surfaced on the task level though it s attached to the flow run

Ilya Galperin · Answer

But it seems like around the time the flow crashed yes

Ilya Galperin · Answer

Let me confirm but I think the tracebacks I posted above those were the tasks called in flow py at the time

Ilya Galperin · Answer

Yep confirming that

Zanie · Answer

Suspicious slightly smiling face

Zanie · Answer

Well if these are all in PENDING states that explains the error A pending state cannot have a result attached to it

Zanie · Answer

Are you passing data from upstream tasks into these

Ilya Galperin · Answer

Let me see all 3 of these are different tasks in the same flow wink

Ilya Galperin · Answer

It looks like 2 3 are The one with id `3ae5277e 61f1 4a10 9ff3 cf89af740aa7` only uses values that are passed in as arguments to the flow function

Zanie · Answer

Hm okay Are there logs for these tasks runs Presuming there aren t DEBUG level logs cry

Ilya Galperin · Answer

The other 2 are using values that are captured from a previous task then stored in a var in flow py

Ilya Galperin · Answer

Here s all the logs I see ```3ae5277e 61f1 4a10 9ff3 cf89af740aa7 Task run 3ae5277e 61f1 4a10 9ff3 cf89af740aa7 already finished 9bf44e45 bcc6 413b a67e 0ee8a1b06c77 Task run 9bf44e45 bcc6 413b a67e 0ee8a1b06c77 already finished 9bf44e45 bcc6 413b a67e 0ee8a1b06c77 Task run 9bf44e45 bcc6 413b a67e 0ee8a1b06c77 already finished ```

Ilya Galperin · Answer

All INFO level

Zanie · Answer

Ilya Galperin · Answer

Not sure if it matters but the Dask flow does not have any tasks still remaining in pending state or failed or crashed for that matter

Zanie · Answer

It looks like the server sent back an ABORT instruction here

Ilya Galperin · Answer

Ah interesting

Zanie · Answer

That generally means the server thought the task was already being executed elsewhere

Zanie · Answer

Unfortunately I m not sure we can recover the reason attached to the abort response

Ilya Galperin · Answer

Hmm

Zanie · Answer

I can add logs so we get more information in the future

Ilya Galperin · Answer

How do you recommend we proceed for now

Zanie · Answer

There s not much to do in the meantime You can use `return state=True` on the task call and just like call it again if you happen to get a pending state back but that s pretty hacky

Ilya Galperin · Answer

Hmm ya I don t know if that would solve the issue with the Dask flow though That one is already returning just futures

Ilya Galperin · Answer

And there is no task stuck in pending there shrug

Zanie · Answer

Yeah I m not sure what s going on there yet either I ll add more logs in the next release and then I d recommend running this with DEBUG logs

Ilya Galperin · Answer

Will do thanks MIchael

Zanie · Answer

I think once we get a better picture of what s going on it ll be an easy enough fix

Ilya Galperin · Answer

Do we need to do anything to enable another level of debug logs I thought cloud captures debug level logs too

Ilya Galperin · Answer

Or do we need to set some env var on the flow container

Zanie · Answer

`PREFECT LOGGING LEVEL=DEBUG` we only capture what you set the level to

Ilya Galperin · Answer

Our Prefect logger debug level logs are captured

Zanie · Answer

Oh okay yeah there just aren t debug level logs in this spot yet

Ilya Galperin · Answer

Gotcha

Ilya Galperin · Answer

Oh duh I set that env var in the docker image for these flows stuck out tongue

Ilya Galperin · Answer

Will keep up with the next release thanks again

Ilya Galperin · Answer

I ll let you know if we re able to reproduce

Ilya Galperin · Answer

Thanks again for all the help troubleshooting this

Zanie · Answer

Thank you too

Zanie · Answer

I m sure you ve caught a currently obscure bug that will become more common as people scale up I m eager to see what s going on here

Ilya Galperin · Answer

Me too slightly smiling face

Zanie · Answer

btw the next release will be next Thurdsay I m sure I ll have a PR up before then if you want to test beforehand

Zanie · Answer

You may want to look into getting up to the latest release in the meantime

Ilya Galperin · Answer

Noted thanks We ll upgrade ASAP

Ilya Galperin · Answer

A quick follow up < Michael> we just experienced the same error failure with the same 3 deployments from the Sequential task runner flow that failed last night so it does look like the issue is reproducible We re going to try staggering these runs and see if the problem persists

Zanie · Answer

Here s a patch with better logs <https github com PrefectHQ prefect pull 8097>

Ilya Galperin · Answer

Thank you We ll try to install this PR on the container running the flow and see what we catch

Zanie · Answer

Should be doable with the `EXTRA PIP PACKAGES` variable if you re using our image entrypoint e g ```docker run env EXTRA PIP PACKAGES= git+<https github com PrefectHQ prefect task run abort log> prefecthq prefect 2 latest prefect version ```

Ilya Galperin · Answer

Hi < Michael> an update from our end We ve upgraded to 2 7 7 and have done some more testing 1 The number of flows we have running in parallel at the time doesn t seem to matter we get the same crashing behavior when running just one instance of this flow 2 Flows are virtually always crashing at almost exactly the 4 hour mark 3 We installed the logging PR you created and here is the full output for the pending task ```Task run 8775d5ff 1614 4df3 83bd dc3b3fc4bb0f received abort during orchestration The enclosing flow must be running to begin task execution Task run is in PENDING state ```

Zanie · Answer

Ah interesting This task run failure might be misleading here If the flow run is marked as failed we won t let task runs move into a running state

Zanie · Answer

as that abort reason suggests

Zanie · Answer

So right now we have a missing result error which is caused by a pending task run which is caused by an aborted transition which is caused by a flow run that is not in a running state anymore

Zanie · Answer

Does the flow run itself have any relevant logs

Alix Cook · Answer

we re also seeing similar weirdness in some of our flows though we re seeing it in flows that only run for a couple seconds all the tasks in our flow complete successfully then we see the same errorr ```refect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ``` we re on prefect 2 7 1 we use prefect cloud we don t have any retries setup and we use whatever the default persistence is the docs say that it should be enabled The first time we saw this error was 2022 12 15 16 39 48 UTC

Alix Cook · Answer

we ve only noticed in the last couple days because we have this set of jobs that run monthly that we have failure notifications set up for

Alix Cook · Answer

its not impacting our data flows but our on call has to look at every failure to ensure that the job actually did what it was supposed to do which is a bit cumbersome when you re running thousands of these flows a day

Andrew Huang · Answer

I just encountered this here s a minimal example with prefect dask ```import dask dataframe import dask distributed from prefect import flow task from prefect dask import DaskTaskRunner get dask client client = dask distributed Client task def read data start str end str gt dask dataframe DataFrame df = dask datasets timeseries start end partition freq= 4w return df task def process data df gt dask dataframe DataFrame df yearly avg = df groupby df index year mean return df yearly avg compute flow task runner=DaskTaskRunner address=client scheduler address def dask flow df = read data submit 1988 2022 df yearly average = process data submit df return df yearly average dask flow ``` in this case it s because the memory maxed out on a single worker wrapping with get dask client in my case allows it to run successfully ``` task def process data df gt dask dataframe DataFrame with get dask client df yearly avg = df groupby df index year mean return df yearly avg compute ```

Zanie · Answer

< Alix Cook> are you using Dask as well < Andrew Huang> since we have reproductions without Dask I m hesitant to go down the Dask rabbit hole Especially OOM on a worker is always going to cause some sort of ungraceful shutdown

Alix Cook · Answer

no our flows are running as local processes

Ilya Galperin · Answer

Hi Michael no additional logging on the flow level unfortunately Id o notice the State Message attached to the flow is `Flow run infrastructure exited with non zero status code 1 ` Since this is almost always crashing at the 4 hour mark it makes me suspicious that there is some timeout we re missing However our other very long running flows are not timing out at any point I should also note the other flow where we saw this intermittent behavior on crashed at an arbitrary time after 2 hours so maybe not

Andrew Huang · Answer

oh I missed the reproduction

Zanie · Answer

< Ilya Galperin> that sounds like the failure is being reported by the agent then the flow is crashing since the agent has marked it as failed

Zanie · Answer

You re using Kubernetes jobs

Ilya Galperin · Answer

Yes

Zanie · Answer

The 1 status code is reported by Prefect in a few cases <https github com PrefectHQ prefect blob 2fe69ace54de28f9a47f2360bca43c9aeffbc6ef src prefect infrastructure kubernetes py L606> seems most notable

Zanie · Answer

< Alix Cook> you re using process infrastructure or just running flows without an agent These may be different issues

Alix Cook · Answer

we re running flows with an agent the agent is just running them in a local process

Zanie · Answer

Great thanks Your issue actually sounds distinct from Ilya s Can you open an issue with details like DEBUG level agent and flow run logs

Ilya Galperin · Answer

I m going to try the crash again and see if I can t find anything in the container s logs that might not be being shipped back

Alix Cook · Answer

sure

Ilya Galperin · Answer

Hi < Zanie> unfortunately no additional logs on the container or otherwise Not sure if it s relevant but the pod itself is not crashing the container spins down and the pod stays alive with a succeeded state Any other avenues you think might be worth digging into

Zanie · Answer

I m very confident this is a bug in the Kubernetes library that we will need to add a workaround for <https github com PrefectHQ prefect issues 8050 issuecomment 1379289312>

Ilya Galperin · Answer

Yes this looks really really similar I think you are probably right Thanks for flagging this Michael we will keep track of this issue