I m trying to understand why my flow keeps crashing partway

Question

I m trying to understand why my flow keeps crashing partway thorugh at a seemingly consistent spot I have a parent flow that runs the first subflow fine but half way through the second subflow the parent fails with the exception posted in the thread The UI still shows the second subflow as running This is on EKS by the way Locally things look to run until completion

Josh Paulin · Answer

```Encountered exception during execution Traceback most recent call last File usr local lib python3 9 site packages prefect engine py line 587 in orchestrate flow run result = await run sync flow call File usr local lib python3 9 site packages prefect utilities asyncutils py line 56 in run sync in worker thread return await anyio to thread run sync call cancellable=True File usr local lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File activity processor flow parent py line 43 in api activity processor process hourly api normalized process time dry run validate results File usr local lib python3 9 site packages prefect flows py line 384 in call return enter flow run engine from flow call File usr local lib python3 9 site packages prefect engine py line 160 in enter flow run engine from flow call return run async from worker thread begin run File usr local lib python3 9 site packages prefect utilities asyncutils py line 136 in run async from worker thread return anyio from thread run call File usr local lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File usr local lib python3 9 site packages prefect client py line 104 in with injected client return await fn args kwargs File usr local lib python3 9 site packages prefect engine py line 435 in create and begin subflow run flow run state data cache data await retrieve result flow run state File usr local lib python3 9 site packages prefect results py line 38 in retrieve result serialized result = await retrieve serialized result state data File usr local lib python3 9 site packages prefect client py line 104 in with injected client return await fn args kwargs File usr local lib python3 9 site packages prefect results py line 34 in retrieve serialized result return await filesystem read path result key File usr local lib python3 9 site packages prefect filesystems py line 149 in read path raise ValueError f Path path does not exist ValueError Path root prefect storage 4762715b441f4b3c8011b92dc1d5361f does not exist ```

Zanie · Answer

It looks like your subflow is attempting to retrieve its state from prior run instead of creating a new run

Zanie · Answer

At `create and begin subflow run` we can see that it attempts to do `flow run state data cache data await retrieve result flow run state ` this call is rehydrating the result on the state

Zanie · Answer

That means that we re in this case <https github com PrefectHQ prefect blob main src prefect engine py L430 L442>

Zanie · Answer

tldr it looks like the second subflow thinks it is the first one We ll need a minimal example to fix this

Josh Paulin · Answer

I think I see what might be causing some of this Looks like the pod runs out of memory

Josh Paulin · Answer

Is there any way to understand the memory profile of a flow It jumps as high as 60 GB for something that should be pretty benign confused

Zanie · Answer

Is it possible that all of your tasks together are consuming that much memory We do not release memory eagerly yet

Josh Paulin · Answer

As in when a task finishes the memory isn t released until the flow completes

Zanie · Answer

For tasks that are submitted as futures no We can t know if you ll need the result of the future downstream

Zanie · Answer

We might also hold onto the data for normal task calls to as we track the state of all task runs created in a flow With upcoming work on result handling we can optimize that though

Josh Paulin · Answer

Things get better if I don t submit any tasks as futures but we re still getting up to ~37GB I also don t see any real drop in memory between when each subflow finishes blue lines are where a new subflow starts

Zanie · Answer

Are you returning a value from your subflow If not we default to a state that bundles all the task states from within the subflow

Josh Paulin · Answer

No returns so sounds like I should

Zanie · Answer

Yeah that should help since it ll release all those tasks on completion

Josh Paulin · Answer

That definitely works to clear out some resources between the subflows The biggest hog definitely looks like the futures Is there any upcoming work to optimize that

Zanie · Answer

Yeah there is I ll be working on result handling over the next month