Hi all I have a pipeline flow that is working perfectly fine

Question

Hi all I have a pipeline flow that is working perfectly fine on one cluster but when running it on another I am seeing these errors thrown consistently There are no errors related to the actual work and I am executing this workflow against a known dataset that is weel understood so I am sure the problem is not there Any ideas It is almost like a task ID can not be pulled from the database or something I am running a postgres data and orion server in a VM near said clusters ```Encountered exception during execution Traceback most recent call last File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 610 in orchestrate flow run result = await run sync flow call File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect utilities asyncutils py line 68 in run sync in worker thread return await anyio to thread run sync call cancellable=True File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 bin process holography py line 431 in holography flow return me = log and wait futures return me File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect tasks py line 360 in call return enter task run engine File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 733 in enter task run engine return run async from worker thread begin run File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect utilities asyncutils py line 148 in run async from worker thread return anyio from thread run call File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 concurrent futures base py line 446 in result return self get result File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 concurrent futures base py line 391 in get result raise self exception File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 874 in get task call return value return await future result File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect futures py line 237 in result return await final state result raise on failure=raise on failure fetch=True File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect states py line 86 in get state result raise MissingResult prefect exceptions MissingResult State data is missing Typically this occurs when result persistence is disabled and the state has been retrieved from the API ```

Mathijs Carlu · Answer

I came across this error after an earlier task failed with the error I posted in <https prefect community slack com archives CL09KU1K7 p1667553923158809|this thread> Because the earlier task can t report on its state the result is missing in a task that needs it later If you re experiencing the same you can track the issue also in thread Otherwise do you use result persistence on your task runs And if so where are you storing them

Tim Galvin · Answer

Interesting So I am mapping a set of values to a single task in a for loop similar to your example ``` return me = task grid holography submit beam sbid workdir grid ant return me=return me for beam ms in enumerate mslist holo ``` In the above `return me` is a dummy variable with no meaning that I am only passing around so that prefect can track dependencies I have recently learnt of the `wait for` argument But I am unclear if it is the same issue I am not seeing any earlier tasks failing It all looks exactly like I expect But perhaps if I up the logging level someone might stick out Just so I understand the thinking from your thread is that task C relies on task B Task B fails and does no update the orion service so when Task C attempts and assesses its dependencies it is not able to find a state for Task B Bam boom we have an error Is that correct When you say result persistence on task runs you mean whether I am caching the results from tasks that have previously been called with the same sets of inputs If so I am not knowingly using this I could delete my prefect home and dask work space just to be sure though

Tim Galvin · Answer

The other thought I just had was a concurrency limit at the database I have 20 odd tasks currning concurrently in this loop I am curious what happens if they all hit the database at about the same time at the same time as the previous task that they all rely upon

Mathijs Carlu · Answer

Your understanding is correct Upping the log level would not help i m assuming you always log error level and above The task that fails to report its state always errors By result persistence I don t mean caching which is what you re talking about By default task results are saved in memory However when the kubernetes job is running out of memory it might cause problems Could you try setting the cache result in memory to false and the persist result to true in your task decorator

Tim Galvin · Answer

Thanks for that Let me try out your second suggestion I am reasonably certain no tasks are failing

Tim Galvin · Answer

I am testing this setup out on a new SLURM cluster so I could very well have botched the `dask jobqueue SLURMCluster` specification

Tim Galvin · Answer

More strangness I dropped back the number on activate dask workers and nodes to smaller numbers The error transformed into ```07 23 30 229 | ERROR | Flow run natural dogfish Finished in state Failed Flow run encountered an exception ValueError Failed state result was an iterable of states but none were failed n Traceback most recent call last File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 bin process holography py line 530 in lt module gt sys exit cli File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 bin process holography py line 526 in cli main args workdir args sbid grid ant=args ant cluster=args cluster File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect flows py line 442 in call return enter flow run engine from flow call File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 157 in enter flow run engine from flow call return anyio run begin run File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 asyncio runners py line 44 in run return loop run until complete main File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 asyncio base events py line 647 in run until complete return future result File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 287 in wrapper return await func args File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect client utilities py line 47 in with injected client return await fn args kwargs File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 237 in create then begin flow run return await state result fetch=True File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect states py line 74 in get state result raise await get state exception state File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 610 in orchestrate flow run result = await run sync flow call File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect utilities asyncutils py line 68 in run sync in worker thread return await anyio to thread run sync call cancellable=True File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 bin process holography py line 461 in main holography flow File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect flows py line 442 in call return enter flow run engine from flow call File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 161 in enter flow run engine from flow call return run async from worker thread begin run File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect utilities asyncutils py line 148 in run async from worker thread return anyio from thread run call File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 concurrent futures base py line 446 in result return self get result File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 concurrent futures base py line 391 in get result raise self exception File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect client utilities py line 47 in with injected client return await fn args kwargs File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect engine py line 520 in create and begin subflow run return await terminal state result fetch=True File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect states py line 74 in get state result raise await get state exception state File software projects askaprt tgalvin setonix miniconda3 envs acesprefect2 lib python3 9 site packages prefect states py line 318 in get state exception raise ValueError ValueError Failed state result was an iterable of states but none were failed ``` Everything is running and looks to be doing the right thing I am seeing in my slurm logs warnings from the dask nanny about using more than 80 of memory I will have to take a closer look at these slrum logs to make more sense of it

Mathijs Carlu · Answer

That s weird indeed can t give you any pointers on that one Actually may I ask why you are using submit in a list comprehension rather than map

Tim Galvin · Answer

As I understand the ` map ` the function being mapped needs to have that `async def` declaration That is basically it I am that petty slightly smiling face

Mathijs Carlu · Answer

I don t think it has to though The <https docs prefect io concepts tasks h=map map|most basic example in the docs> seems to say it can just be a sync function

Tim Galvin · Answer

Well look at that You are right <https docs prefect io concepts tasks h=map map> Where ever did I get that loced into my head I swear I was reading it somewhere

Tim Galvin · Answer

Now I look silly Even sillier I seemingly have it working I made no changes to the actual pipeline code The only thing I changed was my `dask jobqueue SLURMCluster` resource requirements