Seeing a strange error when scaling up a prefect w...
# ask-communityj
Jan Domanski
05/10/2022, 8:25 AMSeeing a strange error when scaling up a prefect workflow
Copy code
File "/venv/lib/python3.8/site-packages/prefect/orion/orchestration/rules.py", line 534, in __aexit__
await self.after_transition(*exit_context)
File "/venv/lib/python3.8/site-packages/prefect/orion/database/dependencies.py", line 112, in async_wrapper
return await fn(*args, **kwargs)
File "/venv/lib/python3.8/site-packages/prefect/orion/orchestration/core_policy.py", line 190, in after_transition
cache_key = validated_state.state_details.cache_key
AttributeError: 'NoneType' object has no attribute 'state_details'k
Kevin Kho
05/10/2022, 10:59 AM
Did you try adding a cache key to a task?
a
Anna Geller
05/10/2022, 11:16 AM
@Jan Domanski can you share an example code so that I can use to reproduce the issue? hard to say anything about the root cause without seeing the flow code
Anna Geller
05/10/2022, 11:16 AM
at a first glance, it looks like some misconfiguration in the cache key function
j
Jan Domanski
05/10/2022, 12:00 PMYeah, i’m not actually using a cache function or anything sophisticated like this
Jan Domanski
05/10/2022, 12:01 PMThis is an ongoing issue we’ve been seeing with a RayTaskExecutor
Jan Domanski
05/10/2022, 12:01 PMdoesn’t happen for a simple for loop and a prefect task, but hits us in the face for anything non-trivial – will try to share a repo (including an AWS ray.yaml, which seems needed)
👍 1
z
Zanie
05/10/2022, 2:58 PM
cc @Dustin Ngo we may have missed a race condition in the orchestration policy?
🙏 1
d
Dustin Ngo
05/10/2022, 4:40 PMThanks for the ping--while doing some testing yesterday I saw this error for the first time recently too, I'll look into this
j
Jan Domanski
05/22/2022, 5:37 PMAny luck with that @Dustin Ngo ? the bug is still happening for us when slamming Prefect hard
k
Kevin Kho
05/22/2022, 5:39 PM
Will ping Dustin tom to talk about this and let you know
✅ 1
z
Zanie
05/22/2022, 6:22 PM
We haven't found it yet. We're looking into it though.
🙏 1
4 Views