https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jan Domanski

05/10/2022, 8:25 AM
Seeing a strange error when scaling up a prefect workflow
Copy code
File "/venv/lib/python3.8/site-packages/prefect/orion/orchestration/rules.py", line 534, in __aexit__
    await self.after_transition(*exit_context)
  File "/venv/lib/python3.8/site-packages/prefect/orion/database/dependencies.py", line 112, in async_wrapper
    return await fn(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/prefect/orion/orchestration/core_policy.py", line 190, in after_transition
    cache_key = validated_state.state_details.cache_key
AttributeError: 'NoneType' object has no attribute 'state_details'
Any idea how to debug/interpret this?
k

Kevin Kho

05/10/2022, 10:59 AM
Did you try adding a cache key to a task?
a

Anna Geller

05/10/2022, 11:16 AM
@Jan Domanski can you share an example code so that I can use to reproduce the issue? hard to say anything about the root cause without seeing the flow code
at a first glance, it looks like some misconfiguration in the cache key function
j

Jan Domanski

05/10/2022, 12:00 PM
Yeah, i’m not actually using a cache function or anything sophisticated like this
This is an ongoing issue we’ve been seeing with a RayTaskExecutor
doesn’t happen for a simple for loop and a prefect task, but hits us in the face for anything non-trivial – will try to share a repo (including an AWS ray.yaml, which seems needed)
👍 1
z

Zanie

05/10/2022, 2:58 PM
cc @Dustin Ngo we may have missed a race condition in the orchestration policy?
🙏 1
d

Dustin Ngo

05/10/2022, 4:40 PM
Thanks for the ping--while doing some testing yesterday I saw this error for the first time recently too, I'll look into this
j

Jan Domanski

05/22/2022, 5:37 PM
Any luck with that @Dustin Ngo ? the bug is still happening for us when slamming Prefect hard
k

Kevin Kho

05/22/2022, 5:39 PM
Will ping Dustin tom to talk about this and let you know
1
z

Zanie

05/22/2022, 6:22 PM
We haven't found it yet. We're looking into it though.
🙏 1
4 Views