https://prefect.io
Join Slack
Seeing a strange error when scaling up a prefect workflow ```File "/venv/lib/python3.8/site-packages...
# ask-community
j
Seeing a strange error when scaling up a prefect workflow
Copy code
File "/venv/lib/python3.8/site-packages/prefect/orion/orchestration/rules.py", line 534, in __aexit__
    await self.after_transition(*exit_context)
  File "/venv/lib/python3.8/site-packages/prefect/orion/database/dependencies.py", line 112, in async_wrapper
    return await fn(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/prefect/orion/orchestration/core_policy.py", line 190, in after_transition
    cache_key = validated_state.state_details.cache_key
AttributeError: 'NoneType' object has no attribute 'state_details'
Any idea how to debug/interpret this?
k
Did you try adding a cache key to a task?
a
@Jan Domanski can you share an example code so that I can use to reproduce the issue? hard to say anything about the root cause without seeing the flow code
at a first glance, it looks like some misconfiguration in the cache key function
j
Yeah, i’m not actually using a cache function or anything sophisticated like this
This is an ongoing issue we’ve been seeing with a RayTaskExecutor
doesn’t happen for a simple for loop and a prefect task, but hits us in the face for anything non-trivial – will try to share a repo (including an AWS ray.yaml, which seems needed)
👍 1
z
cc @Dustin Ngo we may have missed a race condition in the orchestration policy?
🙏 1
d
Thanks for the ping--while doing some testing yesterday I saw this error for the first time recently too, I'll look into this
j
Any luck with that @Dustin Ngo ? the bug is still happening for us when slamming Prefect hard
k
Will ping Dustin tom to talk about this and let you know
✅ 1
z
We haven't found it yet. We're looking into it though.
🙏 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by