https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> why my prefect jobs hanging on "submi...
# marvin-ai
c
@Marvin why my prefect jobs hanging on "submitting" in the UI: Got this error from the worker:
Copy code
{
  "type": "PydanticSerializationError",
  "message": "Unable to serialize unknown type: <class 'uuid.UUID'>",
  "traceback": "  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/engine.py\", line 87, in handle_engine_signals\n    yield\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/engine.py\", line 178, in <module>\n    _run_result: object = run_flow(\n                          ^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/flow_engine.py\", line 2193, in run_flow\n    ret_val = run_flow_sync(**kwargs)\n              ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/flow_engine.py\", line 2033, in run_flow_sync\n    with engine.start():\n         ^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/contextlib.py\", line 137, in __enter__\n    return next(self.gen)\n           ^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/flow_engine.py\", line 1269, in start\n    self.begin_run()\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/flow_engine.py\", line 729, in begin_run\n    state = self.set_state(new_state)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/flow_engine.py\", line 741, in set_state\n    state = propose_state_sync(\n            ^^^^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/utilities/engine/__init__.py\", line 640, in propose_state_sync\n    response = set_state_and_handle_waits(set_state)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/utilities/engine/__init__.py\", line 626, in set_state_and_handle_waits\n    response = set_state_func()\n               ^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py\", line 403, in set_flow_run_state\n    state=state_create.model_dump(mode=\"json\", serialize_as_any=True),\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/category/prefect/_state_serialization_capture.py\", line 69, in model_dump\n    return original(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/prefect/src/.venv/lib/python3.12/site-packages/pydantic/main.py\", line 463, in model_dump\n    return self.__pydantic_serializer__.to_python(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n"
}
m
✅ thought for 218.8 seconds
This is most likely a Prefect bug, not your flow code. What’s happening - The worker is trying to move the flow run into a new state (
Submitting
) - During that state update, Prefect/Pydantic fails to JSON-serialize a 
UUID
- Because the state transition fails, the run can appear stuck on Submitting in the UI The key part of your traceback is this:
Copy code
state=state_create.model_dump(mode="json", serialize_as_any=True)
and the error:
Copy code
Unable to serialize unknown type: <class 'uuid.UUID'>
Most likely cause This matches a known Prefect 3.x serialization issue affecting some 3.6.x versions. The fix was added in Prefect 3.6.10+. What to do first 1. Check your Prefect version:
Copy code
bash
prefect version
2. If you're on an older 3.6.x release, upgrade:
Copy code
bash
pip install -U "prefect>=3.6.10"
If you can, I'd recommend upgrading to the latest 3.x rather than stopping at 3.6.10. Temporary workaround if you cannot upgrade immediately Add this before your worker/flow process starts:
Copy code
python
from prefect.client.schemas.objects import StateDetails

StateDetails.model_rebuild()
This forces Pydantic to build the serializer up front and can avoid the UUID serialization failure. Why the UI says “Submitting” The worker has picked up the run, but it crashes while reporting the state transition back to the Prefect API. So the orchestration state in UI never advances correctly. Helpful links - Prefect 3.6.10 release notes - Related GitHub issue If upgrading does not fix it Please share: - output of 
prefect version
- worker type you’re using - how you launch the worker - whether this happens for all flows or only one specific deployment Then I can help narrow down whether this is version-related or something specific in your environment.
c
we are on 3.7.5 though?
n
hi @Colton Flowers
"message":"Unable to serialize unknown type: <class 'uuid.UUID'>"
i will take a look at this!
c
Hey @Nate Thank you! Some context that might be useful: We no longer see this problem after we stopped running prefect with 
ddtrace-run
, i.e. replaced 
ddtrace-run prefect flow-run execute
with 
prefect flow-run execute
.
n
aha
that is indeed useful context. they do some horrifying monkeypatching iirc so i suspect that is very related. i think we even had some trouble with that internally a while ago
ill briefly look into whether there's any unfortunate situations we can avoid, but glad you worked around it for the time being
leaving a breadcrumb here, feel free to add any useful details in comments https://github.com/PrefectHQ/prefect/discussions/22381
🙌 1
c
Ah, makes sense. Appreciate you looking into it.
4 Views
Open in Slack
PreviousNext
Powered by