<@ULVA73B9P> I'm using Prefect 3, the function `ge...
# ask-marvin
n
@Marvin I'm using Prefect 3, the function
get_root_flow_run_id
. For some reason, using this function results in a 500 Internal Server error from our self hosted Prefect server. Why?
Copy code
File "/usr/local/lib/python3.10/dist-packages/prefect/runtime/flow_run.py", line 308, in get_root_flow_run_id
    root_flow_run_id = _get_root_flow_run_id(parent_flow_run_id)
  File "/usr/local/lib/python3.10/dist-packages/prefect/runtime/flow_run.py", line 298, in _get_root_flow_run_id
    ).result()
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 329, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 402, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/dist-packages/prefect/runtime/flow_run.py", line 117, in _get_flow_run
    return await client.read_flow_run(flow_run_id)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in read_flow_run
    response = await self.request(
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/orchestration/base.py", line 53, in request
    return await self._client.send(request)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 354, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 162, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://prefect-server.prefect.svc.cluster.local:4200/api/flow_runs/80b194d4-2c2a-44cb-8dc3-69305b5e2411>'


Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>
m
thought for 298.7 seconds
Short answer: a 500 here is coming from your self‑hosted API when
get_root_flow_run_id()
recursively calls the server to read parent flow/task runs. The most common causes are a server-side schema/migration mismatch or a data integrity issue that makes the server fail while serializing a flow run response. Why this function triggers it - In Prefect 3.x,
prefect.runtime.flow_run.get_root_flow_run_id()
walks up the subflow chain by: - reading the parent flow run by ID - if that flow run has a
parent_task_run_id
, reading that task run, then reading its flow run, etc. - Each step hits
/api/flow_runs/{id}
(and sometimes
/api/task_runs/{id}
) on your server. - If your server cannot serialize/validate one of these ORM objects (e.g., due to schema drift, missing related records, or unmigrated DB changes), you’ll get a 500. Most likely root causes - Client/server version mismatch or unmigrated DB - The server’s response models are strict (Pydantic v2 with from_attributes=True). If the DB schema and server code aren’t aligned, serialization can fail and bubble up as a 500. - This often happens right after upgrading the Prefect image but not running DB migrations. - Orphaned/broken relationships - Example: a flow run references a work queue/work pool or parent task run that was deleted or otherwise inconsistent; the server attempts to include related objects and fails during serialization. How to confirm and fix 1) Check versions and migrations - On your client:
Copy code
prefect version
- On your server (Kubernetes: check the image tag or logs on the API pod). - Run server DB migrations (we verified this command exists):
Copy code
prefect server database upgrade
If you deploy via Helm, ensure the migration job has run successfully. 2) Check server logs for the real exception - The 500 hides the actual cause. Look at your API server logs: - Kubernetes example:
Copy code
kubectl logs deploy/prefect-server -n <namespace>
- You’re looking for a Pydantic validation error, AttributeError on a missing field, or relationship load error. 3) Reproduce with a direct call to isolate the failing record - Try the exact failing ID directly:
Copy code
curl -sS <http://prefect-server.prefect.svc.cluster.local:4200/api/flow_runs/80b194d4-2c2a-44cb-8dc3-69305b5e2411>
If this 500s, it’s not
get_root_flow_run_id
specifically—it’s the server failing to return that flow run. - Or run this snippet to mimic what the runtime does:
Copy code
import asyncio
  from prefect.client.orchestration import get_client

  async def main():
      async with get_client() as client:
          fr = await client.read_flow_run("80b194d4-2c2a-44cb-8dc3-69305b5e2411")
          print(fr.id, fr.parent_task_run_id)

  asyncio.run(main())
4) Check for data integrity issues - If it only fails for some runs, look for: - Deleted or missing work pools/queues referenced by those runs - A parent_task_run_id that points to a missing task run - Any recent manual DB changes - Cleaning up orphaned records (or upgrading to a server version that tolerates them) typically resolves it. 5) Temporary workaround in your flow code - If you only need a best-effort root ID (e.g., for logging), guard the call:
Copy code
from prefect.runtime.flow_run import get_root_flow_run_id, get_id

  try:
      root_id = get_root_flow_run_id()
  except Exception:
      # fallback if the server can't read a parent run
      root_id = get_id()
Helpful commands - We verified these CLI namespaces: - Flow runs:
Copy code
prefect flow-run --help
- Database migrations:
Copy code
prefect server database upgrade --help
Next steps / info I need from you - What Prefect version is your client using (
prefect version
)? - What image/tag is your self-hosted API running? - Did you run
prefect server database upgrade
after your last upgrade? - Does a direct GET to
/api/flow_runs/80b1...2411
also 500?
- Any relevant stack trace from the server logs? Docs - Runtime flow run API (includes
get_root_flow_run_id
): https://docs.prefect.io/latest/api-ref/runtime/#prefect.runtime.flow_run.get_root_flow_run_id - CLI reference: https://docs.prefect.io/latest/cli/