https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> this is the full traceback I get when...
# ask-marvin
g
@Marvin this is the full traceback I get when I try to access my results using 
await step.state.result()
where 
step
is just a flow_run. Please note that this code has not change in months and was working perfectly fine until today, what could it be?
Copy code
Traceback (most recent call last):
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/_result_records.py", line 199, in deserialize
    instance = cls.model_validate_json(data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 746, in model_validate_json
    return cls.__pydantic_validator__.validate_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ResultRecord
  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value=b'gAWV6QAAAAAAAAB9lIwHY29...yaXZhdGVfX5ROdWJzLg==\n', input_type=bytes]
    For further information visit <https://errors.pydantic.dev/2.11/v/json_invalid>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/client/schemas/objects.py", line 241, in aresult
    return await get_state_result(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
    return await _get_state_result(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/states.py", line 160, in _get_state_result
    result = await _get_state_result_data_with_retries(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/states.py", line 117, in _get_state_result_data_with_retries
    record = await ResultStore._from_metadata(state.data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/results.py", line 392, in _from_metadata
    result = await store.aread(metadata.storage_key)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/results.py", line 676, in aread
    return await self._read(key=key, holder=holder, _sync=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/results.py", line 633, in _read
    result_record: ResultRecord[Any] = ResultRecord.deserialize(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/_result_records.py", line 204, in deserialize
    result = backup_serializer.loads(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gabor/workspace/sweap/lead-companion-backend/.venv/lib/python3.12/site-packages/prefect/serializers.py", line 149, in loads
    return pickler.loads(base64.decodebytes(blob))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'src'
m
✅ thought for 290.6 seconds
Thanks for the detailed traceback — that helps. What’s happening - Prefect is trying to load the flow’s return value from your result store. - It first attempts JSON and fails (that “Invalid JSON” message). - It then falls back to the backup serializer (Pickle via cloudpickle), which fails with ModuleNotFoundError: No module named 'src'. That last error means the value that was persisted contains an object that references a module named 
src
(e.g., something like 
src.my_module.MyClass
). When Prefect unpickles the result, Python needs to import that module to reconstruct the object. If the environment where you’re calling 
await step.state.result()
does not have 
src
on 
PYTHONPATH
(or the package installed properly), unpickling will fail. Why it worked before and broke “today” - Most often this is environment drift: - The place you’re calling 
.result()
from changed (new shell, container, or machine) and no longer has 
PYTHONPATH
including your repo root or 
src
. - Your worker or local venv was rebuilt without installing your project package (e.g., missing 
pip install -e .
). - Changes to how you launch code (e.g., different working directory) removed 
src
from the import path. - Nothing in your code needs to have changed for this to happen; a subtle environment/path change is enough. Ways to fix/unblock 1) Make the 
src
package importable wherever you call `.result()`: - Preferable: install your project as a package so imports don’t depend on 
src
being on the path.
Copy code
pip install -e .
and import using your package name (not 
src...
). - Or temporarily add it to the path when running:
Copy code
export PYTHONPATH=$PWD
     # or if your code is under ./src, export PYTHONPATH=$PWD/src
Or in Python:
Copy code
import sys, os
     sys.path.append(os.path.abspath("src"))  # or repo root depending on your layout
2) Future-proof: avoid persisting pickled objects - If your results are JSON-serializable (dicts, lists, strings, numbers), switch to JSON for results. This avoids cross-environment import issues entirely. - Set at the flow level:
Copy code
from prefect import flow
     from prefect.serializers import JSONSerializer

     @flow(result_serializer=JSONSerializer())
     def my_flow():
         return {"ok": True}
- Or set globally:
Copy code
export PREFECT_RESULTS_DEFAULT_SERIALIZER=json
- If you need to persist non-JSON objects (e.g., DataFrames), consider writing them to storage (parquet/csv) and returning the URI/path instead of the object itself. 3) If you must stick with Pickle - Ensure any returned classes are defined in a proper, importable module that is installed in every environment that reads the results. - Avoid returning objects from modules only reachable via ad-hoc 
PYTHONPATH=src
hacks. Notes - Existing runs that already persisted pickled results will still require the same import path to be available to deserialize. Switching to JSON will protect future runs, not retroactively fix old results. - The initial “Invalid JSON” message is expected — Prefect tries JSON first, then falls back to the configured/backup serializer. Helpful references - Prefect serializers API: Serializers API - Results overview: Results in Prefect - Source code for serializers: prefect/serializers.py A few quick questions to pinpoint the exact cause for you - Where are you calling 
await step.state.result()
from (local dev shell, notebook, another service), and is that different from before? - Do you import your own code via 
src.*
anywhere or rely on 
PYTHONPATH=src
? What’s your project layout? - Did you recently rebuild a container/venv or upgrade Python/Prefect?
If you share those details, I can suggest the smallest change to get you unblocked and set you up to avoid this going forward.
g
@Marvin So I am sure the issue is because in my prefect flows, I was returning pydantic models, but doesn't prefect have some kind of built-in elegant way to automatically dump and seralize pydantic models into dictionaries?
m
✅ thought for 62.9 seconds
Short answer: returning Pydantic models is what’s causing the pickle round-trip and the “No module named 'src'” error. Prefect can handle this cleanly if you use JSON results and (a) return dicts or (b) provide a small encoder that converts Pydantic models to dicts automatically. What works well - Switch to JSON results and return primitives:
Copy code
from prefect import flow
  from prefect.serializers import JSONSerializer

  @flow(result_serializer=JSONSerializer())
  def my_flow():
      model = MyModel(...)
      return model.model_dump()  # Pydantic v2
      # or: return model.dict()  # Pydantic v1
Or set globally:
Copy code
export PREFECT_RESULTS_DEFAULT_SERIALIZER=json
If you want it automatic for Pydantic models - Use a custom JSON encoder that converts BaseModel to dict, then defers to Prefect’s default for everything else. 1) Create an encoder in your project (important: make it importable):
Copy code
# my_project/serialization.py
     from pydantic import BaseModel
     from prefect.serializers import prefect_json_object_encoder

     def pydantic_first_encoder(obj):
         if isinstance(obj, BaseModel):
             dump = getattr(obj, "model_dump", None)  # v2
             return dump() if dump else obj.dict()    # v1 fallback
         return prefect_json_object_encoder(obj)
2) Use it in your flow:
Copy code
from prefect import flow
     from prefect.serializers import JSONSerializer

     @flow(result_serializer=JSONSerializer(
         object_encoder="my_project.serialization.pydantic_first_encoder"
     ))
     def my_flow():
         return MyModel(...)
This will persist results as JSON dicts instead of pickled objects, avoiding the import-path problem entirely. You’ll get dicts back when you call 
await state.result()
. Why this helps - Pickle requires the exact class to be importable at load time (hence the failure on 
src.*
). JSON avoids that by storing plain data. - Existing runs that already stored pickled results will still need the original module import path to be available; the change helps all future runs. Docs - Serializers API: Serializers API - Results guide: Results in Prefect If you can share your Pydantic version (v1 or v2) and how you import your project (package vs. 
src
path), I can tailor the encoder or recommend the smallest change to your flows.
Open in Slack
PreviousNext
Powered by