Robert Esteves
08/25/2025, 3:27 PMMarvin
08/25/2025, 3:27 PMMarvin
08/25/2025, 3:32 PMProcessPoolTaskRunner
) and an input/return value contains a non-pickleable object, or
- You’ve enabled result persistence and Prefect’s default pickle-based serializer attempts to persist a non-pickleable result.
The specific type _struct.Struct
(from Python’s struct
module) is not pickleable, so passing/returning a struct.Struct
instance across processes or trying to persist it will fail with:
TypeError: cannot pickle '_struct.Struct' object
Common ways to fix it
- Don’t pass or return the Struct
object. Pass a serializable representation (e.g., the format string) and recreate it inside the task.
- Use threads instead of processes to avoid pickling across process boundaries:
from prefect import flow
from prefect.task_runners import ThreadPoolTaskRunner
@flow(task_runner=ThreadPoolTaskRunner())
def my_flow():
...
- Disable result persistence for the flows/tasks touching this object:
from prefect import flow, task
@task(persist_result=False)
def work(...):
...
@flow(persist_result=False)
def my_flow():
...
- If you do need persistence, use a serializer that matches your data. JSON avoids pickle entirely, but only works if your outputs are JSON-serializable:
from prefect import flow, task
from prefect.serializers import JSONSerializer
@task(persist_result=True, result_serializer=JSONSerializer())
def work(...):
return {"ok": True}
- If you must persist a Struct
-like object, write a custom serializer that stores its format string and reconstructs it on load.
Example pattern to avoid passing non-pickleable objects with processes
import struct
from prefect import flow, task
from prefect.task_runners import ProcessPoolTaskRunner
@task
def unpack_with_format(fmt: str, data: bytes):
s = struct.Struct(fmt) # Recreate inside the process
return s.unpack(data)
@flow(task_runner=ProcessPoolTaskRunner())
def run():
return unpack_with_format.submit(">I", b"\x00\x00\x00\x01")
If you can share:
- Are you using ProcessPoolTaskRunner
?
- Are you persisting results (persist_result=True
) or seeing this during cache/result storage?
- Which object are you passing/returning that contains struct.Struct
?
Relevant docs
- Results and serialization overview: Results and serialization in Prefect 3
- Serializers API: prefect.serializers
- Results API: prefect.results
- Task runner implementations (source): Task runners source
- Serializer implementations (source): Serializers source