<@ULVA73B9P> could you explain this error? Crash d...
# marvin-ai
r
@Marvin could you explain this error? Crash detected! Execution was interrupted by an unexpected exception: TypeError: cannot pickle '_struct.Struct' object
m
thought for 284.6 seconds
Short version: Python tried to serialize an object that cannot be pickled. In Prefect, this usually happens when: - You’re running tasks in separate processes (e.g., with
ProcessPoolTaskRunner
) and an input/return value contains a non-pickleable object, or - You’ve enabled result persistence and Prefect’s default pickle-based serializer attempts to persist a non-pickleable result. The specific type
_struct.Struct
(from Python’s
struct
module) is not pickleable, so passing/returning a
struct.Struct
instance across processes or trying to persist it will fail with: TypeError: cannot pickle '_struct.Struct' object Common ways to fix it - Don’t pass or return the
Struct
object. Pass a serializable representation (e.g., the format string) and recreate it inside the task. - Use threads instead of processes to avoid pickling across process boundaries:
Copy code
from prefect import flow
from prefect.task_runners import ThreadPoolTaskRunner

@flow(task_runner=ThreadPoolTaskRunner())
def my_flow():
    ...
- Disable result persistence for the flows/tasks touching this object:
Copy code
from prefect import flow, task

@task(persist_result=False)
def work(...):
    ...

@flow(persist_result=False)
def my_flow():
    ...
- If you do need persistence, use a serializer that matches your data. JSON avoids pickle entirely, but only works if your outputs are JSON-serializable:
Copy code
from prefect import flow, task
from prefect.serializers import JSONSerializer

@task(persist_result=True, result_serializer=JSONSerializer())
def work(...):
    return {"ok": True}
- If you must persist a
Struct
-like object, write a custom serializer that stores its format string and reconstructs it on load. Example pattern to avoid passing non-pickleable objects with processes
Copy code
import struct
from prefect import flow, task
from prefect.task_runners import ProcessPoolTaskRunner

@task
def unpack_with_format(fmt: str, data: bytes):
    s = struct.Struct(fmt)   # Recreate inside the process
    return s.unpack(data)

@flow(task_runner=ProcessPoolTaskRunner())
def run():
    return unpack_with_format.submit(">I", b"\x00\x00\x00\x01")
If you can share: - Are you using
ProcessPoolTaskRunner
? - Are you persisting results (
persist_result=True
) or seeing this during cache/result storage? - Which object are you passing/returning that contains
struct.Struct
? Relevant docs - Results and serialization overview: Results and serialization in Prefect 3 - Serializers API: prefect.serializers - Results API: prefect.results - Task runner implementations (source): Task runners source - Serializer implementations (source): Serializers source