https://prefect.io logo
#prefect-community
Title
# prefect-community
k

Krishnan Chandra

09/15/2022, 6:33 PM
Hey folks! I’m seeing this rather cryptic pickle error when running Prefect locally (Prefect 2.4.0, Python 3.10) - my code doesn’t use Pydantic’s
AnyUrl
or subclasses within the code, do you have any idea what might be causing the pickle error?
I think based on the error message that it has to do with Pydantic models being compiled with Cython: https://pydantic-docs.helpmanual.io/install/#compiled-with-cython but would be good to understand whether this is a known issue or not
z

Zanie

09/15/2022, 6:45 PM
Yeah this is some madness related to the compiled pydantic that I have a pretty low level workaround for so we can pickle our
Settings
object
It looks like your task is raising an exception that contains a Pydantic model?
k

Krishnan Chandra

09/15/2022, 6:46 PM
Hmm, that shouldn’t be the case - in fact, the task doesn’t raise an exception at all
Here’s the full task definition, and the error still occurs with this barebones implementation:
Copy code
@task(retries=3, retry_delay_seconds=10)
def process_record_stream(
    record_type: Type[OpenAlexBaseModel],
    records: List[OpenAlexBaseModel],
    batch_size: PositiveInt = DEFAULT_BATCH_SIZE,
) -> None:
    print("foo")
The print statement never executes, so I can only assume that the error gets thrown before the task begins executing
3 Views