https://prefect.io
Join Slack
Hey folks! I’m seeing this rather cryptic pickle error when running Prefect locally (Prefect 2.4.0, ...
# ask-community
k
Hey folks! I’m seeing this rather cryptic pickle error when running Prefect locally (Prefect 2.4.0, Python 3.10) - my code doesn’t use Pydantic’s 
AnyUrl
or subclasses within the code, do you have any idea what might be causing the pickle error?
Pydantic pickle error.txt
I think based on the error message that it has to do with Pydantic models being compiled with Cython: https://pydantic-docs.helpmanual.io/install/#compiled-with-cython but would be good to understand whether this is a known issue or not
z
Yeah this is some madness related to the compiled pydantic that I have a pretty low level workaround for so we can pickle our 
Settings
object
It looks like your task is raising an exception that contains a Pydantic model?
k
Hmm, that shouldn’t be the case - in fact, the task doesn’t raise an exception at all
Here’s the full task definition, and the error still occurs with this barebones implementation:
Copy code
@task(retries=3, retry_delay_seconds=10)
def process_record_stream(
    record_type: Type[OpenAlexBaseModel],
    records: List[OpenAlexBaseModel],
    batch_size: PositiveInt = DEFAULT_BATCH_SIZE,
) -> None:
    print("foo")
The print statement never executes, so I can only assume that the error gets thrown before the task begins executing
7 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by