Hey folks! I’m seeing this rather cryptic pickle e...
# prefect-communityk
Krishnan Chandra
09/15/2022, 6:33 PMHey folks! I’m seeing this rather cryptic pickle error when running Prefect locally (Prefect 2.4.0, Python 3.10) - my code doesn’t use Pydantic’s
AnyUrlKrishnan Chandra
09/15/2022, 6:37 PMI think based on the error message that it has to do with Pydantic models being compiled with Cython: https://pydantic-docs.helpmanual.io/install/#compiled-with-cython
but would be good to understand whether this is a known issue or not
Krishnan Chandra
09/15/2022, 6:43 PMz
Zanie
09/15/2022, 6:45 PM
Yeah this is some madness related to the compiled pydantic that I have a pretty low level workaround for so we can pickle our
SettingsZanie
09/15/2022, 6:46 PM
It looks like your task is raising an exception that contains a Pydantic model?
k
Krishnan Chandra
09/15/2022, 6:46 PMHmm, that shouldn’t be the case - in fact, the task doesn’t raise an exception at all
Krishnan Chandra
09/15/2022, 6:47 PMHere’s the full task definition, and the error still occurs with this barebones implementation:
Copy code
@task(retries=3, retry_delay_seconds=10)
def process_record_stream(
record_type: Type[OpenAlexBaseModel],
records: List[OpenAlexBaseModel],
batch_size: PositiveInt = DEFAULT_BATCH_SIZE,
) -> None:
print("foo")Krishnan Chandra
09/15/2022, 6:48 PMThe print statement never executes, so I can only assume that the error gets thrown before the task begins executing
3 Views