https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Apostolos Papafragkakis

06/15/2022, 12:20 PM
Hi all, has anyone tried using pydantic models as objects exchanged between tasks? It seems that it is not supported (exceptions are thrown). I am using the latest Prefect 2 release.
1
a

Anna Geller

06/15/2022, 12:24 PM
There is a discussion about this here https://github.com/PrefectHQ/prefect/issues/5883 Response from our Senior engineer: "we include pydantic validation for flows so we can JSON parameters from the API but task runs do not track their parameters this way as they are always passed inline. It seems like it'd be weird to add this to tasks as an option and have it off by default but on by default for flows?"
Could you add more info about your use case and why you would need Pydantic validation for tasks directly to the issue? It would make it easier to assess what needs to be done, which use cases would need to be supported and would help us prioritize accordingly Thanks in advance!
a

Apostolos Papafragkakis

06/15/2022, 12:28 PM
Hi @Anna Geller, thanks for the prompt reply; I am trying to return pydantic objects after running tasks and pass them to other tasks. Upon trying to return the pydantic object an exception is thrown. I don't need task parameter validation, I just want to be able to pass pydantic objects between tasks.
a

Anna Geller

06/15/2022, 12:34 PM
I see; thanks for that background. Does it mean you are getting cloudpickle errors? Can you share the code you were using and the error you were getting? also curious, why would you pass a pydantic model between tasks? I usually see pydantic being used to define a data structure, not as an object being passed in data pipelines - I'm definitely curious to hear about your use case and what you are trying to do
a

Apostolos Papafragkakis

06/15/2022, 12:35 PM
Hi, it seems relevant to issue #5891, the model is generated in internal code elsewhere and just has to be passed between tasks. I will try the suggestion provided on github and let you know.
🙌 1
a

Anna Geller

06/15/2022, 12:37 PM
makes sense, great to see you found the relevant issue
😎 1
a

Apostolos Papafragkakis

06/15/2022, 12:41 PM
It seems I still get an exception but I will try to investigate it further
👍 1
I think I found the bug, pydantic aliases cause the problem
I will open an issue on github with reproducible examples later
👍 2
a

Anna Geller

06/16/2022, 11:33 AM
awesome, thanks a lot for that!
9 Views