I m back with may be another bug It looks like Prefect ` flo Prefect Community #ask-community

I'm back with may be another bug. It looks like Pr...

Joe

11/27/2024, 2:45 PM

I'm back with may be another bug. It looks like Prefect

@flow()

fails if I pass an argument that is either a dataclass or a Pydantic model that contains a numpy array. Here is the code for a minimal repro:

Copy code

from dataclasses import dataclass

import numpy as np
from prefect import flow
from pydantic import BaseModel


@dataclass
class T1:
    data: np.ndarray


@flow(name="test1", validate_parameters=True)
def test_flow1(t: T1):
    print(t)


class T2(BaseModel):
    data: np.ndarray


@flow(name="test2", validate_parameters=True)
def test_flow2(t: T2):
    print(t)


test_flow1(T1(data=np.random.rand(2, 2)))
test_flow2(T2(data=np.random.rand(2, 2)))

Bianca Hoch

11/27/2024, 3:45 PM

Hi Joe! Quick house keeping question, could you move the stack trace into the 🧵 to keep the main slack channel tidy? 🙏 Please & thank you!

Joe

11/27/2024, 3:49 PM

Sorry about that!

Bianca Hoch

11/27/2024, 5:12 PM

No worries at all!

Bianca Hoch

11/27/2024, 5:14 PM

Hmm.. Admittedly, I'm not terribly familiar with the error you got, but I know Prefect

3.1.4

has fixes that address some breaking changes with a recent pydantic release

Bianca Hoch

11/27/2024, 5:14 PM

i'll try to run your MRE with that version

Bianca Hoch

11/27/2024, 5:21 PM

Ahh okay I think I know what's up. Can you try this?

Copy code

from dataclasses import dataclass

import numpy as np
from prefect import flow
from pydantic import BaseModel, ConfigDict


@dataclass
class T1:
    data: np.ndarray

@flow(name="test1", validate_parameters=True)
def test_flow1(t: T1):
    print(t)


class T2(BaseModel):
    data: np.ndarray
    model_config = ConfigDict(arbitrary_types_allowed=True)


@flow(name="test2", validate_parameters=True)
def test_flow2(t: T2):
    print(t)


test_flow1(T1(data=np.random.rand(2, 2)))
test_flow2(T2(data=np.random.rand(2, 2)))

Bianca Hoch

11/27/2024, 6:16 PM

Moving traceback here just to 🧹 ✨

Copy code

raceback (most recent call last):
  File "/Users/joe/code/feature-library/features/repro.py", line 13, in <module>
    @flow(name="test1", validate_parameters=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/prefect/flows.py", line 1597, in flow
    Flow(
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/prefect/flows.py", line 334, in __init__
    ValidatedFunction(self.fn, config={"arbitrary_types_allowed": True})
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/v1/decorator.py", line 126, in __init__
    self.create_model(fields, takes_args, takes_kwargs, config)
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/prefect/_internal/pydantic/v2_validated_func.py", line 93, in create_model
    self.model = create_model(
                 ^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/main.py", line 1673, in create_model
    return meta(
           ^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py", line 226, in __new__
    complete_model_class(
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py", line 658, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/main.py", line 697, in __get_pydantic_core_schema__
    return handler(source)
           ^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 881, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 693, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1073, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1261, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2051, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2032, in inner_handler
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 984, in match_type
    return self._dataclass_schema(obj, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1765, in _dataclass_schema
    args = sorted(
           ^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1766, in <genexpr>
    (self._generate_dc_field_schema(k, v, decorators) for k, v in fields.items()),
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1090, in _generate_dc_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1261, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2051, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2032, in inner_handler
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 997, in match_type
    return self._unknown_type_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 515, in _unknown_type_schema
    raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'numpy.ndarray'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.

If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.

For further information visit <https://errors.pydantic.dev/2.10/u/schema-for-unknown-type>

Joe

11/27/2024, 6:38 PM

Adding the ConfigDict line still doesn't change the behavior. Also, that'd only apply to the Pydantic model, not the dataclass.

Bianca Hoch

11/27/2024, 7:05 PM

Fair point, adding the

ConfigDict

line to the data class isn't necessary (example above has been adjusted).

Joe

11/27/2024, 7:05 PM

Looking at Prefect source code, it looks like that option is already enabled:

Joe

11/27/2024, 7:05 PM

https://github.com/PrefectHQ/prefect/blob/c1c70c255126ebdd5a625e364db7139cc4873aea/src/prefect/flows.py#L334C1-L335C1

Bianca Hoch

11/27/2024, 7:06 PM

I was able to get the example I shared* to work with 3.1.4. Did you upgrade your version as well?

Joe

11/27/2024, 7:06 PM

Let me upgrade and see if that works.

Joe

11/27/2024, 7:08 PM

Yup, 3.1.4 fixed it. Thank you!

Bianca Hoch

11/27/2024, 7:08 PM

panda dancing

5 Views

Open in Slack

Previous Next