https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> When did task start applying pydantic...
# ask-marvin
o
@Marvin When did task start applying pydantic validataion? There is an error that did not occur in the previously used version. Here is the error Also, is it possible to remove this validation?
Copy code
Crash detected! Execution was interrupted by an unexpected exception: KeyError: '__fields__'
m
It appears that Prefect has started using Pydantic validators to ensure the general structure and types of inputs in your flow. You can read more about it in their documentation. It seems the specific error 
KeyError: '__fields__'
is related to this validation step, possibly due to a mismatch or missing fields. ### Removing Validation Directly removing Pydantic validation from Prefect might not be straightforward because it's integrated into how Prefect processes and verifies inputs. However, there are two ways to handle this: 1. Custom Validators: Use custom Pydantic validators to catch and handle the discrepancies.
Copy code
python
    import pydantic
    from prefect.input import RunInput

    class YourInputClass(RunInput):
        field1: str
        field2: int

        @pydantic.validator("field1", pre=True, always=True)
        def validate_field1(cls, value):
            if value is None:
                return "default_value"  # or any default handling
            return value
2. Modify Input Data: Ensure that the input data conforms to your RunInput or BaseModel specifications. For further assistance, you might want to raise an issue in their GitHub repository. Please let me know if you need more specific examples or further help.
n
hi @orangekame3 - hmm that seems odd. do you have a stack trace for that?
o
@Nate Hi, thank you following! Sorry, I am unable to show you the stack trace for this code. All we know at this point is that if we define a task like the following and execute it in a flow, we get
Copy code
Crash detected!Execution was interrupted by an unexpected exception: KeyError: '__fields__'
The only thing that happens is that is generated. When I remove the task's decorator from execute_experiment, the process works fine, so I suspect that there is some kind of validation in the task part. To add, I have not set the key fields in any of my Classes, and I do not use Pydantic in any of the Classes I have defined. I don't remember this error occurring before, so I think this behavior may have started in some version. I am currently investigating the version, so I can't give you specific values.
Copy code
#task definition
@task(name="execute-experiment", task_run_name="test")
def execute_experiment(
    exp_name: str,
	...
) -> str:
    try:
        logger = get_run_logger()
        logger.warn("test")
        dataset = exp.take_data(tdm, notes, save=save)  # type: ignore
        logger.info(f"start analyze: {exp_name}")  # type: ignore
        return "success"
    except Exception as e:
        raise Exception(f"Error: {e}")
Copy code
# in flow
status = execute_experiment(
	exp_name=exp_name,
	...
)  # error occured here
logger.info("experiment done")　# this code is not executed, when code crash
and, now i use
Copy code
prefect = "^2.14.17"
because, I want to use following feature https://github.com/PrefectHQ/prefect/pull/11476
This is stack trace(DEBUG LOG) foo/bar is my internal package
Copy code
Crash details:
Traceback (most recent call last):
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/engine.py", line 2211, in report_flow_run_crashes
    yield
  File "/home/edge/.pyenv/versions/3.10.14/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/engine.py", line 1629, in create_task_run_then_submit
    await create_task_run(
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/engine.py", line 1667, in create_task_run
    task_inputs = {k: await collect_task_run_inputs(v) for k, v in parameters.items()}
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/engine.py", line 1667, in 
    task_inputs = {k: await collect_task_run_inputs(v) for k, v in parameters.items()}
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/engine.py", line 107, in collect_task_run_inputs
    visit_collection(
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/collections.py", line 337, in visit_collection
    items = [(visit_nested(k), visit_nested(v)) for k, v in expr.items()]
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/collections.py", line 337, in 
    items = [(visit_nested(k), visit_nested(v)) for k, v in expr.items()]
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/collections.py", line 279, in visit_nested
    return visit_collection(
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/collections.py", line 297, in visit_collection
    result = visit_expression(expr)
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/collections.py", line 293, in visit_expression
    return visit_fn(expr)
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/utilities/engine.py", line 96, in add_futures_and_states_to_inputs
    elif is_state(obj):
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/prefect/states.py", line 409, in is_state
    return isinstance(obj, classes_)
  File "/home/edge/work/qdash/.venv/lib/python3.10/site-packages/pydantic/v1/main.py", line 304, in __instancecheck__
    return hasattr(instance, '__fields__') and super().__instancecheck__(instance)
  File "/home/edge/work/foo/bar/wrapper/note.py", line 229, in __getattr__
    return self.globals[key].value
  File "/home/edge/work/foo/bar/wrapper/attribute_dict.py", line 72, in __getitem__
    return dict.__getitem__(self, key)
KeyError: '__fields__'

```
solved about this. It seems that my internal package foo/bar/attribute_dict has been inadvertently involved in Pydantic's validation. (attribute_dict is a regular Class) The reason why this has been involved in Pydantic's validation is still a mystery.
9 Views
Open in Slack
PreviousNext
Powered by