https://prefect.io logo
Join Slack
Powered by
Hi, We’re suddenly getting the error “ModuleNotFou...
# ask-community
y
Hi, We’re suddenly getting the error “ModuleNotFoundError: No module named ‘pydantic.plugin._loader’” We are using the new ECS push work pool. Just few hours everything worked.
👀 1
🆘 1
✅ 1
It’s a simple flow:
prefect.yaml
the error:
Copy code
Validation of flow parameters failed with error: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 422, in retrieve_flow_then_begin_flow_run
    parameters = flow.validate_parameters(flow_run.parameters)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 469, in validate_parameters
    validated_fn = ValidatedFunction(
  File "/usr/local/lib/python3.10/site-packages/pydantic/v1/decorator.py", line 126, in __init__
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/pydantic/v2_validated_func.py", line 51, in create_model
    class DecoratorBaseModel(BaseModel):
  File "/usr/local/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 184, in __new__
  File "/usr/local/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 499, in complete_model_class
  File "/usr/local/lib/python3.10/site-packages/pydantic/plugin/_schema_validator.py", line 28, in create_schema_validator
ModuleNotFoundError: No module named 'pydantic.plugin._loader'
We didn’t change anything at our side, code wise.
j
Hey I've the same error...
a
👋 @Yaron Levi I’ve got eyes on this
@Yaron Levi if you append a pin to your internal requirements.txt to, say pydantic==1.10.0 can you let me know what you get
y
@Alexander Azzam Thanks. If you need any help debugging the issue, please reach out.
Sure let me check this.
a
just to let you and @Johan sh know where my head's at: • in 2.13.5 we added a lot of conditional imports to support force install of pydantic v2. But it's literally things like if (the user's pydantic version < 2){do nothing different}. • In 2.13.7 (which we released yesterday) we removed our pin to pydantic < 2. so pydantic errors very much on the mind for me and 🤞 it's something I did. so the most surprising thing to me is the pin to 2.13.5 which was working that's suddenly not.
y
@Alexander Azzam I’ve set pydantic to 1.10.0
but the flow still fails
a
🫡 thanks @Yaron Levi that's very helpful
👍 1
okay @Yaron Levi (also let me know if tagging you directly is redundant): my next suspicion now is that the image you're using might not be pinned? if it's unpinned (e.g. 
prefect:latest
) then you'll be using 2.13.7 as a base
y
What do you mean?
What config to check?
We never chose or configed any “image”.
We are using the new ECS push work pool if that helps.
This is the pool config (regular config, we haven’t changed almost anything):
The prefect.yaml we use is here in this thread (above)
Can you point me to a place I should look at?
a
ahh gotcha, thanks! let me recreate real quick and should be able to be a bit more prescriptive one sec
👍 1
j
Hey @Yaron Levi under image in the Work pool, can you change it to be:
Copy code
docker.io/prefecthq/prefect:2.13.5-python3.10
(or w/e python you run on) and see how that works
y
@Jake Kaplan Sure, one second.
@Jake Kaplan Any idea how to check which python version we are using? (I don’t think we’ve defined something specific)
a
from the screen shots you had above the trace errors look like they were coming from python3.10
✅ 1
y
@Alexander Azzam @Jake Kaplan OK, this solves the problem for now:
The flow is running again.
j
@Yaron Levi Push Work Pools default to the latest prefect image tag (it's not annotated correctly on the page, will get a fix for that). With a release of 2.13.7 there seems to be an incompatible with installing 2.13.7 first and then downgrading to 2.13.5 via your requirements file. Pinning the image on your work pools should fix for now, alternatively bumping your prefect version up could also resolve.
y
@Jake Kaplan I’ve tried setting Prefect to 2.13.7
But I would still get the error
I can try again just to be sure.
@Jake Kaplan I’ve defined 2.13.7 in two places:
And we still see the error.
a
quick summary of where we're at: I think there's two issues present here: 1. no code changes -> different behavior: this is because of push work pools pulling prefect:latest (2.13.7) by default instead of what it promises to do: pull the image associated with your local prefect version (2.13.5). This means that your code didn't change and yet the image that was being pulled for your push work pool was changing, explaining the change in behavior. 2. prefect 2.13.7 throwing errors on your flow code Both are issues, but I think (1) >> (2) in severity. 1. Pinning your push work pool's docker image to 2.13.5 (the version you were on) appears to have fixed (1)'s effect on you. Heads up we're putting in a fix to (1) more holistically on our side to make it respect your local version as we promise in the UI. 2. This is unresolved, but I'll elevate to a ticket for now and tag you in it so you can subscribe to it.
y
@Alexander Azzam Thank you for the detailed information! So for now we will use the custom image configuration until those two issue will be fixed.
a
Thanks you @Yaron Levi! While I have you just wanted to say I've seen how active you've been here both in giving feedback, taking time to detail issues as they come up, and being a real partner to us to help us debug. Mornings like these I'm super grateful to have folks like you in our community and will mirror that empathy you have with work on our end 💪
🙏 2
j
Thank you guys, using 
<http://docker.io/prefecthq/prefect:2.13.5-python3.10|docker.io/prefecthq/prefect:2.13.5-python3.10>
worked well for me too! (Cloud Run Push Work Pool)
👍 1
13 Views
Open in Slack
PreviousNext
Powered by