https://prefect.io logo
Join SlackCommunities
Powered by
<@ULVA73B9P> Hi All, we are using prefect version ...
# marvin-in-the-wild
m
@Marvin Hi All, we are using prefect version 2.14.20 and we had pydantic library installed version "1.10.9" but when we are installing pyiceberg pydantic version is being upgraded to "2.7.0" and now when we run any flow we are getting below error can anyone help here RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see 
arbitrary_types_allowed
in Config we are managing our dependencies using pypoetry.toml
m
The search did not yield information directly related to the 
RuntimeError
encountered with Pydantic and the 
arbitrary_types_allowed
configuration when using Prefect 2.14.20 and Pydantic 2.7.0. However, I will try searching for more specific issues regarding Prefect's compatibility with Pydantic on GitHub to see if this is a known issue. Please wait while I perform the search. It seems there are a couple of relevant issues and discussions on GitHub regarding Prefect and Pydantic that might be related to the problem you are encountering: 1. There is an issue related to the 
SecretStr
type from Pydantic where using Pydantic version 2 causes a 
RuntimeError
with the message 
"no validator found for <class 'pydantic.types.SecretStr'>, see arbitrary_types_allowed in Config"
. The suggested change is to use 
from pydantic.v1 import SecretStr
when Pydantic version 2 or greater is in use. 2. Another pull request mentions a change to Prefect's validation configuration to include 
arbitrary_types_allowed
, to allow non-Pydantic classes to be used as type annotations without raising a 
RuntimeError
. It appears that some issues arise when using Python types as type hints with newer versions of Pydantic and the Prefect library. The solutions generally point towards fine-tuning Pydantic configuration to work around these validation-related errors. For now, you might want to consider checking your Pydantic usage within Prefect flows and tasks and adjust it according to the above suggestions. For long-term compatibility, you may need to keep track of updates and fixes to these issues in the Prefect repository. Meanwhile, ensure that your 
pyproject.toml
reflects the correct dependency versions to avoid unexpected upgrades with tools like 
poetry
or 
pip
.
a
from pydantic.v1 import SecretStr
@Mohit Singhal
m
this we need to add somewhere?
in all our flows?
a
Are you explicitly importing SecretStr somewhere?
m
No
a
Or you’re saying that errors are being thrown just on import of basic stuff?
m
yes
a
Sorry I’ve seen the other thing come up so I figured that was your issue
m
if we use pydantic 1.10.9 then we don't see any issue
a
On a phone so can’t immediately reproduce - can you leave this as a GitHub issue so I can point it at the right person and get you an answer?
m
do we need to set somewhere arbitrary_types_allowed = true ?
a
Nah. If you do this in a fresh virtual environment you see this issue?
m
do we need to upgrade prefect version as well?
currently using 2.14.0
n
hi @Mohit Singhal - can you share the actual traceback?
m
File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\validators.py", line 765, in find_validators raise RuntimeError(f'no validator found for {type_}, see 
arbitrary_types_allowed
in Config') RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see 
arbitrary_types_allowed
in Config
n
is that the whole thing? im curious where pydantic is interfacing with prefect here
m
as soon as we start the flow getting this error
Traceback (most recent call last): File "C:\Users\mosingha\github\de-orchestration\qa\src\flows\de_ingest_youtube\de_ingest_youtube.py", line 22, in <module> from src.utils.send_slack_alert import send_slack_warning_count_check File "C:\Users\mosingha\github\de-orchestration\qa\src\utils\send_slack_alert.py", line 2, in <module> from prefect_slack import SlackWebhook File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\prefect_slack\__init__.py", line 2, in <module> from .credentials import SlackCredentials, SlackWebhook # noqa File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\prefect_slack\credentials.py", line 13, in <module> class SlackCredentials(Block): File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\main.py", line 197, in new fields[ann_name] = ModelField.infer( File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\fields.py", line 504, in infer return cls( File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\fields.py", line 434, in init self.prepare() File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\fields.py", line 555, in prepare self.populate_validators() File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\fields.py", line 829, in populate_validators *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))), File "C:\Users\mosingha\AppData\Local\pypoetry\Cache\virtualenvs\qa-qC4fsm8t-py3.10\lib\site-packages\pydantic\v1\validators.py", line 765, in find_validators raise RuntimeError(f'no validator found for {type_}, see 
arbitrary_types_allowed
in Config') RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see 
arbitrary_types_allowed
in Config
@Nate
n
im pretty sure this is what you get when you have a pydantic v2 
SecretStr
on a pydantic v1 
BaseModel
are you subclassing 
SlackCredentials
?
m
from prefect_slack import SlackWebhook from prefect_slack.messages import send_incoming_webhook_message def send_slack_warning_count_check( webhook_url: str, flow_name: str, table: str, source_count: int, destination_count: int, ): """Send slack alert if row count check fails Args: webhook_url (str): Slack webhook url flow_name (str): Flow name table (str): Table name source_count (int): Count for source destination_count (int): Count for destination """ send_incoming_webhook_message( slack_webhook=SlackWebhook(url=webhook_url), text=f"""⚠️ Alert ... Flow : {flow_name} Table : {table} Source count: {source_count} vs Destination count: {destination_count} """, )
this is how we have setup
a
Do you keep your slack credentials in code? Usually SecretStr is used to keep credentials. Is that defined in a class or an environment variable somewhere (or both)?
m
we are using config block to store creds
and aws secret manager
a
Sweet I don’t need to see any of the credentials obviously but can you paste the imports from that file?
m
which imports?
import json import logging import boto3 from botocore.exceptions import ClientError from prefect.blocks.system import JSON
a
Where you define the config block
m
these are the imports which we use
we just set up config block manually in UI and then this is how we retrieve the values
@Adam @Nate
{ "json_blocks": { "shared-configuration":{
we also have this in one python file wheer we have all th elinks to our s3 bucket, slack webhoook and all
{ "json_blocks": { "shared-configuration":{ "env": "qa", "bucket_raw": "", "gdm_db_name": "", "region_name": "", "bucket_unload": "", "slack_webhook": "", "bucket_prefect": "",
do we need to do something here?
a
I can’t diagnose from this thread so far, so I will need more 👀 on it a path to reproduce. Can you create a GitHub issue and try to recreate this error in a new virtual environment?
r
@Marvin Is there anything new with this thread? I tried to create the following custom block:
Copy code
from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient
from prefect.blocks.core import Block
from pydantic import SecretStr, Field
from typing import Optional


class MongoDBConnection(Block):
    host: str = Field(default="localhost", description="MongoDB host")
    port: int = Field(default=27017, description="MongoDB port")
    username: Optional[str] = Field(default=None, description="MongoDB username")
    password: Optional[SecretStr] = Field(default=None, description="MongoDB password")
    database: str = Field(..., description="MongoDB database name")
    auth_source: Optional[str] = Field(default="admin", description="Authentication database")
    tls: bool = Field(default=False, description="Use TLS/SSL for connection")

    def get_client(self) -> AsyncIOMotorDatabase:
        # Construct the connection URI
        if self.username and self.password:
            uri = f"mongodb://{self.username}:{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}"
        else:
            uri = f"mongodb://{self.host}:{self.port}/{self.database}"

        # Add additional options
        uri += f"?authSource={self.auth_source}"
        if self.tls:
            uri += "&tls=true"

        # Create the client
        client = AsyncIOMotorClient(uri)
        return client.get_database(self.database)

    class Config:
        block_type_name = "MongoDB Connection"
        block_schema_capabilities = ["connection"]
And then to register it this way:
Copy code
prefect block register --file blocks/mongodb_block.py
But I am getting the same error as in this thread:
Copy code
Script at 'blocks/mongodb_block.py' encountered an exception: RuntimeError("no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config")
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/ran/medorion/prefect_poc/prefect_flows/blocks/mongodb_block.py", line 7, in <module>
    class MongoDBConnection(Block):
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 197, in __new__
    fields = ModelField.infer(
                       ^^^^^^^^^^^^^^^^^
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 504, in infer
    return cls(
           ^^^^
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 434, in __init__
    self.prepare()
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 555, in prepare
    self.populate_validators()
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 829, in populate_validators
    *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))),
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/validators.py", line 765, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config

Unable to load file at blocks/mongodb_block.py. Please make sure the file path is correct and the file contains valid Python.
(venv) ran@MedOrion-Rans-MacBook-Pro-2 prefect_flows % prefect block register --file blocks/mongodb_block.py 
Script at 'blocks/mongodb_block.py' encountered an exception: RuntimeError("no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config")
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/ran/medorion/prefect_poc/prefect_flows/blocks/mongodb_block.py", line 7, in <module>
    class MongoDBConnection(Block):
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 197, in __new__
    fields = ModelField.infer(
                       ^^^^^^^^^^^^^^^^^
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 504, in infer
    return cls(
           ^^^^
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 434, in __init__
    self.prepare()
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 555, in prepare
    self.populate_validators()
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 829, in populate_validators
    *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))),
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/validators.py", line 765, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config
m
Your message was too long by 1182 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
Is there anything new with this thread?
I tried to create the following custom block: ```from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient from prefect.blocks.core import Block from pydantic import SecretStr, Field from typing import Optional class MongoDBConnection(Block): host: str = Field(default="localhost", description="MongoDB host") port: int = Field(default=27017, description="MongoDB port") username: Optional[str] = Field(default=None, description="MongoDB username") password: Optional[SecretStr] = Field(default=None, description="MongoDB password") database: str = Field(..., description="MongoDB database name") auth_source: Optional[str] = Field(default="admin", description="Authentication database") tls: bool = Field(default=False, description="Use TLS/SSL for connection") def get_client(self) -> AsyncIOMotorDatabase: # Construct the connection URI if self.username and self.password: uri = f"mongodb//{self.username}{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}" else: uri = f"mongodb//{self.host}{self.port}/{self.database}" # Add additional options uri += f"?authSource={self.auth_source}" if self.tls: uri += "&tls=true" # Create the
r
@Marvin Is there anything new with this thread? I tried to create the following custom block:
Copy code
from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient
from prefect.blocks.core import Block
from pydantic import SecretStr, Field
from typing import Optional


class MongoDBConnection(Block):
    host: str = Field(default="localhost", description="MongoDB host")
    port: int = Field(default=27017, description="MongoDB port")
    username: Optional[str] = Field(default=None, description="MongoDB username")
    password: Optional[SecretStr] = Field(default=None, description="MongoDB password")
    database: str = Field(..., description="MongoDB database name")
    auth_source: Optional[str] = Field(default="admin", description="Authentication database")
    tls: bool = Field(default=False, description="Use TLS/SSL for connection")

    def get_client(self) -> AsyncIOMotorDatabase:
        # Construct the connection URI
        if self.username and self.password:
            uri = f"mongodb://{self.username}:{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}"
        else:
            uri = f"mongodb://{self.host}:{self.port}/{self.database}"

        # Add additional options
        uri += f"?authSource={self.auth_source}"
        if self.tls:
            uri += "&tls=true"

        # Create the client
        client = AsyncIOMotorClient(uri)
        return client.get_database(self.database)

    class Config:
        block_type_name = "MongoDB Connection"
        block_schema_capabilities = ["connection"]
And then to register it this way:
Copy code
prefect block register --file blocks/mongodb_block.py
But I am getting the same error as in this thread:
Copy code
File "/Users/ran/medorion/prefect_poc/venv/lib/python3.11/site-packages/pydantic/v1/validators.py", line 765, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config
m
Your message was too long by 179 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
Is there anything new with this thread?
I tried to create the following custom block: ```from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient from prefect.blocks.core import Block from pydantic import SecretStr, Field from typing import Optional class MongoDBConnection(Block): host: str = Field(default="localhost", description="MongoDB host") port: int = Field(default=27017, description="MongoDB port") username: Optional[str] = Field(default=None, description="MongoDB username") password: Optional[SecretStr] = Field(default=None, description="MongoDB password") database: str = Field(..., description="MongoDB database name") auth_source: Optional[str] = Field(default="admin", description="Authentication database") tls: bool = Field(default=False, description="Use TLS/SSL for connection") def get_client(self) -> AsyncIOMotorDatabase: # Construct the connection URI if self.username and self.password: uri = f"mongodb//{self.username}{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}" else: uri = f"mongodb//{self.host}{self.port}/{self.database}" # Add additional options uri += f"?authSource={self.auth_source}" if self.tls: uri += "&tls=true" # Create the
r
@Marvin Is there anything new with this thread? I tried to create the following custom block:
Copy code
from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient
from prefect.blocks.core import Block
from pydantic import SecretStr, Field
from typing import Optional


class MongoDBConnection(Block):
    host: str = Field()
    port: int = Field()
    username: Optional[str] = Field()
    password: Optional[SecretStr] = Field()
    database: str = Field()
    auth_source: Optional[str] = Field()
    
    def get_client(self) -> AsyncIOMotorDatabase:
        if self.username and self.password:
            uri = f"mongodb://{self.username}:{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}"
        else:
            uri = f"mongodb://{self.host}:{self.port}/{self.database}"

        uri += f"?authSource={self.auth_source}"
    
        client = AsyncIOMotorClient(uri)
        return client.get_database(self.database)

    class Config:
        block_type_name = "MongoDB Connection"
        block_schema_capabilities = ["connection"]
And then :
Copy code
prefect block register --file blocks/mongodb_block.py
But I am getting the same error as in this thread:
Copy code
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config
m
Your message was too long by 5 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
Is there anything new with this thread?
I tried to create the following custom block:
Copy code
from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient
from prefect.blocks.core import Block
from pydantic import SecretStr, Field
from typing import Optional


class MongoDBConnection(Block):
    host: str = Field()
    port: int = Field()
    username: Optional[str] = Field()
    password: Optional[SecretStr] = Field()
    database: str = Field()
    auth_source: Optional[str] = Field()
    
    def get_client(self) -> AsyncIOMotorDatabase:
        if self.username and self.password:
            uri = f"mongodb://{self.username}:{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}"
        else:
            uri = f"mongodb://{self.host}:{self.port}/{self.database}"

        uri += f"?authSource={self.auth_source}"
    
        client = AsyncIOMotorClient(uri)
        return client.get_database(self.database)

    class Config:
        block_type_name = "MongoDB Connection"
        block_schema_capabilities = ["connection"]
And then :
Copy code
prefect block register --file blocks/mongodb_block.py
But I am getting the same error as in this thread: ```RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types
r
@Marvin Is there anything new with this thread? I created the following block:
Copy code
from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient
from prefect.blocks.core import Block
from pydantic import SecretStr, Field
from typing import Optional


class MongoDBConnection(Block):
    host: str = Field()
    port: int = Field()
    username: Optional[str] = Field()
    password: Optional[SecretStr] = Field()
    database: str = Field()
    auth_source: Optional[str] = Field()
    
    def get_client(self) -> AsyncIOMotorDatabase:
        if self.username and self.password:
            uri = f"mongodb://{self.username}:{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}"
        else:
            uri = f"mongodb://{self.host}:{self.port}/{self.database}"
        uri += f"?authSource={self.auth_source}"
    
        client = AsyncIOMotorClient(uri)
        return client.get_database(self.database)

    class Config:
        block_type_name = "MongoDB Connection"
        block_schema_capabilities = ["connection"]
then:
Copy code
prefect block register --file blocks/mongodb_block.py
I am getting the same error:
Copy code
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config
m
It seems the issue you are encountering with the 
RuntimeError
due to 
SecretStr
in Pydantic v2 has been identified and discussed in the Prefect community. Here are the key points and suggestions: 1. Issue with 
SecretStr
in Pydantic v2: - The problem stems from Pydantic v2, where importing 
SecretStr
directly as 
from pydantic import SecretStr
leads to the error 
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see arbitrary_types_allowed in Config
. - Suggested fix: Use 
from pydantic.v1 import SecretStr
when using Pydantic version >= 2. 2. Adjusting the Pydantic Configuration: - Prefect has discussed this issue and suggested ensuring the Pydantic configuration includes 
arbitrary_types_allowed
to handle arbitrary types without causing validation errors. To resolve the issue in your block definition, you should modify your import statement for 
SecretStr
when using Pydantic v2. Here’s an updated version of your MongoDBConnection block:
Copy code
python
from motor.motor_asyncio import AsyncIOMotorDatabase, AsyncIOMotorClient
from prefect.blocks.core import Block
from pydantic.v1 import SecretStr, Field  # Updated import for SecretStr
from typing import Optional

class MongoDBConnection(Block):
    host: str = Field()
    port: int = Field()
    username: Optional[str] = Field()
    password: Optional[SecretStr] = Field()
    database: str = Field()
    auth_source: Optional[str] = Field()
    
    def get_client(self) -> AsyncIOMotorDatabase:
        if self.username and self.password:
            uri = f"mongodb://{self.username}:{self.password.get_secret_value()}@{self.host}:{self.port}/{self.database}"
        else:
            uri = f"mongodb://{self.host}:{self.port}/{self.database}"
        if self.auth_source:
            uri += f"?authSource={self.auth_source}"
    
        client = AsyncIOMotorClient(uri)
        return client.get_database(self.database)

    class Config:
        block_type_name = "MongoDB Connection"
        block_schema_capabilities = ["connection"]

# Register the updated block
prefect block register --file blocks/mongodb_block.py
By updating the import statement to 
from pydantic.v1 import SecretStr
, you can avoid the validator error. Ensure that Pydantic v1 compatibility is available when using Pydantic v2 in your project. Let me know if this resolves your issue or if you need further assistance.
7 Views
Open in Slack
PreviousNext
Powered by