https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have recently updated prefect-agent...
# ask-marvin
s
@Marvin I have recently updated prefect-agent dependencies from from pydantic v1 to pydantic v2 and am encountering this error.
Copy code
line 5, in 
    from prefect_dbt import DbtCliProfile
  File "/usr/local/lib/python3.11/site-packages/prefect_dbt/__init__.py", line 3, in 
    from .cloud import DbtCloudCredentials  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_dbt/cloud/__init__.py", line 1, in 
    from .credentials import DbtCloudCredentials  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_dbt/cloud/credentials.py", line 14, in 
    class DbtCloudCredentials(CredentialsBlock):
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/main.py", line 197, in __new__
    fields[ann_name] = ModelField.infer(
                       ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/fields.py", line 504, in infer
    return cls(
           ^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/fields.py", line 434, in __init__
    self.prepare()
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/fields.py", line 555, in prepare
    self.populate_validators()
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/fields.py", line 829, in populate_validators
    *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))),
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/v1/validators.py", line 768, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
RuntimeError: no validator found for , see `arbitrary_types_allowed` in Config
m
✅ thought for 287.9 seconds
Thanks for the traceback — that helps. What’s happening here is: - When you do 
from prefect_dbt import DbtCliProfile
, the package’s 
__init__
imports the dbt Cloud models (
DbtCloudCredentials
) as well. - Those cloud models are still defined against Pydantic v1-style models. Under Pydantic v2’s v1-compat layer, a field type in that model doesn’t have a validator, so you get the “no validator found … see arbitrary_types_allowed” error. A couple of ways to fix or work around this: Quick workaround if you only need the dbt CLI (not dbt Cloud) - Import from the submodule so you don’t pull in the cloud models at import time:
Copy code
# instead of: from prefect_dbt import DbtCliProfile
from prefect_dbt.cli.credentials import DbtCliProfile
# or:
from prefect_dbt.cli import DbtCliProfile
Make sure your versions are compatible - If you’re on Prefect 3.x (assumed unless you say otherwise), you should use a version of 
prefect-dbt
that supports Pydantic v2. Upgrade both Prefect and prefect-dbt to their latest releases:
Copy code
pip install -U "prefect" "prefect-dbt" "pydantic>=2"
- If you are still on Prefect 2.x, you’ll need to pin Pydantic to v1 because many Prefect 2.x collections (including older 
prefect-dbt
releases) were built on Pydantic v1:
Copy code
pip install -U "prefect<3" "prefect-dbt<1.0.0" "pydantic<2"
Double-check what you’re currently running
Copy code
python -c "import prefect, prefect_dbt, pydantic; print('prefect', prefect.__version__, 'prefect-dbt', prefect_dbt.__version__, 'pydantic', pydantic.__version__)"
If you do need dbt Cloud models on Pydantic v2 and you’re already on the latest 
prefect-dbt
, please share: - Your Prefect version - Your 
prefect-dbt
version - Your Pydantic version - The exact import/code snippet you’re running I can then reproduce and suggest a targeted fix. References: - Prefect 3 dbt integration docs: prefect-dbt integration (Prefect 3) - Prefect 2 dbt Cloud credentials docs (for reference if you’re on 2.x): prefect-dbt cloud credentials (Prefect 2)
s
@Marvin My pyproject defines my versions as the below. 
prefect = "2.20.16"
pydantic = "<2.11.4"
prefect-dbt = "^0.3.1"
m
✅ thought for 64.4 seconds
Thanks for sharing your versions — that explains the error. You’re on Prefect 2.x: - prefect = 2.20.16 - prefect-dbt = ^0.3.1 - pydantic = <2.11.4 prefect 2.x and prefect-dbt 0.3.x are built on Pydantic v1. Your constraint 
pydantic = "<2.11.4"
still installs Pydantic v2.x (just below 2.11.4), which triggers the v1-compat layer and leads to the “no validator found … see arbitrary_types_allowed” error when 
DbtCloudCredentials
is imported. Fix (stay on Prefect 2.x) - Pin Pydantic to v1:
Copy code
# poetry
poetry add 'pydantic<2'

# or pip
pip install -U 'pydantic<2'
- Keep your existing Prefect and prefect-dbt versions:
Copy code
prefect==2.20.16
prefect-dbt==0.3.1
pydantic<2
- If you use Poetry, re-lock after changing the constraint:
Copy code
poetry lock --no-update
poetry install
Alternative (upgrade to Pydantic v2) - Move to Prefect 3.x and the latest prefect-dbt:
Copy code
pip install -U 'prefect>=3' 'prefect-dbt' 'pydantic>=2'
Note: Prefect 3.x has some breaking changes vs 2.x. Docs for reference: - Prefect 2 dbt Cloud credentials: prefect-dbt (Prefect 2) - Prefect 3 dbt integration: prefect-dbt (Prefect 3) If you only need dbt CLI and must temporarily work around the import crash, you can avoid importing cloud models by importing from the submodule:
Copy code
from prefect_dbt.cli.credentials import DbtCliProfile
…but the correct long-term fix on Prefect 2.x is to pin 
pydantic<2
. Want me to suggest exact pyproject entries or help validate the environment after the change?
s
@Marvin The DbtCliProfile is used to create the dbt_cli_profile used in the trigger below. Will it be fine just getting the credentials and avoiding importing cloud models?
Copy code
dbt_cli_profile = DbtCliProfile.load("analytics-anon")
Copy code
result = trigger_dbt_cli_command(
        dbt_command,
        dbt_cli_profile=dbt_cli_profile,
        overwrite_profiles=True,
        helper_command=f"cd {workdir}",
        return_all=True,
    )
m
✅ thought for 83.4 seconds
Short answer: Yes. If you’re only using the dbt CLI, you can avoid importing the cloud models entirely and things will work. Do these imports instead of importing from the top-level package (which pulls in the cloud code and triggers the Pydantic v2 error):
Copy code
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.commands import trigger_dbt_cli_command
Your snippet is fine as-is once you change the imports:
Copy code
dbt_cli_profile = DbtCliProfile.load("analytics-anon")

result = trigger_dbt_cli_command(
    dbt_command,
    dbt_cli_profile=dbt_cli_profile,
    overwrite_profiles=True,
    helper_command=f"cd {workdir}",
    return_all=True,
)
Notes: - 
DbtCliProfile.load("analytics-anon")
expects you already created a DbtCliProfile block with that name in Prefect (via code or UI). - This submodule import sidesteps the dbt Cloud models and avoids the Pydantic-related crash. That said, since you’re on Prefect 2.x and prefect-dbt 0.3.x, the correct long-term fix is to pin Pydantic to v1 to prevent similar surprises elsewhere:
Copy code
# Poetry
poetry add 'pydantic<2'
poetry lock --no-update && poetry install

# or pip
pip install -U 'pydantic<2'
Docs for the CLI helpers (Prefect 2.x): prefect-dbt CLI (Prefect 2) If you want, share your pyproject and I’ll suggest exact constraints to keep everything compatible.
s
@Marvin I am still getting the below error after changing the import
Copy code
from prefect_dbt.cli.credentials import DbtCliProfile
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect_dbt/__init__.py", line 3, in <module>
    from .cloud import DbtCloudCredentials  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect_dbt/cloud/__init__.py", line 1, in <module>
    from .credentials import DbtCloudCredentials  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect_dbt/cloud/credentials.py", line 14, in <module>
    class DbtCloudCredentials(CredentialsBlock):
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 197, in __new__
    fields[ann_name] = ModelField.infer(
                       ^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 504, in infer
    return cls(
           ^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 434, in __init__
    self.prepare()
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 555, in prepare
    self.populate_validators()
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 829, in populate_validators
    *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))),
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/validators.py", line 768, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect/flows.py", line 1737, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 205, in import_object
    module = load_script_as_module(script_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 168, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/dbt/anonymization_dbt_run_flow.py' encountered an exception: RuntimeError("no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config")
My pyproject dependencies are below:
Copy code
[tool.poetry.dependencies]
python = "~3.11"
prefect = "2.20.16"
opentok = "^3.2.0"
dbt-postgres = "^1.3.0"
pygit2 = "^1.11.1"
mysql-connector-python = "^8.0.32"
prefect-airbyte = "^0.2.0"
prefect-dbt = "^0.3.1"
ipython = "^8.14.0"
pydantic = "<2.11.4"
mailerlite = "^0.1.3"
great-expectations = "^0.18.8"
fuzzywuzzy = "^0.18.0"
rapidfuzz = "^3.8.1"
Unidecode = "^1.3.8"
spacy = "^3.7.4"
Office365-REST-Python-Client = "^2.5.14"
python-docx = "^1.1.2"
apprise = "1.9.3"
mailersend = "^2.0.0"
m
✅ thought for 33.0 seconds
Thanks for the detailed trace — this explains why the change didn’t help. Why the error persists - Even with 
from prefect_dbt.cli.credentials import DbtCliProfile
, Python still imports the package 
prefect_dbt
first, which executes its 
__init__.py
. In 
prefect_dbt==0.3.1
, the package 
__init__
imports the dbt Cloud models (
DbtCloudCredentials
), so the Pydantic v2/v1-compat issue is still triggered during import time. Root cause - You still have Pydantic v2 installed. The path in your stack trace shows 
pydantic/v1/...
, which only exists in Pydantic v2 (it’s the v1-compat layer). Your constraint 
pydantic = "<2.11.4"
allows v2.10.x etc. - Prefect 2.x + prefect-dbt 0.3.x require Pydantic v1. Fix: pin Pydantic to v1 and reinstall - Update your pyproject to pin strictly below 2.0:
Copy code
pydantic = "<2"
- Then re-lock and install. With Poetry:
Copy code
poetry add "pydantic<2"
poetry lock --no-update
poetry install
- If Poetry still pulls v2 due to another dependency, identify it:
Copy code
poetry why pydantic
- If another package forces pydantic>=2, you’ll need to either: - Pin that package to a version compatible with pydantic v1, or - Upgrade to Prefect 3.x + latest prefect-dbt and run on pydantic v2. Verify your environment
Copy code
poetry run python -c "import pydantic, prefect, prefect_dbt; print('pydantic', pydantic.__version__, 'prefect', prefect.__version__, 'prefect-dbt', prefect_dbt.__version__)"
Expected for Prefect 2.x: 
pydantic 1.10.x
, 
prefect 2.20.16
, 
prefect-dbt 0.3.1
. Once pydantic is <2, your original imports will work (including the submodule import):
Copy code
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.commands import trigger_dbt_cli_command
If you’d like, paste the output of: - 
poetry why pydantic
- The version check snippet above I’ll help pinpoint any dependency forcing Pydantic v2 and suggest the exact pins to keep your 2.x stack stable.
s
@Marvin It is my mailersend package that requires pydantic v2 and it doesn’t support pydantic v1. Is there another way of being able run 
trigger_dbt_cli_command
without having to import dbt_cli_profile?
Copy code
def dbt_cmd(dbt_command, dbt_cli_profile, workdir):
    logger = get_run_logger()

    result = trigger_dbt_cli_command(
        dbt_command,
        dbt_cli_profile=dbt_cli_profile,
        overwrite_profiles=True,
        helper_command=f"cd {workdir}",
        return_all=True,
    )
    logger.info("\n".join(result))
    return result
m
✅ thought for 139.5 seconds
Short answer: yes. You don’t have to use or import 
DbtCliProfile
to run 
trigger_dbt_cli_command
. In current prefect-dbt (Prefect 3.x), 
trigger_dbt_cli_command
accepts 
project_dir
and 
profiles_dir
so you can point at your dbt project and an existing 
profiles.yml
on disk and skip 
DbtCliProfile
entirely. Also note: the function no longer supports 
helper_command
or 
return_all
. Example update based on your snippet:
Copy code
from prefect import get_run_logger
from prefect_dbt.cli.commands import trigger_dbt_cli_command

def dbt_cmd(dbt_command: str, workdir: str, profiles_dir: str | None = None):
    logger = get_run_logger()

    result = trigger_dbt_cli_command(
        command=dbt_command,
        project_dir=workdir,        # replaces your prior helper_command=cd ...
        profiles_dir=profiles_dir,  # directory that contains profiles.yml (or None to use default)
        overwrite_profiles=False,   # only True if you're programmatically writing profiles.yml
        stream_output=True,         # stream dbt output to Prefect logs
    )
    # Output is streamed to logs, so no need to join lines
    return result
Important notes: - Your previous 
helper_command
is not in the current API; use 
project_dir=workdir
. - 
return_all
is not in the current API; 
stream_output=True
streams logs to Prefect. - The return value is a structured result (not a list of strings), so 
"\n".join(result)
will not work in this version. About your Pydantic v2 constraint - If your environment contains packages requiring Pydantic v2 (e.g., mailersend), you should use a prefect-dbt version compatible with Pydantic v2. In Prefect 3.x, the integrations are aligned with Pydantic v2—upgrading to the latest 
prefect-dbt
is recommended. - Simply “not importing” 
DbtCliProfile
won’t resolve an installation-time conflict if your installed 
prefect-dbt
depends on Pydantic v1; for that, you need a compatible 
prefect-dbt
version or isolate environments. If upgrading 
prefect-dbt
isn’t possible, a fallback is to run dbt via a subprocess (no prefect-dbt dependency at all):
Copy code
import os
import subprocess
from prefect import get_run_logger

def dbt_cmd_cli(dbt_command: str, workdir: str, profiles_dir: str | None = None):
    logger = get_run_logger()
    env = os.environ.copy()
    if profiles_dir:
        env["DBT_PROFILES_DIR"] = profiles_dir
    proc = subprocess.run(
        dbt_command.split(),
        cwd=workdir,
        env=env,
        check=True,
        capture_output=True,
        text=True,
    )
    <http://logger.info|logger.info>(proc.stdout)
    return proc.returncode
Docs - Prefect dbt integration overview: Prefect dbt integration (3.x) Quick questions to make sure we point you the right way: - Which versions of Prefect and prefect-dbt are you using now? - Do you already have a 
profiles.yml
on disk (and where is it located)?
s
@Marvin I am using 
prefect = "2.20.16"
and 
prefect-dbt = "^0.3.1"
. My profiles.yml is not stored on disk
m
✅ thought for 79.8 seconds
Thanks — that context helps a lot. Key points for your setup (Prefect 2.20.16 + prefect-dbt 0.3.1): - You do not need to import or use 
DbtCliProfile
to run 
trigger_dbt_cli_command
. - In 0.3.1, 
trigger_dbt_cli_command
supports 
project_dir
and 
profiles_dir
and does not have 
helper_command
or 
return_all
. - Since you don’t have a 
profiles.yml
on disk, you can generate one at runtime (no DbtCliProfile involved) and point 
profiles_dir
to a temporary folder. Why you’re seeing conflicts - Prefect 2.x depends on Pydantic v1. - Your 
mailersend
package requires Pydantic v2. - That means a single environment cannot satisfy both without isolation. If you must keep Prefect 2.x, consider running your flows in an isolated runtime (e.g., a worker/agent environment or Docker image) that has Pydantic v1, while your app with mailersend runs elsewhere. Otherwise, upgrading to Prefect 3.x (Pydantic v2) resolves this class of conflicts. Option A — Stay on Prefect 2.x and avoid DbtCliProfile by writing profiles.yml at runtime
Copy code
from prefect import get_run_logger
from prefect_dbt.cli.commands import trigger_dbt_cli_command
import tempfile
import os
import yaml

def dbt_cmd(dbt_command: str, workdir: str, profile_name: str, target_name: str, target_config: dict):
    """
    profile_name: the name used in dbt_project.yml (profiles: <name>)
    target_name:  the target key to select (e.g., 'dev')
    target_config: the adapter config (depends on the warehouse adapter)
                   e.g., for Postgres:
                   {
                     "type": "postgres",
                     "host": "localhost",
                     "user": "user",
                     "password": "pass",
                     "port": 5432,
                     "dbname": "db",
                     "schema": "analytics"
                   }
    """
    logger = get_run_logger()

    with tempfile.TemporaryDirectory() as tmpdir:
        profiles_path = os.path.join(tmpdir, "profiles.yml")
        profiles = {
            profile_name: {
                "target": target_name,
                "outputs": {
                    target_name: target_config
                }
            }
        }
        with open(profiles_path, "w") as f:
            yaml.safe_dump(profiles, f)

        result = trigger_dbt_cli_command(
            command=dbt_command,
            project_dir=workdir,     # replaces the old helper_command=cd ...
            profiles_dir=tmpdir,     # points to the temp profiles.yml we just wrote
            stream_output=True,      # stream dbt output to Prefect logs
        )

    return result
- Replace your previous 
helper_command
with 
project_dir=workdir
. - Don’t use 
return_all
— it’s not part of the API. - The return value is not a list of strings; output is streamed when 
stream_output=True
. Option B — Upgrade to Prefect 3.x to align with Pydantic v2 - Prefect 3.x uses Pydantic v2, so it’ll play nicely with mailersend. - The latest 
prefect-dbt
for Prefect 3.x exposes the same general `project_dir`/`profiles_dir` approach and no longer includes the legacy params you mentioned. Docs - Prefect 2.x dbt CLI commands: prefect-dbt (2.x) CLI commands - Prefect 3.x dbt integration overview: prefect-dbt (3.x) integration If you share which adapter you’re using (Snowflake, Postgres, BigQuery, etc.), I can give you an exact 
target_config
template for the temporary profiles.yml. Also, do you plan to keep Prefect 2.x or are you open to upgrading to 3.x to resolve the Pydantic conflict entirely?
s
@Marvin I’m not using prefect-dbt 0.3.1 it is pinning it to a version above 0.3.1, so it could be the most recent version of prefect-dbt
m
✅ thought for 104.9 seconds
Got it — thanks for clarifying. A few important points based on your setup: - Mixing Prefect 2.20.16 with the most recent prefect-dbt is likely incompatible. Newer prefect-dbt releases target Prefect 3.x (and Pydantic v2). Prefect 2.x depends on Pydantic v1. - Not importing 
DbtCliProfile
won’t avoid a Pydantic resolver conflict if the installed prefect-dbt requires Pydantic v2. The conflict happens at install/resolve time. What you can do Option 1 — Stay on Prefect 2.x and avoid DbtCliProfile by writing profiles.yml at runtime - Pin prefect-dbt to a Prefect 2-compatible version (e.g., <1.0.0). - Because mailersend needs Pydantic v2, you’ll need to isolate environments (e.g., run your Prefect flow in a separate worker image/env that uses Pydantic v1, and keep mailersend in another env/service). Then call 
trigger_dbt_cli_command
without 
DbtCliProfile
by generating a temporary `profiles.yml`:
Copy code
from prefect import get_run_logger
from prefect_dbt.cli.commands import trigger_dbt_cli_command
import tempfile
import os
import yaml

def dbt_cmd(dbt_command: str, workdir: str, profile_name: str, target_name: str, target_config: dict):
    logger = get_run_logger()

    with tempfile.TemporaryDirectory() as tmpdir:
        profiles_path = os.path.join(tmpdir, "profiles.yml")
        profiles = {
            profile_name: {
                "target": target_name,
                "outputs": { target_name: target_config }
            }
        }
        with open(profiles_path, "w") as f:
            yaml.safe_dump(profiles, f)

        result = trigger_dbt_cli_command(
            command=dbt_command,
            project_dir=workdir,   # replaces helper_command=cd ...
            profiles_dir=tmpdir,   # points to the temp profiles.yml
            stream_output=True,    # stream dbt output to Prefect logs
        )

    return result
- Note: 
helper_command
and 
return_all
are not part of this API. Option 2 — Upgrade to Prefect 3.x - Prefect 3.x uses Pydantic v2 and aligns with most recent prefect-dbt. This resolves the Pydantic conflict with mailersend. - You can still avoid 
DbtCliProfile
and use the same `project_dir`/`profiles_dir` approach (or the temp profiles.yml technique above). Option 3 — Drop prefect-dbt and call dbt via subprocess - If you must keep Prefect 2.x and mailersend (Pydantic v2) in the same environment, removing the prefect-dbt dependency avoids the Pydantic constraint entirely:
Copy code
import os
import subprocess
from prefect import get_run_logger

def dbt_cmd_cli(dbt_command: str, workdir: str, profiles_dir: str | None = None):
    logger = get_run_logger()
    env = os.environ.copy()
    if profiles_dir:
        env["DBT_PROFILES_DIR"] = profiles_dir
    proc = subprocess.run(
        dbt_command.split(),
        cwd=workdir,
        env=env,
        check=True,
        capture_output=True,
        text=True,
    )
    <http://logger.info|logger.info>(proc.stdout)
    return proc.returncode
Next steps and a quick check - Can you confirm your exact installed 
prefect-dbt
version? You can run: - Python: 
import prefect_dbt; print(prefect_dbt.__version__)
- Or 
pip show prefect-dbt
- Are you open to upgrading to Prefect 3.x to resolve the Pydantic v2 conflict cleanly? Docs - Prefect 2.x dbt CLI commands: prefect-dbt (2.x) CLI commands - Prefect 3.x dbt integration: prefect-dbt (3.x) integration
s
@Marvin is it possible to temporarily bypass this issue by using @flow(validate_parameters=False) until I find a workaround?
m
✅ thought for 45.0 seconds
Short answer: you can try it, but it only helps in a narrow case. What @flow(validate_parameters=False) does - It disables Prefect’s parameter validation for that flow. This can bypass errors that occur specifically when Prefect builds/validates the flow’s parameter model. - It will not fix package-level incompatibilities (e.g., Prefect 2.x expecting Pydantic v1 while another package forces Pydantic v2). If your error happens at import/resolve time or elsewhere in runtime code that uses Pydantic, this won’t help. When it may help - If your failure is triggered only when the flow is called (or when Prefect builds the parameter schema) due to parameter type annotations that rely on Pydantic models, disabling validation can let the run proceed. - You may also need to set validate_parameters=False on any tasks that have problematic parameter types. When it won’t help - If the error happens when importing Prefect/prefect-dbt due to Pydantic v1/v2 API differences. - If runtime code (not just parameter validation) uses Pydantic models that are incompatible with your installed Pydantic. Safer temporary workarounds - Avoid Pydantic-typed parameters entirely; use plain types (str, int, dict) and do any validation inside the function. - If you must stay on Prefect 2.x, pin a Prefect-2-compatible prefect-dbt and isolate it in an environment that uses Pydantic v1; keep mailersend (Pydantic v2) in a separate env/service. - Or upgrade to Prefect 3.x so everything is on Pydantic v2. If you share the exact error message and where it occurs (import-time vs when calling the flow), I can tell you whether validate_parameters=False will actually help and suggest the minimal change to get you unblocked. Docs - Flows in Prefect 2.x: Flows (2.x)
s
@Marvin if I manage to get it to work without importing DbtCliProfile will it still fail as it needs to import 
trigger_dbt_cli_command
and on the third line at 
prefect_dbt/__init__.py
it is 
from .cloud import DbtCloudCredentials  # noqa
which is where it tries to use pydantic v1. Is there a way to avoid this line in the init?
m
✅ thought for 42.8 seconds
Short answer: No — importing any submodule like 
prefect_dbt.cli.commands
will execute 
prefect_dbt/__init__.py
first. There isn’t a supported way to “skip” that 
from .cloud import DbtCloudCredentials
line. So if that import is what triggers the Pydantic mismatch, it will still fail even if you never use 
DbtCliProfile
. Practical ways forward: - Cleanest short-term: avoid importing prefect-dbt entirely and call dbt via subprocess (set 
DBT_PROFILES_DIR
and 
cwd
), as shown earlier. This sidesteps the package and the Pydantic issue. - Stable long-term: - Either isolate environments: run your Prefect 2.x + prefect-dbt (<1.0, Pydantic v1) in a separate worker image/env; keep mailersend (Pydantic v2) elsewhere. - Or upgrade to Prefect 3.x so Prefect/prefect-dbt/mailersend all use Pydantic v2. If you absolutely must import 
trigger_dbt_cli_command
right now, there’s a very hacky, not-recommended shim that fakes the 
prefect_dbt.cloud
module so the top-level import doesn’t try to import the real Pydantic models. This can break in subtle ways, so only use it as a last resort to unblock temporarily:
Copy code
# WARNING: brittle hack to bypass top-level cloud import in prefect_dbt.__init__
import sys, types

dummy_cloud = types.ModuleType("prefect_dbt.cloud")
class Dummy: pass
dummy_cloud.DbtCloudCredentials = Dummy
sys.modules["prefect_dbt.cloud"] = dummy_cloud

from prefect_dbt.cli.commands import trigger_dbt_cli_command
Caveats: - If any other parts of prefect-dbt need real Cloud types, this will fail later. - This does not fix the underlying dependency conflict; it just dodges one import. Given the risk, I’d recommend either the subprocess path or isolating/upgrading environments. If you share your exact error traceback from the 
prefect_dbt/__init__.py
import, I can confirm whether this shim would even help and suggest the least risky workaround.
s
@Marvin Is there an old version of prefect_dbt that can be pinned that doesn’t have the 
from .cloud import DbtCloudCredentials
line to avoid this?
m
✅ thought for 466.8 seconds
Short answer: yes. Based on module inspection, the package-level re-export of 
DbtCloudCredentials
(which requires 
prefect_dbt/__init__.py
to do something like 
from .cloud import DbtCloudCredentials
) appears in prefect-dbt 0.7.0+. Pinning to 0.6.4 (or earlier) should avoid that top-level import. - Recommended pin: 
prefect-dbt<=0.6.4
- In those versions, 
DbtCloudCredentials
still exists, but only under the submodule; import it where needed with:
Copy code
from prefect_dbt.cloud import DbtCloudCredentials
References: - PyPI project: prefect-dbt on PyPI - GitHub repo: PrefectHQ/prefect-dbt - For comparison, 0.7.0 exposes 
DbtCloudCredentials
at the package root (implying the re-export): prefect_dbt 0.7.0 If you can share the exact error you’re trying to avoid, I can suggest a more targeted workaround (e.g., lazy importing, restructuring imports, or a specific compatible version).
s
@Marvin this is the specific error:
Copy code
File "/var/opt/circleci/workdir/flows/dbt/anonymization_dbt_run_flow.py", line 5, in <module>
    from prefect_dbt.cli.credentials import DbtCliProfile
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect_dbt/__init__.py", line 3, in <module>
    from .cloud import DbtCloudCredentials  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect_dbt/cloud/__init__.py", line 1, in <module>
    from .credentials import DbtCloudCredentials  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect_dbt/cloud/credentials.py", line 14, in <module>
    class DbtCloudCredentials(CredentialsBlock):
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 197, in __new__
    fields[ann_name] = ModelField.infer(
                       ^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 504, in infer
    return cls(
           ^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 434, in __init__
    self.prepare()
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 555, in prepare
    self.populate_validators()
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/fields.py", line 829, in populate_validators
    *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))),
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/pydantic/v1/validators.py", line 768, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect/flows.py", line 1737, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 205, in import_object
    module = load_script_as_module(script_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/opt/circleci/workdir/.venv/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 168, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/dbt/anonymization_dbt_run_flow.py' encountered an exception: RuntimeError("no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config")
m
✅ thought for 83.4 seconds
Thanks for sharing the traceback — that helps a lot. What’s happening: - Your code imports 
prefect_dbt.cli.credentials.DbtCliProfile
, but Python still executes 
prefect_dbt/__init__.py
first. - In newer releases, 
prefect_dbt/__init__.py
re-exports 
DbtCloudCredentials
via 
from .cloud import DbtCloudCredentials
, which triggers Pydantic model construction on import. - In your env, that hits a Pydantic v1/v2 mismatch and crashes with “no validator found for SecretStr”. Two ways to fix it: 1) If you’re on Prefect 3.x (Pydantic v2), use the Prefect 3-compatible prefect-dbt - Upgrade to a version that is Pydantic v2 compatible so the import won’t crash:
Copy code
pip install -U "prefect>=3" "pydantic>=2" "prefect-dbt>=0.7.0"
- Docs: Prefect 3 + dbt integration 2) If you just want to avoid that top-level import - Pin to a version before the re-export existed. Based on inspection, 0.7.0+ re-exports 
DbtCloudCredentials
at the package root, while 0.6.4 and earlier do not.
Copy code
pip install "prefect-dbt<=0.6.4"
- Important: those versions target Prefect 2.x and Pydantic 1.x. If you go this route, align the rest of your stack:
Copy code
pip install "prefect<3" "pydantic<2" "prefect-dbt<=0.6.4"
Can you share the versions you’re on? Please run:
Copy code
python -V
pip show prefect prefect-dbt pydantic
With that, I can recommend the exact compatible set. References: - PyPI: prefect-dbt - Repo: PrefectHQ/prefect-dbt - Example of 0.7.0 package (where 
DbtCloudCredentials
is re-exported): prefect_dbt 0.7.0
s
@Marvin My pydantic version is v2 as I need it to be v2 for my mailersend package. Is it possible to use 
prefect-dbt = 0.6.4
with pydantic v2?
m
✅ thought for 47.5 seconds
Short answer: No — prefect-dbt 0.6.4 targets Prefect 2.x and Pydantic v1. Running it in an environment with Pydantic v2 is not supported and is exactly what’s triggering your “no validator found for SecretStr” error. Your options: - Recommended: Move to Prefect 3 + prefect-dbt >= 0.7.0 (built for Pydantic v2)
Copy code
pip install -U "prefect>=3" "pydantic>=2" "prefect-dbt>=0.7.0"
Docs: Prefect 3 dbt integration - If you must stay on 0.6.x: isolate it in a separate environment with Pydantic v1 and Prefect 2.x
Copy code
pip install "prefect<3" "pydantic<2" "prefect-dbt<=0.6.4"
This avoids the top-level import and keeps all dependencies aligned. A fragile workaround (not recommended) if you’re blocked and experimenting: - Monkey-patch the v1 SecretStr into pydantic before importing prefect_dbt so downstream imports pick up the v1 class. This can sometimes bypass the mismatch, but it’s brittle and can break unexpectedly.
Copy code
try:
      import pydantic
      from pydantic.v1 import SecretStr as V1SecretStr
      pydantic.SecretStr = V1SecretStr
  except Exception:
      pass

  from prefect_dbt.cli.credentials import DbtCliProfile
Again, this is not a supported path. If you can share your Prefect version (
pip show prefect
), I can give you an exact, compatible set of pins.
Open in Slack
PreviousNext
Powered by