https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> how do i programmatically create the ...
# ask-marvin
a
@Marvin how do i programmatically create the DatabaseCredentials block?
m
✅ thought for 159.5 seconds
In Prefect 3.x, there isn't a 
DatabaseCredentials
block to create. What you want instead is 
SqlAlchemyConnector
from the 
prefect-sqlalchemy
integration. Example: create and save it programmatically
Copy code
python
from prefect_sqlalchemy import SqlAlchemyConnector, ConnectionComponents, SyncDriver

connector = SqlAlchemyConnector(
    connection_info=ConnectionComponents(
        driver=SyncDriver.POSTGRESQL_PSYCOPG2,
        username="myuser",
        password="mypassword",
        host="localhost",
        port=5432,
        database="mydatabase",
    )
)

connector.save("my-database-block")
Later, you can load it with:
Copy code
python
from prefect_sqlalchemy import SqlAlchemyConnector

connector = SqlAlchemyConnector.load("my-database-block")
SQLite example
Copy code
python
from prefect_sqlalchemy import SqlAlchemyConnector, ConnectionComponents, SyncDriver

connector = SqlAlchemyConnector(
    connection_info=ConnectionComponents(
        driver=SyncDriver.SQLITE_PYSQLITE,
        database="example.db",
    )
)

connector.save("sqlite-block")
A couple of notes: - 
password
is handled as a secret field by the block schema. - 
DatabaseCredentials
appears to be an older/historical name, not a current Prefect 3.x block. - You'll need the integration installed:
Copy code
bash
pip install "prefect[sqlalchemy]"
You may also need to register the block types once:
Copy code
bash
prefect block register -m prefect_sqlalchemy
Relevant docs: - https://prefecthq.github.io/prefect-sqlalchemy/database/ - https://docs.prefect.io/latest/concepts/blocks/ If you want, I can also show: - how to create this from a single connection URL, or - how to use the saved block inside a flow/task.
a
@Marvin how do i do this with dbt CLI Target Configs block ?
m
✅ thought for 85.1 seconds
Yes — for dbt, there is a block class you can create programmatically: 
TargetConfigs
from 
prefect-dbt
. Import
Copy code
python
from prefect_dbt.cli.configs import TargetConfigs
Create and save a generic dbt CLI Target Configs block
Copy code
python
from prefect_dbt.cli.configs import TargetConfigs

target_configs = TargetConfigs(
    type="postgres",
    schema_="analytics",   # use schema_ in Python; dbt receives "schema"
    threads=4,
    extras={
        "host": "localhost",
        "port": 5432,
        "user": "myuser",
        "password": "mypassword",
        "dbname": "mydatabase",
    },
)

target_configs.save("my-dbt-target-configs", overwrite=True)
Load it later
Copy code
python
from prefect_dbt.cli.configs import TargetConfigs

target_configs = TargetConfigs.load("my-dbt-target-configs")
print(target_configs.get_configs())
That 
get_configs()
call returns a dbt-style target config dict, like:
Copy code
python
{
    "type": "postgres",
    "schema": "analytics",
    "threads": 4,
    "host": "localhost",
    "port": 5432,
    "user": "myuser",
    "password": "mypassword",
    "dbname": "mydatabase",
}
Database-specific variants also exist, for example: - 
SnowflakeTargetConfigs
- 
BigQueryTargetConfigs
- 
PostgresTargetConfigs
Example import:
Copy code
python
from prefect_dbt.cli.configs import PostgresTargetConfigs
A couple of important notes: - This is part of the 
prefect-dbt
integration, not core Prefect. - The 
prefect_dbt.cli
module is deprecated in favor of 
prefect_dbt.core
, but 
TargetConfigs
still exists and is usable in Prefect 3.x. - If you're specifically using dbt CLI Target Configs block in existing code, this is the correct object to create. You’ll need the integration installed:
Copy code
bash
pip install prefect-dbt
Helpful links: - prefect-dbt CLI configs API - prefect-dbt integration docs If you want, I can also give you: - a PostgresTargetConfigs example, - a SnowflakeTargetConfigs example, or - how to wire this into a 
DbtCoreOperation
/ modern Prefect 3 dbt workflow.
a
im getting this error:
Copy code
from dbt.artifacts.resources.types import NodeType
ModuleNotFoundError: No module named 'dbt.artifacts'
@Marvin im getting this error:
Copy code
from dbt.artifacts.resources.types import NodeType
ModuleNotFoundError: No module named 'dbt.artifacts'
m
✅ thought for 110.1 seconds
That error usually means your environment is missing a compatible 
dbt-core
install. 
dbt.artifacts.resources.types
comes from 
dbt-core
, and 
prefect-dbt
expects a newer dbt version that includes it. Most likely causes - 
dbt-core
is not installed - 
dbt-core
is installed, but it's too old Recommended fix
Copy code
bash
pip install "prefect[dbt]"
If you also need a specific adapter, install that too. For example:
Copy code
bash
pip install "prefect[dbt]" dbt-postgres
or:
Copy code
bash
pip install "prefect[dbt]" dbt-snowflake
Check what version you have now
Copy code
bash
pip show dbt-core
You want 
dbt-core >= 1.7.0
. You can also verify directly:
Copy code
bash
python -c "from dbt.artifacts.resources.types import NodeType; print(NodeType)"
If that fails, your 
dbt-core
install is the issue. If you want to force-upgrade cleanly
Copy code
bash
pip install -U "dbt-core>=1.7.0"
Or, often better, reinstall the relevant set together in a fresh venv:
Copy code
bash
pip install -U prefect prefect-dbt dbt-core dbt-postgres
One more important note: 
prefect_dbt.cli
is deprecated, so if you're using older examples around CLI config blocks, it may be worth checking whether you can move to the newer 
prefect_dbt.core
pattern. Docs: - prefect-dbt integration docs - dbt-core on PyPI If you paste your 
pip show prefect prefect-dbt dbt-core
output, I can tell you exactly which package/version mismatch is causing it.
a
@Marvin my dbt core version is 1.11.9
m
✅ thought for 9.6 seconds
If your 
dbt-core
is really 
1.11.9
, then the likely issue is not the version itself — it's that you're running Prefect code in a different Python environment than the one where 
dbt-core
1.11.9 is installed. Also, 
dbt-core
1.11 is newer than what 
prefect-dbt
was originally built around, so there may also be a compatibility mismatch depending on your 
prefect-dbt
version. Most likely possibilities - 
python
/ 
pip
point to different environments - your worker/runtime env differs from your shell env - 
prefect-dbt
is old and incompatible with 
dbt-core 1.11
- another package named 
dbt
is shadowing the real one Please run these in the exact environment where the error happens:
Copy code
bash
python -c "import sys; print(sys.executable)"
python -c "import dbt; print(dbt.__file__)"
python -c "from dbt.artifacts.resources.types import NodeType; print(NodeType)"
python -c "import prefect_dbt; print(prefect_dbt.__file__)"
pip show prefect prefect-dbt dbt-core
That will tell us whether: 1. the module is actually importable there, and 2. which installed 
prefect-dbt
version is trying to use it. I’d also strongly suggest checking your 
prefect-dbt
version specifically. If it’s older, try upgrading it:
Copy code
bash
pip install -U prefect-dbt
Or install the aligned bundle:
Copy code
bash
pip install -U "prefect[dbt]"
If this is happening inside a deployment/worker, make sure the worker image or runtime environment has the same packages as your local shell. That mismatch is very common. If you send me the output of:
Copy code
bash
pip show prefect prefect-dbt dbt-core
and
Copy code
bash
python -c "import dbt; print(dbt.__file__)"
I can narrow it down quickly.
a
prefect-dbt is 0.7.16
@Marvin prefect-dbt is 0.7.16
m
✅ thought for 116.3 seconds
Thanks — that helps. 
prefect-dbt 0.7.16
should be compatible with 
dbt-core 1.11.9
, so this points even more strongly to an environment/import-path problem rather than a simple version mismatch. Given your error:
Copy code
python
from dbt.artifacts.resources.types import NodeType
ModuleNotFoundError: No module named 'dbt.artifacts'
and your reported version, the most likely explanations are: Most likely causes - 
prefect-dbt
and 
dbt-core
are installed in different environments - the code is running under a worker/container/venv that does not actually have 
dbt-core 1.11.9
- a local package/file named 
dbt.py
or 
dbt/
is shadowing the real 
dbt-core
package - 
pip show dbt-core
was run in a different interpreter than the one running Prefect Please run these in the exact runtime where the failure occurs:
Copy code
bash
python -c "import sys; print(sys.executable)"
python -c "import dbt; print(dbt, dbt.__file__)"
python -c "import importlib.util; print(importlib.util.find_spec('dbt.artifacts'))"
python -c "from dbt.artifacts.resources.types import NodeType; print(NodeType)"
python -c "import prefect_dbt; print(prefect_dbt.__file__)"
pip show prefect prefect-dbt dbt-core
What I expect - If 
find_spec('dbt.artifacts')
returns 
None
, then the runtime environment does not have the expected dbt package layout. - If 
import dbt; print(dbt.__file__)
points at something in your project directory, then you have a shadowing problem. - If 
sys.executable
is not the interpreter you expect, then this is an env mismatch. A very common fix Use the same interpreter for install and run:
Copy code
bash
python -m pip install -U prefect prefect-dbt dbt-core
If you need an adapter too:
Copy code
bash
python -m pip install -U prefect prefect-dbt dbt-core dbt-postgres
That avoids 
pip
vs 
python
mismatches. Also worth checking If this happens in a deployment or worker, inspect the worker image/environment, not just your local shell. Local package versions often differ from the runtime actually executing the flow. If you want, paste the output of these two commands and I can pinpoint it quickly:
Copy code
bash
python -c "import dbt; print(dbt, dbt.__file__)"
python -c "import importlib.util; print(importlib.util.find_spec('dbt.artifacts'))"
a
@Marvin second one is none. First one : <module ‘dbt’ from ‘/a/flows/dbt/__init__.py’> /a/flows/dbt/__init__.py
m
✅ thought for 10.8 seconds
That’s the problem: your own project folder is shadowing the real 
dbt-core
package. Python is importing:
Copy code
python
/a/flows/dbt/__init__.py
instead of the installed 
dbt-core
package. Since your local 
dbt
package doesn’t contain 
artifacts
, 
from dbt.artifacts...
fails. Fix Rename your local 
dbt
package/directory to something else, e.g.: - 
dbt_project
- 
dbt_assets
- 
my_dbt
- 
analytics_dbt
Then make sure there are no stale 
.pyc
files / 
__pycache__
left behind. For example:
Copy code
bash
mv /a/flows/dbt /a/flows/dbt_project
find /a/flows -type d -name "__pycache__" -exec rm -rf {} +
Then verify again:
Copy code
bash
python -c "import dbt; print(dbt, dbt.__file__)"
python -c "import importlib.util; print(importlib.util.find_spec('dbt.artifacts'))"
python -c "from dbt.artifacts.resources.types import NodeType; print(NodeType)"
After the rename, 
import dbt
should point into your site-packages, not 
/a/flows/dbt/...
. So in short: this is not a Prefect bug or a dbt version issue — it’s a Python module name collision. If you want, I can also help you identify the safest rename pattern for a Prefect project structure.
a
@Marvin works. how do i ensure the extras is json format
m
✅ thought for 11.8 seconds
If you mean the 
extras
field on 
TargetConfigs
, the simplest way is: - pass a normal Python 
dict
- make sure all keys are strings - make sure all values are JSON-serializable For example:
Copy code
python
from prefect_dbt.cli.configs import TargetConfigs
import json

extras = {
    "host": "localhost",
    "port": 5432,
    "user": "myuser",
    "password": "mypassword",
    "dbname": "mydatabase",
    "sslmode": "require",
}

# optional validation step
json.dumps(extras)

target_configs = TargetConfigs(
    type="postgres",
    schema_="analytics",
    threads=4,
    extras=extras,
)

target_configs.save("my-dbt-target-configs", overwrite=True)
If 
json.dumps(extras)
succeeds, then your 
extras
is valid JSON-compatible data. Good JSON-compatible value types - strings - numbers - booleans - 
None
- lists of the above - dicts of the above Bad/non-JSON-safe examples - 
datetime
objects - 
Path
objects - sets - custom classes - bytes Example of converting non-JSON-safe values:
Copy code
python
from pathlib import Path

extras = {
    "path": str(Path("/tmp/my.duckdb")),
    "threads": 4,
}
If you want to be strict, you can wrap validation in a helper:
Copy code
python
import json

def ensure_json_dict(value):
    if not isinstance(value, dict):
        raise TypeError("extras must be a dict")
    try:
        json.dumps(value)
    except TypeError as exc:
        raise ValueError(f"extras must be JSON-serializable: {exc}") from exc
    return value
Usage:
Copy code
python
extras = ensure_json_dict({
    "host": "localhost",
    "port": 5432,
})
If your goal is specifically to store a JSON string instead of a Python dict, you’d do:
Copy code
python
json_string = json.dumps(extras)
but for 
TargetConfigs(extras=...)
, you usually want a dict, not a JSON string. Docs: - prefect-dbt TargetConfigs API If you want, I can show a concrete example for the exact dbt target type you’re using, like Snowflake, Postgres, BigQuery, or DuckDB.
a
@Marvin what if i want to create a dbt cli profile the same way, that references the target_config
m
✅ thought for 17.5 seconds
You generally don’t create a separate Prefect block for a full dbt 
profiles.yml
that “references” a 
TargetConfigs
block. In the 
prefect-dbt
CLI-style pattern, the usual approach is: - store the target details in a 
TargetConfigs
block - load that block in Python - call 
get_configs()
- build the dbt profile dictionary yourself - optionally write it to 
profiles.yml
Example
Copy code
python
from pathlib import Path
import yaml

from prefect_dbt.cli.configs import TargetConfigs

target_configs = TargetConfigs.load("my-dbt-target-configs")

profile = {
    "my_profile": {
        "target": "dev",
        "outputs": {
            "dev": target_configs.get_configs()
        },
    }
}

profiles_dir = Path("./.dbt")
profiles_dir.mkdir(parents=True, exist_ok=True)

with open(profiles_dir / "profiles.yml", "w") as f:
    yaml.safe_dump(profile, f, sort_keys=False)
That produces a dbt profile shaped like:
Copy code
python
{
    "my_profile": {
        "target": "dev",
        "outputs": {
            "dev": {
                "type": "postgres",
                "schema": "analytics",
                "threads": 4,
                "host": "localhost",
                "port": 5432,
                "user": "myuser",
                "password": "mypassword",
                "dbname": "mydatabase",
            }
        }
    }
}
A helper function version
Copy code
python
from pathlib import Path
import yaml

def write_dbt_profile(profile_name, target_name, target_configs_block_name, profiles_dir=".dbt"):
    from prefect_dbt.cli.configs import TargetConfigs

    target_configs = TargetConfigs.load(target_configs_block_name)

    profile = {
        profile_name: {
            "target": target_name,
            "outputs": {
                target_name: target_configs.get_configs()
            },
        }
    }

    profiles_path = Path(profiles_dir)
    profiles_path.mkdir(parents=True, exist_ok=True)

    with open(profiles_path / "profiles.yml", "w") as f:
        yaml.safe_dump(profile, f, sort_keys=False)

    return profiles_path / "profiles.yml"
Usage:
Copy code
python
path = write_dbt_profile(
    profile_name="my_profile",
    target_name="dev",
    target_configs_block_name="my-dbt-target-configs",
)
print(path)
A couple of important notes - dbt itself expects a real 
profiles.yml
file or equivalent profile directory. - 
TargetConfigs
is a Prefect block, but dbt does not know how to “reference” a Prefect block directly inside YAML. - So the reference happens in Python code, not natively inside 
profiles.yml
. If you want to keep it all in Prefect, you could also save the whole generated YAML as a 
Secret
, 
JSON
, or 
String
-type block, but usually that’s less useful than just generating the file from 
TargetConfigs
at runtime. If you want, I can show: - how to do this in a flow/task, - how to write multiple outputs (`dev`/`prod`) into one profile, or - how to use 
DbtCoreOperation
with the generated profile.
a
@Marvin blocks in the UI have “dbt CLI Profile”, how can create one of this programatically
m
✅ thought for 97.3 seconds
Yes — that UI block is implemented by 
DbtCliProfile
. Import paths
Copy code
python
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.configs import TargetConfigs, GlobalConfigs
Basic example: create a 
dbt CLI Profile
block programmatically
Copy code
python
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.configs import TargetConfigs, GlobalConfigs

target_configs = TargetConfigs(
    type="postgres",
    schema_="analytics",
    threads=4,
    extras={
        "host": "localhost",
        "port": 5432,
        "user": "myuser",
        "password": "mypassword",
        "dbname": "mydatabase",
    },
)

global_configs = GlobalConfigs(
    use_colors=False,
    send_anonymous_usage_stats=False,
)

profile = DbtCliProfile(
    name="my_dbt_profile",      # profile name in profiles.yml
    target="dev",               # output name
    target_configs=target_configs,
    global_configs=global_configs,
)

profile.save("my-dbt-cli-profile", overwrite=True)
Load it later
Copy code
python
from prefect_dbt.cli.credentials import DbtCliProfile

profile = DbtCliProfile.load("my-dbt-cli-profile")
Generate the dbt profile dict from it
Copy code
python
profile_dict = profile.get_profile()
print(profile_dict)
That will produce something shaped like:
Copy code
python
{
    "config": {
        "use_colors": False,
        "send_anonymous_usage_stats": False,
    },
    "my_dbt_profile": {
        "target": "dev",
        "outputs": {
            "dev": {
                "type": "postgres",
                "schema": "analytics",
                "threads": 4,
                "host": "localhost",
                "port": 5432,
                "user": "myuser",
                "password": "mypassword",
                "dbname": "mydatabase",
            }
        }
    }
}
A few important notes - 
DbtCliProfile
is the block behind the UI label “dbt CLI Profile”. - It contains a 
target_configs
object, so yes, this is how you reference the target config programmatically. - 
prefect_dbt.cli
is deprecated, but this block still exists and works in current Prefect 3.x setups. If you already have a saved 
TargetConfigs
block and want to reuse it, you can do:
Copy code
python
from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.configs import TargetConfigs

target_configs = TargetConfigs.load("my-dbt-target-configs")

profile = DbtCliProfile(
    name="my_dbt_profile",
    target="dev",
    target_configs=target_configs,
)

profile.save("my-dbt-cli-profile", overwrite=True)
Docs: - prefect-dbt integration docs - `DbtCliProfile` API If you want, I can give you the exact snippet for: - Postgres - Snowflake - BigQuery - or how to write the saved 
DbtCliProfile
back out to a 
profiles.yml
file.
a
@Marvin how do i create a secret this way
m
✅ thought for 9.2 seconds
You can create a Secret block programmatically with 
prefect.blocks.system.Secret
. Example
Copy code
python
from prefect.blocks.system import Secret

secret = Secret(value="super-secret-value")
secret.save("my-secret", overwrite=True)
Load it later
Copy code
python
from prefect.blocks.system import Secret

secret = Secret.load("my-secret")
value = secret.get()
print(value)
If you want to use it for something like a dbt password:
Copy code
python
from prefect.blocks.system import Secret

db_password = Secret(value="mypassword")
db_password.save("dbt-password", overwrite=True)
Then later:
Copy code
python
from prefect.blocks.system import Secret

password = Secret.load("dbt-password").get()
A couple notes - 
save("my-secret")
is the block document name you’ll see in the UI. - Use 
.get()
to retrieve the underlying secret value. - The value is masked in the UI. Docs: - Prefect blocks docs - system blocks API If you want, I can also show how to: - store a JSON payload as a secret, - use a secret inside a 
DbtCliProfile
, - or create the same block via CLI/UI.
2 Views
Open in Slack
PreviousNext
Powered by