https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm testing this task but it is faili...
# marvin-in-the-wild
m
@Marvin I'm testing this task but it is failing with an error 
Could not find profile named 'default'
Copy code
@task
def run_dbt_models(
    host: str,
    user: str,
    database: str,
    password: str,
    port: int,
    schema: str,
    provider: str = "postgres",
    threads: int = 4,
):
    logger = get_run_logger()

    <http://logger.info|logger.info>("Fetching DBT CLI Profile...")

    target_configs_extras = dict(
        host=host,
        user=user,
        password=password,
        port=port,
        dbname=database,
    )
    target_configs = TargetConfigs(
        type=provider, schema=schema, threads=4, extras=target_configs_extras
    )

    dbt_cli_profile = DbtCliProfile(
        name="pomera_data_provisioning_dbt_profile",
        target="target",
        target_configs=target_configs,
    )

    <http://logger.info|logger.info>(f"DBT CLI Profile is {dbt_cli_profile.get_profile()}")

    <http://logger.info|logger.info>("Triggering DBT Core Operation...")

    with DbtCoreOperation(
        commands=["dbt build"],
        stream_output=True,
        overwrite_profiles=True,
        working_directory="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA",
        project_dir="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA",
        dbt_cli_profile=dbt_cli_profile,
    ) as dbt_operation:
        dbt_process = dbt_operation.trigger()
        dbt_process.wait_for_completion()
        result = dbt_process.fetch_result()
    return result
m
Your message was too long by 67 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
I'm testing this task but it is failing with an error 
Could not find profile named 'default'
```@task def run_dbt_models( host: str, user: str, database: str, password: str, port: int, schema: str, provider: str = "postgres", threads: int = 4, ): logger = get_run_logger() logger.info("Fetching DBT CLI Profile...") target_configs_extras = dict( host=host, user=user, password=password, port=port, dbname=database, ) target_configs = TargetConfigs( type=provider, schema=schema, threads=4, extras=target_configs_extras ) dbt_cli_profile = DbtCliProfile( name="pomera_data_provisioning_dbt_profile", target="target", target_configs=target_configs, ) logger.info(f"DBT CLI Profile is {dbt_cli_profile.get_profile()}") logger.info("Triggering DBT Core Operation...") with DbtCoreOperation( commands=["dbt build"], stream_output=True, overwrite_profiles=True, working_directory="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA", project_dir="/Users
m
@Marvin I'm getting an error 
Could not find profile named 'default'
for this task: @task def run_dbt_models( host: str, user: str, database: str, password: str, port: int, schema: str, provider: str = "postgres", threads: int = 4, ): target_configs_extras = dict(host=host, user=user, password=password, port=port, dbname=database, ) target_configs = TargetConfigs(type=provider, schema=schema, threads=4, extras=target_configs_extras ) dbt_cli_profile = DbtCliProfile(name="pomera_data_provisioning_dbt_profile", target="target", target_configs=target_configs, ) with DbtCoreOperation( commands=["dbt build"], stream_output=True, overwrite_profiles=True, working_directory="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA", project_dir="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA", dbt_cli_profile=dbt_cli_profile, ) as dbt_operation: dbt_process = dbt_operation.trigger() dbt_process.wait_for_completion() result = dbt_process.fetch_result() return result
m
It looks like the error "Could not find profile named 'default'" could be due to several reasons, often related to the configuration of the Dbt profile or Prefect environment. Here are a few steps to troubleshoot and resolve this issue: 1. Check Dbt Profile Configuration: Ensure the 
dbt_project.yml
and profiles in the directory specified by 
dbt_cli_profile
are correctly set up. Prefect will need to find the appropriate profiles in these directories. 2. Setting Profiles Directory: When using 
DbtCoreOperation
, make sure the 
profiles_dir
parameter is correctly set to the directory containing your Dbt profiles. From the document excerpt, it looks like:
Copy code
python
   with DbtCoreOperation(
       commands=["dbt build"],
       stream_output=True,
       overwrite_profiles=True,
       working_directory="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA",
       project_dir="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA",
       profiles_dir="/PATH/TO/DBT/PROFILES",  # Add this line
       dbt_cli_profile=dbt_cli_profile,
   ) as dbt_operation:
3. Profile Name: Make sure that the profile name specified in 
DbtCliProfile(name="pomera_data_provisioning_dbt_profile", ...)
is correctly defined in your Dbt profiles.yml file. 4. Prefect Profiles: It may also be related to Prefect's settings or active profile. Set and inspect profiles from the CLI:
Copy code
bash
   prefect profile ls
   prefect profile use <your-profile>
   prefect profile inspect <your-profile>
### References: 1. Prefect Profiles Guide 2. Prefect DBT - Examples Catalog Make sure your environment and paths are correctly configured and match the expected settings for both Prefect and Dbt. If you still encounter issues, consider sharing specific setup details alongside seeking more targeted help from the community or official support channels. Let me know if this helps or if you need further assistance!
m
@Marvin I don't have a profiles.yaml created, I'm expecting that it's the one I'm passing via DbtCliProfile?
m
Ah, I see the crux of the problem now. When you're using 
DbtCoreOperation
and passing a custom 
DbtCliProfile
, it should indeed use the configurations provided in that profile dynamically without needing a 
profiles.yaml
on disk. Given that you're encountering the error, it seems there might be an issue with how the 
DbtCliProfile
is being defined or passed. Let's go through an example of correctly configuring and using 
DbtCliProfile
. ### Example of Using 
DbtCliProfile
Here's a refined version of how you might define and use the `DbtCliProfile`:
Copy code
python
from prefect import task
from prefect_dbt.cli import DbtCoreOperation, DbtCliProfile
from prefect_dbt.cli.configs import TargetConfigs

@task
def run_dbt_models(
    host: str, user: str, database: str, password: str, port: int, schema: str, provider: str = "postgres", threads: int = 4
):
    target_configs_extras = dict(
        host=host, user=user, password=password, port=port, dbname=database
    )
    target_configs = TargetConfigs(
        type=provider, schema=schema, threads=threads, extras=target_configs_extras
    )
    
    dbt_cli_profile = DbtCliProfile(
        name="pomera_data_provisioning_dbt_profile",
        target="target",
        target_configs=target_configs,
    )
    
    with DbtCoreOperation(
        commands=["dbt build"],
        stream_output=True,
        overwrite_profiles=True,
        working_directory="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA",
        project_dir="/Users/mmoncadaisla/Cambium/data-canopy/dbt/DP_POMERA",
        profiles_dir=None,  # Explicitly set profiles_dir to None
        dbt_cli_profile=dbt_cli_profile,
    ) as dbt_operation:
        dbt_process = dbt_operation.trigger()
        dbt_process.wait_for_completion()
        result = dbt_process.fetch_result()
    
    return result
### Key Points: 1. 
profiles_dir
Parameter: Explicitly set 
profiles_dir
to 
None
to ensure it uses the profile defined in 
DbtCliProfile
. 2. Target Configuration: Ensure the 
TargetConfigs
and other parameters match what 
dbt
expects. 3. Stream Output and Overwrite Profiles: These configurations help debug and force using the provided profile. ### Documentation References: - Prefect Dbt CLI Configuration - Example from Prefect's Dbt documentation and ensuring alignment with how configurations are expected. Make sure all dependencies are correctly installed, and paths provided are accurate. This setup should help resolve the issue provided configurations are correctly set in the 
DbtCliProfile
. If issues persist, it might be helpful to check Prefect and dbt logs for more details or share those here for more specific troubleshooting.
m
@Marvin still does not work
m
Let's troubleshoot this step-by-step to pinpoint the exact issue. Here are the steps: ### Step 1: Verify Dbt CLI Profile Configuration Ensure that your 
DbtCliProfile
is correctly defined. Each field must be accurately set as expected by dbt. Here’s a minimal example:
Copy code
python
from prefect_dbt.cli.configs import TargetConfigs
from prefect_dbt.cli.profiles import DbtCliProfile

target_configs_extras = {
    "host": "your_host",
    "user": "your_user",
    "password": "your_password",
    "port": your_port,
    "dbname": "your_database"
}

target_configs = TargetConfigs(
    type="postgres",  # or the correct type
    schema="your_schema",
    threads=4,
    extras=target_configs_extras
)

dbt_cli_profile = DbtCliProfile(
    name="my_dbt_profile",  # This should be a unique and identifiable name
    target="dev",  # This should match the target section in the profile
    target_configs=target_configs
)
### Step 2: Ensure Correct Directories Double-check the paths: - `working_directory`: Should be where your dbt project's 
dbt_project.yml
file is located. - `project_dir`: Should also point to the directory containing 
dbt_project.yml
. ### Step 3: Update 
DbtCoreOperation
Usage Here's the complete code snippet including 
DbtCoreOperation
configuration:
Copy code
python
from prefect import task
from prefect_dbt.cli import DbtCoreOperation
from prefect_dbt.cli.configs import TargetConfigs
from prefect_dbt.cli.profiles import DbtCliProfile

@task
def run_dbt_models(
    host: str, user: str, database: str, password: str, port: int, schema: str, provider: str = "postgres", threads: int = 4
):
    target_configs_extras = dict(
        host=host, user=user, password=password, port=port, dbname=database
    )
    target_configs = TargetConfigs(
        type=provider, schema=schema, threads=threads, extras=target_configs_extras
    )
    
    dbt_cli_profile = DbtCliProfile(
        name="my_dbt_profile",
        target="dev",
        target_configs=target_configs,
    )
    
    with DbtCoreOperation(
        commands=["dbt build"],
        stream_output=True,
        overwrite_profiles=True,
        working_directory="/path/to/dbt/project",
        project_dir="/path/to/dbt/project",
        profiles_dir=None,  # Explicitly set to None
        dbt_cli_profile=dbt_cli_profile,
    ) as dbt_operation:
        dbt_process = dbt_operation.trigger()
        dbt_process.wait_for_completion()
        result = dbt_process.fetch_result()
    
    return result
### Step 4: Prefect Profile Management Ensure Prefect is using the correct profile. You can check the current active profile and list all available profiles:
Copy code
sh
prefect profile ls
prefect profile use default
Additionally, inspect the profile settings:
Copy code
sh
prefect profile inspect default
### Step 5: Debugging Run the task with debugging enabled to capture detailed logs:
Copy code
python
import prefect
from prefect import flow
from prefect.logging import get_run_logger

@flow
def main_flow():
    logger = get_run_logger()
    try:
        result = run_dbt_models(
            host="your_host",
            user="your_user",
            database="your_database",
            password="your_password",
            port=your_port,
            schema="your_schema"
        )
        <http://logger.info|logger.info>(f"DBT Run Result: {result}")
    except Exception as e:
        logger.error(f"Error running DBT models: {str(e)}")

main_flow()
Incorporate these steps and refine as needed. Make sure paths, configurations, and environment settings are accurate. This should help in identifying and resolving the core issue. If it persists, inspect logs for any additional clues.
s
@Miguel Moncada What's the name of the profile you've specified in 
dbt_project.yml
? It looks like it might be 
default
when it should match the profile you're creating, 
pomera_data_provisioning_dbt_profile
m
Thanks @Sean Williams, yes that's right - I got this working in the end thank you!
Open in Slack
PreviousNext
Powered by