https://prefect.io logo
Title
f

fozan talat

04/13/2023, 5:22 PM
Hi EveryOne ! I'm getting this error with dbtCoreOperation , can anyone tell me what this error means?
Finished in state Failed('Flow run encountered an exception. RuntimeError: PID 17072 failed with return code 1.\n')
p

Phornkrit

04/21/2023, 7:00 AM
Did you found the solutions yet? I'm getting this Error too.
f

fozan talat

04/21/2023, 7:16 AM
Nope.
🥲 1
@Kevin Ruprecht This one.
s

Stephen Lloyd

05/04/2023, 6:43 AM
Can you share your flow code?
s

Stephen Lloyd

05/05/2023, 4:49 AM
I have my DbtCoreOperation call in my flow function. I would double-check the docs for dbtCliProfile because the way you are calling it does not match the docs. (You don't need to load this block like you do a secret because this is handled inside DbtCoreOperation)
f

fozan talat

05/05/2023, 5:01 AM
can you share your flow code , how you are calling it ?
give me an example from code
@Stephen Lloyd There ?
s

Stephen Lloyd

05/05/2023, 5:05 AM
yes.
import os
from prefect import task, flow, get_run_logger

from prefect.blocks.system import Secret
from prefect_dbt.cli import DbtCliProfile, DbtCoreOperation
from prefect_shell import ShellOperation


ENV = os.environ["ENV"].upper() if "ENV" in os.environ else "DEV"
DBT_ENV = ENV.lower()


@task(retries=10, retry_delay_seconds=60)
def checkout_dbt_repo(token: str):
    ShellOperation(
        commands=[
            "rm -rf edw-dbt && git clone --depth=1 --branch=main https://{github_access_token}@github.com/cloudfactory/edw-dbt".format(
                github_access_token=token
            )
        ]
    ).run()


@flow(name="dbt_flow")
def dbt_flow(dbt_build_arg: str = ""):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"flow parameter dbt_build_arg has value : {dbt_build_arg}")
    gh_access_token = Secret.load("github-access-token").get()

    <http://logger.info|logger.info>("Downloading dbt repo")
    checkout_dbt_repo(token=gh_access_token)

    dbt_cli_profile = DbtCliProfile.load("dbt-cli-profile")
    <http://logger.info|logger.info>(f"Running - dbt build {dbt_build_arg} --target {DBT_ENV}")
    with DbtCoreOperation(
        commands=["dbt debug", "dbt deps", "dbt build " + dbt_build_arg + " --target " + DBT_ENV],
        project_dir="edw-dbt",
        overwrite_profiles=True,
        dbt_cli_profile=dbt_cli_profile,
        stream_output=True,
    ) as dbt_operation:
        dbt_process = dbt_operation.trigger()
        dbt_process.wait_for_completion()
        result = dbt_process.fetch_result()

    return result


if __name__ == "__main__":
    print(dbt_flow("-s base_users_users"))
acha he?
😀 1
f

fozan talat

05/05/2023, 5:09 AM
I think, it does not matter if you put it inside flow or outside it as long as you calling it in your flow code
s

Stephen Lloyd

05/05/2023, 5:10 AM
tk. I think your problem is that you are loading the profile.
f

fozan talat

05/05/2023, 5:10 AM
So how it should be ?
s

Stephen Lloyd

05/05/2023, 5:14 AM
oh, I'm sorry. I'm remembering wrong. Load is fine, but you couldn't use
get_profile
in the version we are/were using. The current docs are reflecting
get_profile()
, so I'm not sure.
My code above runs locally on 2.8.5
f

fozan talat

05/05/2023, 5:16 AM
ok