[DBT] [Prefect] Thanks for the relentless support everyone. I am amazed by Prefect and really want t...

Vincenzo

03/21/2023, 9:05 AM

[DBT] [Prefect] Thanks for the relentless support everyone. I am amazed by Prefect and really want to dive deeper into it! I am trying to orchestrate my dbt job with prefect. Following these instructions I was able to set up the credentials block. Unfortunately the job is not executed and it throws no error 🧵 - somebody with a similar setup that encountered this issue as well?

Vincenzo

03/21/2023, 9:06 AM

Here my code

Copy code

from prefect import flow, task
from <http://prefect_dbt.cloud.jobs|prefect_dbt.cloud.jobs> import run_dbt_cloud_job
from prefect_dbt.cloud import DbtCloudCredentials, DbtCloudJob

your_dbt_api_key = "API"  # (1) insert your dbt Cloud API key
your_dbt_account_id = "ACCOUNT"  # (2) insert your dbt Cloud account ID
your_dbt_job_id = "JOB"  # (3) insert your dbt Cloud job ID
dbt_cloud_credential_name = "magic-the-gathering-credentials"

DbtCloudCredentials(api_key=your_dbt_api_key, account_id=your_dbt_account_id).save(
    dbt_cloud_credential_name, overwrite=True
)
dbt_cloud_credentials = DbtCloudCredentials.load(dbt_cloud_credential_name)


@task(log_prints=True, name="Run dbt Cloud Job")
def run_dbt_job_flow():
    """Runs the dbt Cloud job and returns the result"""
    result = run_dbt_cloud_job(
        dbt_cloud_job=DbtCloudJob.load(dbt_cloud_credentials, your_dbt_job_id),
        targeted_retries=5,
    )
    return result


@flow(log_prints=True, name="[Magic: The Gathering] Load latest data to BigQuery")
def dbt_job_orchestration():
    """Orchestrates the flow of the dbt Cloud job"""
    run_dbt_job_flow()

Chris Reuter

03/21/2023, 11:34 AM

You may want to cross post to #C0470L5UV2M - smaller channel but specific to dbt

🙌 1

Vincenzo

03/21/2023, 3:28 PM

Thanks, I read that in the free tier, the DBT API is not accessible.. That changes a lot.

Adam

03/21/2023, 3:36 PM

have you tried the

trigger_dbt_cloud_job_run_and_wait_for_completion()

Adam

03/21/2023, 3:36 PM

here is an example that worked for me on the free tier

Copy code

from <http://prefect_dbt.cloud.jobs|prefect_dbt.cloud.jobs> import trigger_dbt_cloud_job_run_and_wait_for_completion        

trigger_dbt_cloud_job_run_and_wait_for_completion(
        dbt_cloud_credentials = DbtCloudCredentials.load("test-creds"),
        job_id = x,
        poll_frequency_seconds = 60
        )

🙌 1

Adam

03/21/2023, 3:37 PM

need to call it at the end but that function should work

Vincenzo

03/21/2023, 4:24 PM

Thanks @Adam - for me this code results in the same: No reaction, No error logs.

Adam

03/21/2023, 4:26 PM

do you have the cred block for dbt in the UI?

Vincenzo

03/21/2023, 4:28 PM

Yes, I create it via code, on top.

Vincenzo

03/22/2023, 7:04 AM

Final answer: The API in DBT is only usable if the account is paid for.

Vincenzo

03/22/2023, 7:05 AM

hence the tutorial is not for free tier users of dbt.

Adam

03/22/2023, 3:20 PM

😭

23 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.