Robert
11/08/2023, 3:11 AMfrom prefect import flow
from prefect_dbt.cloud import DbtCloudCredentials, DbtCloudJob
@flow
def staging_job(JOB_ID = 123456):
dbt_cloud_credentials = DbtCloudCredentials.load("dbt-cloud-credentials")
dbt_cloud_job = DbtCloudJob(dbt_cloud_credentials=dbt_cloud_credentials, job_id=JOB_ID)
dbt_cloud_job_run = dbt_cloud_job.trigger()
dbt_cloud_job_run.wait_for_completion()
dbt_cloud_job_run.fetch_result()
return dbt_cloud_job_run
staging_job()
Nate
11/08/2023, 7:14 AMprefect-dbt
?Robert
11/08/2023, 5:07 PM# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: flows
prefect-version: 2.14.2
# build section allows you to manage and build docker images
build:
- prefect.deployments.steps.run_shell_script:
id: get-commit-hash
script: git rev-parse --short HEAD
stream_output: false
- prefect_docker.deployments.steps.build_docker_image:
id: build-image
requires: prefect-docker>=0.4.0
image_name: '{{ $PREFECT_IMAGE_NAME }}'
tag: '{{ get-commit-hash.stdout }}'
dockerfile: auto
platform: linux/arm64
# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.4.0
image_name: '{{ build-image.image_name }}'
tag: '{{ build-image.tag }}'
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/swdotcom/prefect.git>
branch: main
credentials: '{{ prefect.blocks.github-credentials.github-access-token }}' # Requires creation of a Secret block
# the definitions section allows you to define reusable components for your deployments
definitions:
tags: &common_tags
- eks
- '{{ get-commit-hash.stdout }}'
work_pool: &common_work_pool
name: kubernetes
job_variables:
image: '{{ build-image.image }}'
# the deployments section allows you to provide configuration for deploying flows
deployments:
# - name: default
# tags: *common_tags
# schedule:
# entrypoint: flows/prefect_dbt_cloud.py:dbt_cloud_job_flow
# work_pool: *common_work_pool
- name: staging-job
version:
tags:
- staging
- dbt_cloud
description:
entrypoint: flows/prefect_dbt_cloud.py:staging_job
parameters: {}
work_pool:
name: kubernetes
work_queue_name:
job_variables: {}
schedule: null
Robert
11/08/2023, 11:39 PMimage: "{{ build-image.image }}"
for the job_variables
Nate
11/08/2023, 11:39 PM