Laura Lorenz (she/her)
06/22/2020, 6:20 PMRobin
07/12/2020, 9:44 PMdbt
and prefect
. How can one exactly use the DbtShellTask
to run dbt models?
Where would the dbt code lie?Mark McDonald
07/12/2020, 9:53 PMRobin
07/12/2020, 10:51 PMMark McDonald
07/13/2020, 2:06 PMRobin
07/13/2020, 4:22 PMprofiles_dir
? Is it to create the profiles.yml
file in the docker container?
Once I have fully understood (and run) the DbtShellTask
I could add some further description and a small example from a beginner’s perspective to the prefect documentation.
Code:
import prefect
from prefect import Flow, task
from prefect.environments.storage import Docker
from prefect.tasks.aws import AWSSecretsManager
from prefect.tasks.dbt import DbtShellTask
PROFILES = "." # "~/.dbt/"
secret_name = "sf_credentials"
with Flow("dbt_flow") as flow:
s = AWSSecretsManager(secret_name)
task = DbtShellTask(
environment="Development",
dbt_kwargs={
"type": "snowflake",
"threads": 1,
"account": s["sf_account"],
"user": s["sf_user"],
"password": s["sf_password"],
},
profiles_dir=PROFILES,
)(command="dbt run")
# flow.register(project_name="eks_test_01")
flow.storage = Docker(files={"/Users/robinbeer/dev/code/accure-etl": "dbt"})
Mark McDonald
07/23/2020, 2:26 PMhelper_files = dict()
helper_directory = Path("src/dbt")
for filename in os.listdir(helper_directory):
source_file_path = os.path.join(os.getcwd(), "src/dbt", filename)
dest_file_path = os.path.join("src/dbt", filename)
helper_files[source_file_path] = dest_file_path
storage = Docker(
...
files=helper_files,
env_vars=envars,
)
3. you need to provide a profiles_dir to let dbt know where your profiles.yml file lives. Perhaps profiles.yml located in the same directory as where your flow files are located, but also very possible it's not.Robin
07/28/2020, 11:58 AMprofiles_dir
parameter, basically with similar questions as mentioned in this PR.Mark McDonald
07/29/2020, 4:25 PMRobin
07/29/2020, 4:30 PM