https://prefect.io logo
#prefect-community
Title
# prefect-community
k

Ken Nguyen

04/05/2022, 3:43 PM
Hi, I’m currently pulling a dbt repo with pygit2, then executing dbt using the DbtShellTask. We have a profiles.yml in the dbt repo, but I wanted to ask if there was a way to dynamically make a profiles.yml. The idea is to have a prefect flow where users can input in a schema name (normally defined under
schema
within the profiles.yml), and have the dbt task within that flow output to a schema of the user’s choice.
k

Kevin Kho

04/05/2022, 3:47 PM
From my understanding you just wanted to make a
yaml
file in general. You can with the
yaml
package. Prefect has code that does some of this here but it’s a bit hard to follow so I think it would just be easier to try playing with that Python library.
k

Ken Nguyen

04/05/2022, 4:49 PM
Thanks Kevin. I actually found this argument
Copy code
dbt_kwargs(dict, optional): keyword arguments used to populate the profiles.yml file
reading through the code for DbtShellTask I’m going to try that out first, then if that doesn’t work I’ll try out the yml package.
m

Matthias

04/05/2022, 4:52 PM
Another thing you could do is to leverage the Jinja2 capabilities of dbt combined with environment variables. In your
profiles.yml
, you could do something like:
Copy code
...
schema: "{{ env_var('DBT_SCHEMA') }}"
...
and then set the
DBT_SCHEMA
environment variable from within a task using the input parameter.
upvote 1
k

Ken Nguyen

04/05/2022, 5:05 PM
That’s a great suggestion @Matthias!!
m

Matthias

04/05/2022, 5:47 PM
You can even supply these parameters directly in DBTShellTask as a dict in the
env
argument https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask. So no need to create an extra task 😄
upvote 1
k

Ken Nguyen

04/05/2022, 5:57 PM
My day just keeps getting better and better 😊
highfive 1
22 Views