Hi, I’m currently pulling a dbt repo with pygit2, then executing dbt using the DbtShellTask. We have a profiles.yml in the dbt repo, but I wanted to ask if there was a way to dynamically make a profiles.yml. The idea is to have a prefect flow where users can input in a schema name (normally defined under

schema

within the profiles.yml), and have the dbt task within that flow output to a schema of the user’s choice.

From my understanding you just wanted to make a

yaml

file in general. You can with the

yaml

package. Prefect has code that does some of this here but it’s a bit hard to follow so I think it would just be easier to try playing with that Python library.

Thanks Kevin. I actually found this argument

dbt_kwargs(dict, optional): keyword arguments used to populate the profiles.yml file

reading through the code for DbtShellTask I’m going to try that out first, then if that doesn’t work I’ll try out the yml package.

Another thing you could do is to leverage the Jinja2 capabilities of dbt combined with environment variables. In your

profiles.yml

, you could do something like:

...
schema: "{{ env_var('DBT_SCHEMA') }}"
...

and then set the

DBT_SCHEMA

environment variable from within a task using the input parameter.

That’s a great suggestion @Matthias!!

You can even supply these parameters directly in DBTShellTask as a dict in the

env

argument https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask. So no need to create an extra task 😄

My day just keeps getting better and better 😊