Thread
#prefect-community
    Ken Nguyen

    Ken Nguyen

    5 months ago
    Hi, I’m currently pulling a dbt repo with pygit2, then executing dbt using the DbtShellTask. We have a profiles.yml in the dbt repo, but I wanted to ask if there was a way to dynamically make a profiles.yml. The idea is to have a prefect flow where users can input in a schema name (normally defined under
    schema
    within the profiles.yml), and have the dbt task within that flow output to a schema of the user’s choice.
    Kevin Kho

    Kevin Kho

    5 months ago
    From my understanding you just wanted to make a
    yaml
    file in general. You can with the
    yaml
    package. Prefect has code that does some of this here but it’s a bit hard to follow so I think it would just be easier to try playing with that Python library.
    Ken Nguyen

    Ken Nguyen

    5 months ago
    Thanks Kevin. I actually found this argument
    dbt_kwargs(dict, optional): keyword arguments used to populate the profiles.yml file
    reading through the code for DbtShellTask I’m going to try that out first, then if that doesn’t work I’ll try out the yml package.
    Matthias

    Matthias

    5 months ago
    Another thing you could do is to leverage the Jinja2 capabilities of dbt combined with environment variables. In your
    profiles.yml
    , you could do something like:
    ...
    schema: "{{ env_var('DBT_SCHEMA') }}"
    ...
    and then set the
    DBT_SCHEMA
    environment variable from within a task using the input parameter.
    Ken Nguyen

    Ken Nguyen

    5 months ago
    That’s a great suggestion @Matthias!!
    Matthias

    Matthias

    5 months ago
    You can even supply these parameters directly in DBTShellTask as a dict in the
    env
    argument https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask. So no need to create an extra task 😄
    Ken Nguyen

    Ken Nguyen

    5 months ago
    My day just keeps getting better and better 😊