Just wanted to give a clap shout out clap to

Question

Just wanted to give a clap shout out clap to < Mark McDonald> who made a DbtShellTask a while back in the task library and with his help Prefect now has a section in the dbt docs here Hoping to see more dbt users in the Prefect community now marvin

Robin · Answer

< Mark McDonald> we are new to `dbt` and `prefect` How can one exactly use the `DbtShellTask` to run dbt models Where would the dbt code lie

Mark McDonald · Answer

the dbt project or any helper files for that matter can be deployed along side with the prefect python flow code in the Docker image So assuming you re using Docker storage there is an option to copy over files I add my dbt project files through this <https github com PrefectHQ prefect blob 0b3ef62bc99c5901b1fb9ae77142f7e5c5c74187 src prefect environments storage docker py L57 L58>

Robin · Answer

Wow thanks for the immediate answer Is it a good practice to put all code dependencies in the docker image of prefect and then execute it like you described Or is it rather an exception for dbt

Mark McDonald · Answer

hey < Robin> yes it s good practice to have the dbt code in the docker image to execute it as part of the flow If you don t do it this way I guess you could have a task that pulls down the dbt code from another location like s3 at the start of the flow but I don t see much benefit in doing that

Robin · Answer

Alright thanks a lot slightly smiling face

Robin · Answer

Hey < Mark McDonald> two follow up questions hope it s fine to ask here 1 If I understood correctly I have to add the files to the docker not using one dict with the sourcepath and the targetpath within the docker but with all sourcefiles in the dbt folder and all respective targetpaths in the docker folder Is that correct 2 If that s correct how do you generate this dict 3 Why do we have to provide the `profiles dir` Is it to create the `profiles yml` file in the docker container Once I have fully understood and run the `DbtShellTask` I could add some further description and a small example from a beginner s perspective to the prefect documentation Code ```import prefect from prefect import Flow task from prefect environments storage import Docker from prefect tasks aws import AWSSecretsManager from prefect tasks dbt import DbtShellTask PROFILES = ~ dbt secret name = sf credentials with Flow dbt flow as flow s = AWSSecretsManager secret name task = DbtShellTask environment= Development dbt kwargs= type snowflake threads 1 account s sf account user s sf user password s sf password profiles dir=PROFILES command= dbt run flow register project name= eks test 01 flow storage = Docker files= Users robinbeer dev code accure etl dbt ```

Mark McDonald · Answer

Hi < Robin> I hope these answers help 1 There are ways to copy the files into docker image without using this feature within Prefect s Docker storage so don t feel beholden to using this I think the dict is a key pair for each file you want to copy over If you have a huge dbt project this would be a potentially large dict 2 You could iterate through the directory where your dbt files live and build the dict that way Maybe something like the following ```helper files = dict helper directory = Path src dbt for filename in os listdir helper directory source file path = os path join os getcwd src dbt filename dest file path = os path join src dbt filename helper files source file path = dest file path storage = Docker files=helper files env vars=envars ``` 3 you need to provide a profiles dir to let dbt know where your profiles yml file lives Perhaps profiles yml located in the same directory as where your flow files are located but also very possible it s not

Robin · Answer

OK got it I am still having trouble with the `profiles dir` parameter basically <https github com PrefectHQ prefect issues 3037|with similar questions as mentioned in this PR>

Mark McDonald · Answer

< Robin> sorry for the delay in my response I just saw this message on the thread I will try to add that default in or make that parameter required I should be able to get to it tonight I have to think it over a little bit more Is the task working for you otherwise

Robin · Answer

Thanks a lot