Hi, prefect noob here confused about dbt

profiles.yml

. I understand that for local dev,

profiles.yml

from the user’s .dbt folder can be used ( and hopefully not overwritten by setting

overwrite_profiles=True

on

DbtShellTask

) , but for deployment in K8 it is not clear what the correct thing to do is. Seems like it would be to include a

profiles.yml

file along with the project, but I could be wrong about that. I’ve been reading the docs and tutorials but either haven’t found this information yet or just missed it. If anyone can point me in the right direction I would appreciate it.

I think you need to include it in the image

Yeah I read that earlier, doesn't answer my question though. And i don't think including it in the image is a good option.

I’ll find someone around who knows dbt more than me here 🙂

Hey @Jeff Bradley , the recommendation in the article is not quite to overwrite your local profiles. The idea is that the flow runs are spinning up on something like a kubernetes pod, where there is no dbt profile by default. So by passing the

dbt_kwargs

(defining your desired profile), the DBTShellTask will build your profile on the fly (i.e. during runtime) on the pod that is running your job, so it can be used to perform your dbt transformation using the correct target the creation of the profile on the pod running your flow on k8s should have no direct relation to or impact on your local dbt profile. let me know if you have any questions here!

Yea, I get all that. However, when running the code in the article locally it will overwrite your default profiles.yml, will it not?

@Jeff Bradley if you use

_overwrite_profiles_=True

, then yes it will overwrite your profiles. Its hard to know without seeing your actual file structure and profiles, but if I had to guess the cause of that error I'd suggest that the value of

PROJECT_NAME

you're supplying might not match the profile name defined in

dbt_project.yml

- I'm currently unable to reproduce this error myself, unless I purposefully pass in the wrong value for _`profile_name`_

IDK man, its all the same string and works when I use the default user .dbt folder, but I may be mixing something up somewhere

hmm if its generating the profile and it has the correct name, I guess the only other thing is that it could be looking in the wrong place? is

dbt/jaffle_shop/profiles

an absolute path?

from my application root directory, yes, from the os, no

would you be able to try providing an absolute path? for the sake of troubleshooting if nothing else

def get_dbt_task() -> DbtShellTask:

_return_ DbtShellTask(

_return_all_=True,

_profile_name_=PROJECT_NAME,

_environment_=ENVIRONMENT.lower(),

_profiles_dir_=f"dbt/{PROJECT_NAME}/profiles",

_overwrite_profiles_=True,

_log_stdout_=True,

_log_stderr_=True,

_helper_script_=f"cd dbt/{PROJECT_NAME}",

_dbt_kwargs_= get_dbt_args(),

)

cool, anything else you need help with here?

Where can i open a ticket for this?

you can here - thank you!