Jeff Bradley
06/06/2022, 5:56 PMprofiles.yml
. I understand that for local dev, profiles.yml
from the user’s .dbt folder can be used ( and hopefully not overwritten by setting overwrite_profiles=True
on DbtShellTask
) , but for deployment in K8 it is not clear what the correct thing to do is. Seems like it would be to include a profiles.yml
file along with the project, but I could be wrong about that. I’ve been reading the docs and tutorials but either haven’t found this information yet or just missed it. If anyone can point me in the right direction I would appreciate it.Kevin Kho
06/06/2022, 6:45 PMJeff Bradley
06/06/2022, 6:50 PMprofile_dir
like below, the task creates the profiles.yml
file in the directory, but then throws an exception about not finding the profile, which seems weird since it just created the file._return_ DbtShellTask(
_return_all_=True,
_profile_name_=PROJECT_NAME,
_environment_=ENVIRONMENT.lower(),
_profiles_dir_="dbt/jaffle_shop/profiles",
_overwrite_profiles_=True,
_log_stdout_=True,
_log_stderr_=True,
_helper_script_=f"cd dbt/{PROJECT_NAME}",
_dbt_kwargs_= get_dbt_args(),
)
Kevin Kho
06/06/2022, 7:31 PMNate
06/06/2022, 7:41 PMdbt_kwargs
(defining your desired profile), the DBTShellTask will build your profile on the fly (i.e. during runtime) on the pod that is running your job, so it can be used to perform your dbt transformation using the correct target
the creation of the profile on the pod running your flow on k8s should have no direct relation to or impact on your local dbt profile.
let me know if you have any questions here!Jeff Bradley
06/06/2022, 8:34 PMprofiles_dir
option, it writes the file to that location and overwrites an existing one, HOWEVER, it throws that error right after, apparently when reading it back. What might I be doing wrong?Nate
06/06/2022, 9:07 PM_overwrite_profiles_=True
, then yes it will overwrite your profiles.
Its hard to know without seeing your actual file structure and profiles, but if I had to guess the cause of that error I'd suggest that the value of PROJECT_NAME
you're supplying might not match the profile name defined in dbt_project.yml
- I'm currently unable to reproduce this error myself, unless I purposefully pass in the wrong value for _`profile_name`_Jeff Bradley
06/06/2022, 9:44 PMNate
06/06/2022, 9:49 PMdbt/jaffle_shop/profiles
an absolute path?Jeff Bradley
06/06/2022, 9:50 PMNate
06/06/2022, 9:50 PMJeff Bradley
06/06/2022, 9:51 PMdef get_dbt_task() -> DbtShellTask:
_return_ DbtShellTask(
_return_all_=True,
_profile_name_=PROJECT_NAME,
_environment_=ENVIRONMENT.lower(),
_profiles_dir_=f"dbt/{PROJECT_NAME}/profiles",
_overwrite_profiles_=True,
_log_stdout_=True,
_log_stderr_=True,
_helper_script_=f"cd dbt/{PROJECT_NAME}",
_dbt_kwargs_= get_dbt_args(),
)
Nate
06/06/2022, 9:56 PMJeff Bradley
06/06/2022, 9:56 PMNate
06/06/2022, 9:58 PM