https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jeff Bradley

06/06/2022, 5:56 PM
Hi, prefect noob here confused about dbt
profiles.yml
. I understand that for local dev,
profiles.yml
from the user’s .dbt folder can be used ( and hopefully not overwritten by setting
overwrite_profiles=True
on
DbtShellTask
) , but for deployment in K8 it is not clear what the correct thing to do is. Seems like it would be to include a
profiles.yml
file along with the project, but I could be wrong about that. I’ve been reading the docs and tutorials but either haven’t found this information yet or just missed it. If anyone can point me in the right direction I would appreciate it.
k

Kevin Kho

06/06/2022, 6:45 PM
I think you need to include it in the image
did you see this? you can just clone the repo
j

Jeff Bradley

06/06/2022, 6:50 PM
Yeah I read that earlier, doesn't answer my question though. And i don't think including it in the image is a good option.
Also, I don't think overwriting profiles is a good idea in general. We keep lots of profiles in that file, and overwriting them like in that blog would be silly.
When I specify a custom
profile_dir
like below, the task creates the
profiles.yml
file in the directory, but then throws an exception about not finding the profile, which seems weird since it just created the file.
_return_ DbtShellTask(
_return_all_=True,
_profile_name_=PROJECT_NAME,
_environment_=ENVIRONMENT.lower(),
_profiles_dir_="dbt/jaffle_shop/profiles",
_overwrite_profiles_=True,
_log_stdout_=True,
_log_stderr_=True,
_helper_script_=f"cd dbt/{PROJECT_NAME}",
_dbt_kwargs_= get_dbt_args(),
)
[2022-06-06 150339-0400] ERROR - prefect.DbtShellTask | 190339 Encountered an error while reading the project: 190339 ERROR: Runtime Error Could not find profile named 'jaffle_shop'
k

Kevin Kho

06/06/2022, 7:31 PM
I’ll find someone around who knows dbt more than me here 🙂
n

Nate

06/06/2022, 7:41 PM
Hey @Jeff Bradley , the recommendation in the article is not quite to overwrite your local profiles. The idea is that the flow runs are spinning up on something like a kubernetes pod, where there is no dbt profile by default. So by passing the
dbt_kwargs
(defining your desired profile), the DBTShellTask will build your profile on the fly (i.e. during runtime) on the pod that is running your job, so it can be used to perform your dbt transformation using the correct target the creation of the profile on the pod running your flow on k8s should have no direct relation to or impact on your local dbt profile. let me know if you have any questions here!
j

Jeff Bradley

06/06/2022, 8:34 PM
Yea, I get all that. However, when running the code in the article locally it will overwrite your default profiles.yml, will it not?
and by locally, I mean running with python
another question, related to above error
So after setting the
profiles_dir
option, it writes the file to that location and overwrites an existing one, HOWEVER, it throws that error right after, apparently when reading it back. What might I be doing wrong?
n

Nate

06/06/2022, 9:07 PM
@Jeff Bradley if you use
_overwrite_profiles_=True
, then yes it will overwrite your profiles. Its hard to know without seeing your actual file structure and profiles, but if I had to guess the cause of that error I'd suggest that the value of
PROJECT_NAME
you're supplying might not match the profile name defined in
dbt_project.yml
- I'm currently unable to reproduce this error myself, unless I purposefully pass in the wrong value for _`profile_name`_
j

Jeff Bradley

06/06/2022, 9:44 PM
IDK man, its all the same string and works when I use the default user .dbt folder, but I may be mixing something up somewhere
n

Nate

06/06/2022, 9:49 PM
hmm if its generating the profile and it has the correct name, I guess the only other thing is that it could be looking in the wrong place? is
dbt/jaffle_shop/profiles
an absolute path?
j

Jeff Bradley

06/06/2022, 9:50 PM
from my application root directory, yes, from the os, no
n

Nate

06/06/2022, 9:50 PM
would you be able to try providing an absolute path? for the sake of troubleshooting if nothing else
j

Jeff Bradley

06/06/2022, 9:51 PM
def get_dbt_task() -> DbtShellTask:
_return_ DbtShellTask(
_return_all_=True,
_profile_name_=PROJECT_NAME,
_environment_=ENVIRONMENT.lower(),
_profiles_dir_=f"dbt/{PROJECT_NAME}/profiles",
_overwrite_profiles_=True,
_log_stdout_=True,
_log_stderr_=True,
_helper_script_=f"cd dbt/{PROJECT_NAME}",
_dbt_kwargs_= get_dbt_args(),
)
sure
that worked
so has to be an absolute path for the profiles_dir
n

Nate

06/06/2022, 9:56 PM
cool, anything else you need help with here?
j

Jeff Bradley

06/06/2022, 9:56 PM
Where can i open a ticket for this?
Since DbtShellTask is writing and reading to different directories
n

Nate

06/06/2022, 9:58 PM
you can here - thank you!
3 Views