Jeff Bradley06/06/2022, 5:56 PM
. I understand that for local dev,
from the user’s .dbt folder can be used ( and hopefully not overwritten by setting
) , but for deployment in K8 it is not clear what the correct thing to do is. Seems like it would be to include a
file along with the project, but I could be wrong about that. I’ve been reading the docs and tutorials but either haven’t found this information yet or just missed it. If anyone can point me in the right direction I would appreciate it.
Kevin Kho06/06/2022, 6:45 PM
Jeff Bradley06/06/2022, 6:50 PM
like below, the task creates the
file in the directory, but then throws an exception about not finding the profile, which seems weird since it just created the file.
Kevin Kho06/06/2022, 7:31 PM
Nate06/06/2022, 7:41 PM
(defining your desired profile), the DBTShellTask will build your profile on the fly (i.e. during runtime) on the pod that is running your job, so it can be used to perform your dbt transformation using the correct target the creation of the profile on the pod running your flow on k8s should have no direct relation to or impact on your local dbt profile. let me know if you have any questions here!
Jeff Bradley06/06/2022, 8:34 PM
option, it writes the file to that location and overwrites an existing one, HOWEVER, it throws that error right after, apparently when reading it back. What might I be doing wrong?
Nate06/06/2022, 9:07 PM
, then yes it will overwrite your profiles. Its hard to know without seeing your actual file structure and profiles, but if I had to guess the cause of that error I'd suggest that the value of
you're supplying might not match the profile name defined in
- I'm currently unable to reproduce this error myself, unless I purposefully pass in the wrong value for _`profile_name`_
Jeff Bradley06/06/2022, 9:44 PM
Nate06/06/2022, 9:49 PM
an absolute path?
Jeff Bradley06/06/2022, 9:50 PM
Nate06/06/2022, 9:50 PM
Jeff Bradley06/06/2022, 9:51 PM
def get_dbt_task() -> DbtShellTask:
Nate06/06/2022, 9:56 PM
Jeff Bradley06/06/2022, 9:56 PM