Hi all wonder of anyone can point me to an official task for

Question

Hi all wonder of anyone can point me to an official task for pulling a git repo I am wanting to use such a task in the context of running DBT can I perhaps use something from the underlying Github tasks don t see an explicit clone task

Anna Geller · Answer

There is no separate task because it s a single line of code Check <https www prefect io blog flow of flows orchestrating elt with prefect and dbt |this blog post> if you need an example

Anna Geller · Answer

```pygit2 clone repository url=repo url path=DBT PROJECT checkout branch=branch ```

Joshua Greenhalgh · Answer

Ahh ok thought maybe there would be something that deals with secrets etc but this works fine for me

Joshua Greenhalgh · Answer

< Anna Geller> just coming back to this do you have any thoughts about best way to get the dbt profile setup when running on k8s Just bake a profile yml into my container

Joshua Greenhalgh · Answer

Uisng oauth so it just picks up the service account of the pod

Anna Geller · Answer

I don t know how to do that with OAuth but rehandling dbt profile information you could set it directly on the `DbtShellTask` and for the Secrets you can use PrefectSecret Maybe can help e g

Joshua Greenhalgh · Answer

Thanks I was literally just reading that Its all a little involved I feel think I need to create a dbt user service account mount the key to the pod via k8s secrets then point to the key file

Joshua Greenhalgh · Answer

I kind just hoped I could rely on the pods default service account probably not good practice though

Anna Geller · Answer

does dbt support that I didn t even know that what credential does it even store Snowflake BigQuery credentials

Anna Geller · Answer

you need to also consider that `DbtShellTask` command runs as a subprocess so even if you mount it to the underlying pod I m not sure whether this would be available to this dbt subprocess doing this via `DbtShellTask` args and Prefect Secrets is easier and I can confirm that it works

Joshua Greenhalgh · Answer

So I would crate a profile yml which points to a service account key bake the profile yml into the container nothing really secret there mount the service account key as k8s secret and then pass profile and profiles dir to DBTShellTask

Joshua Greenhalgh · Answer

At least thats the plan

Joshua Greenhalgh · Answer

using Prefect Secrets how do I then get the service account key into a file so that dbt can read it and authenticate to BQ with it Just have a task that writes it to disk

Anna Geller · Answer

You should have mentioned earlier you are asking about BigQuery slightly smiling face BQ is indeed tricky as it requires ```keyfile your service account file json```

Anna Geller · Answer

in the worst case you could bake the file and profiles yml into your Docker image but this is not a security best practice <https docs getdbt com reference warehouse profiles bigquery profile service account json|This> looks interesting if you use keyfile json instead you could use the same approach with Prefect Secrets as described in the post without mounting files to containers ```my bigquery db target dev outputs dev type bigquery method service account json project GCP project id dataset the name of your dbt dataset threads 1 or more lt optional config gt optional configurations lt value gt These fields come from the service account json keyfile keyfile json type xxx project id xxx private key id xxx private key xxx client email xxx client id xxx auth uri xxx token uri xxx auth provider x509 cert url xxx client x509 cert url xxx```