Hi all wonder of anyone can point me to an official task for Prefect Community #prefect-community

Hi all, wonder of anyone can point me to an offici...

Joshua Greenhalgh

04/19/2022, 12:31 PM

Hi all, wonder of anyone can point me to an official task for pulling a git repo? I am wanting to use such a task in the context of running DBT - can I perhaps use something from the underlying Github tasks - don't see an explicit clone task...

Anna Geller

04/19/2022, 12:46 PM

There is no separate task because it's a single line of code. Check this blog post if you need an example

🙌 1

Anna Geller

04/19/2022, 12:47 PM

Copy code

pygit2.clone_repository(url=repo_url, path=DBT_PROJECT, checkout_branch=branch)

🙌 1

Joshua Greenhalgh

04/19/2022, 12:55 PM

Ahh ok thought maybe there would be something that deals with secrets etc but this works fine for me!

👍 1

Joshua Greenhalgh

04/22/2022, 9:49 AM

@Anna Geller just coming back to this - do you have any thoughts about best way to get the dbt profile setup when running on k8s? Just bake a profile.yml into my container?

Joshua Greenhalgh

04/22/2022, 9:51 AM

Uisng oauth so it just picks up the service account of the pod?

Anna Geller

04/22/2022, 10:21 AM

I don't know how to do that with OAuth, but rehandling dbt profile information, you could set it directly on the

DbtShellTask

, and for the Secrets, you can use PrefectSecret. Maybe this post can help? e.g. this

Joshua Greenhalgh

04/22/2022, 10:32 AM

Thanks! I was literally just reading that!! Its all a little involved I feel - think I need to create a dbt user service account - mount the key to the pod via k8s secrets - then point to the key file...

Joshua Greenhalgh

04/22/2022, 10:33 AM

I kind just hoped I could rely on the pods default service account (probably not good practice though)

Anna Geller

04/22/2022, 10:40 AM

does dbt support that? I didn't even know that - what credential does it even store - Snowflake/BigQuery credentials?

Anna Geller

04/22/2022, 10:42 AM

you need to also consider that

DbtShellTask

command runs as a subprocess, so even if you mount it to the underlying pod, I'm not sure whether this would be available to this dbt subprocess doing this via

DbtShellTask

args and Prefect Secrets is easier and I can confirm that it works

Joshua Greenhalgh

04/22/2022, 10:55 AM

So I would crate a profile.yml - which points to a service account key - bake the profile.yml into the container (nothing really secret there) - mount the service account key as k8s secret and then pass profile and profiles_dir to DBTShellTask

Joshua Greenhalgh

04/22/2022, 10:56 AM

At least thats the plan!

Joshua Greenhalgh

04/22/2022, 10:59 AM

using Prefect Secrets how do I then get the service account key into a file so that dbt can read it and authenticate to BQ with it? Just have a task that writes it to disk?

Anna Geller

04/22/2022, 11:04 AM

You should have mentioned earlier you are asking about BigQuery 🙂 BQ is indeed tricky as it requires:

Copy code

keyfile: your_service_account_file.json

Anna Geller

04/22/2022, 11:07 AM

in the worst case, you could bake the file and profiles.yml into your Docker image, but this is not a security best practice This looks interesting - if you use keyfile_json instead, you could use the same approach with Prefect Secrets as described in the post without mounting files to containers

Copy code

my-bigquery-db:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account-json
      project: [GCP project id]
      dataset: [the name of your dbt dataset]
      threads: [1 or more]
      [<optional_config>](#optional-configurations): <value>

      # These fields come from the service account json keyfile
      keyfile_json:
        type: xxx
        project_id: xxx
        private_key_id: xxx
        private_key: xxx
        client_email: xxx
        client_id: xxx
        auth_uri: xxx
        token_uri: xxx
        auth_provider_x509_cert_url: xxx
        client_x509_cert_url: xxx

7 Views

Open in Slack

Previous Next