https://prefect.io
Join Slack
Hey! Does anyone some good example to share on how Prefect 2.0 dbt local cli run with Postgres? Havi...
# prefect-dbt
a
Hey! Does anyone some good example to share on how Prefect 2.0 dbt local cli run with Postgres? Having abit of difficulties setting up all the blocks and profiles correctly.
a
You could specify it in YAML:
Copy code
your_project:
  outputs:
    dev:
      client_session_keep_alive: false
      dbname: postgres
      host: localhost
      password: your_password
      port: 5432
      schema: your_project
      threads: 4
      type: postgres
      user: postgres
  target: dev
and point to it explicitly or store it in /Users/your_user_name/.dbt/profiles.yml
a
Ideally would do something like this But I'd have to register some special type block then on Prefect Cloud?
Also tried like this, but with Postgre configurations. (could work it out by storing connectionstrinng as normal Secret Block on Prefect Cloud and then just load in the dict()
Maybe it's something to do with my local setup. Getting weird error
Shortly about my goals here 🙂 With Prefect 1.0 I have set up quite simple worker on Heroku 1. deploy worker 2. registers flow on Prefect Cloud 3. starts up localAgent on the worker. 4. trigger scheduled runs from Prefect Cloud UI 5. reads dbt code from github repo 6. reads credentials from prefect cloud 1.0 kv store/secrets 7. inputs credentials into dbt cli run task and does dbt run Trying to achieve something similar now with Prefect 2.0.
a
thanks for the writeup, doing this on a restricted windows machine can be hard, perhaps you can try sth similar to this and run it first on GitHub Codespaces (it's now free for everyone! essentially a free Linux VM for development)
a
oh finally got it working 😅 Github Codespaces was a good tip. Thx! Tried first manually creating profiles.yml in the directory. Now will try to move profiles.yml to Blocks.
a
Nice work!
Feel free to share your solution, maybe I could add it to the demo repo
a
I’ll be back on this topic later during this week hopefully. Little tight on schedule right now 🙂
Didnt get into a such solution that it would be nice to share. Might need abit more polishing 😀
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by