https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-dbt
  • l

    Lashwin Naidoo

    10/18/2022, 3:44 AM
    @Lashwin Naidoo has joined the channel
  • p

    Paul Burden

    10/18/2022, 3:44 AM
    @Paul Burden has joined the channel
  • k

    Kalise Richmond

    10/18/2022, 3:28 PM
    Cross posting @Anna Geller new medium article-- Part 2 guide with Prefect, dbt and Snowflake :dbt: https://medium.com/the-prefect-blog/modular-data-stack-build-a-data-platform-with-prefect-dbt-and-snowflake-part-2-cf753708a19e
    :party-parrot: 3
    :dbt: 5
    ❤️ 1
  • b

    Brian Bergeron

    10/18/2022, 3:57 PM
    Coalesce is underway! A shot from the opening keynote
  • b

    Brian Bergeron

    10/18/2022, 3:58 PM
    and a handful from our happy hour -- Flights and Bytes last night with Deepnote, Airbyte, and Census. We planned for 100 people to show up and had over 300 😅
    🎉 6
  • a

    Andrew Black

    10/18/2022, 8:17 PM
    Proud to be a part of this phenomenal ecosystem of partners — and to get to see so many of them in person for the first time!
    :marvin: 2
    🚀 3
    :dbt: 2
    :party-parrot: 1
  • k

    Kalise Richmond

    10/19/2022, 9:18 PM
    The Prefect booth at Coalesce is going strong today including some live juggling 🤹
    77C40A3A-90F9-49B5-9D73-34394FB0FB66.MOV
    👍 1
    🤹 3
    🚀 1
    :dbt: 4
    🔥 4
    :party-parrot: 2
  • c

    Chris Reuter

    10/20/2022, 1:13 PM
    Live (kind of) from dbt it's @Chris White: https://twitter.com/mehd_io/status/1583076511850332160
    :marvin: 1
  • c

    Chris White

    10/20/2022, 1:13 PM
    @Chris White has joined the channel
  • c

    Chris Reuter

    10/20/2022, 1:15 PM
    Also today whether you're in person at Coalesce or virtual, check out @Dylan’s talk on event driven dbt models at 4p Central! https://coalesce.getdbt.com/agenda/when-the-real-world-messes-with-your-schedule-event-driven-dbt-models-for-the-mds
    :upvote: 4
  • k

    Kalise Richmond

    10/20/2022, 2:09 PM
    A note from Marvin :marvin-duck:
    Video - 2022-10-20 8_51_17 AM.mp4
    🙌 2
    :marvin-duck: 8
    🚀 2
    😅 1
    :marvin-police: 4
    c
    • 2
    • 1
  • c

    Chris Reuter

    10/20/2022, 3:19 PM
    More 🔥 pics from :dbt: Coalesce yesterday, 📸 courtesy of @David Abraham
    ❤️ 1
    :marvin-duck: 1
    :dbt: 1
    :prefect: 1
    🙌 1
    :gratitude-thank-you: 1
    👍 1
    m
    • 2
    • 1
  • d

    David Abraham

    10/20/2022, 3:19 PM
    @David Abraham has joined the channel
  • a

    Adam

    10/31/2022, 1:37 PM
    Hi friends, I have separate git repos for dbt and prefect. I’d like to start using prefect to schedule dbt runs but trying to understand how best to do that. Should I just make prefect first do a
    git clone
    of the dbt repo? Currently just using dbt cloud to run on a schedule but thinking of moving away from that towards the above solution.
    k
    a
    +2
    • 5
    • 7
  • a

    Andreas Nigg

    10/31/2022, 5:03 PM
    Hey folks, I've a question for all the dbt data model masterminds out there (I hope this is the right channel). I’ve a rather huge source table which is partitioned by a column “loaded_at”. I have an incremental model which reads from this source table. To exclude source table partitions to read, I could make use of the _dbt_max_partitions scripting variable - something like below
    {% if is_incremental() %}
        where loaded_at>= coalesce(_dbt_max_partition, '2022-01-01')
      {% else %}
    But, the problem is, that in my incremental model I do not partition by “loaded_at” but by a different column (due to use-case demands). So _dbt_max_partition would not help here, as it would simply return the maximum partition value of the model (which I can’t use as filter for the source table). In “native” BigQuery I would simply use a scripting variable as follows
    declare max_source_partition timestamp;
    set max_source_partition = (select max(loaded_at) as ts from `my_model_table`);
    select * from `my_source_table` where loaded_at > max_source_partition
    How can one implement such a scenario with dbt? Is there a way to create scripting variables as part of my models? Or do I need to add it as a on-start-hook? Or any better strategies to exclude partitions in my source without having the same column as partition field in my model?
    n
    • 2
    • 1
  • m

    Marc Lipoff

    11/02/2022, 6:22 PM
    Im looking to execute some
    dbt ...
    commands from prefect. It seems there is a lot of overhead to getting this working. My steps (my head at least are): • Grab the appropriate dbt docker image • Pull my dbt repo • do a docker run, something like this:
    docker run -v /repo/loc/:/src/ -w /src/ image_name run --select ...
    I see there are both prefect-docker and prefect-dbt collections. Curious what others have done...
    g
    • 2
    • 4
  • c

    Chris Marchetti [Datateer]

    11/14/2022, 9:00 PM
    Hello Prefect folks, I have multiple data extraction tasks that are running in separate pipelines. The longes of these can take about 3 hours. I want to create a flow that waits for the last of these to finish running before my dbt tasks run in a separate pipeline. Is there a way to ensure that the extraction tasks finish before the dbt tasks begin? We are using prefect 1.0 and we are not using the dask executor at present. Mostly we need a good enough solution until we can create a long term solution to run tasks or subflows in parallel. Thanks!
    m
    • 2
    • 3
  • a

    andres aava

    11/19/2022, 4:29 PM
    Hey! Does anyone some good example to share on how Prefect 2.0 dbt local cli run with Postgres? Having abit of difficulties setting up all the blocks and profiles correctly.
    a
    • 2
    • 11
  • k

    keurcien

    11/28/2022, 11:43 AM
    Hi everyone! I am currently trying to implement Prefect with DBT. We have a prefect repo with all organization's data workflows. I'm wondering whether we should put our DBT code inside that Prefect repo or should it have it's own Github repo? What would be the pros and cons? In terms of Developer experience and CI/CD?
    o
    m
    +3
    • 6
    • 10
  • r

    Rachel Molloy

    11/29/2022, 5:03 PM
    Hi all! I'm running into an issue getting started with dbt and Prefect on the CLI. I already have dbt successfully running locally, with a Snowflake connection and profiles.yml file. Using Prefect 2.6.9, dbt 1.3.1 and Python 3.9.15 on Windows running locally. However, when I try to run a simple "dbt debug" command inside a Prefect flow, I get a "NotImplementedError" message (see thread). I've tried two different methods of running the dbt command: one referencing an existing project directory and profiles.yml file, and one manually hardcoding a profiles configuration (see thread for full code). Both give me the same "NotImplementedError". Any ideas what I'm missing?
    ✅ 1
    a
    • 2
    • 13
  • a

    Aaron Gonzalez

    01/11/2023, 3:18 PM
    Hello everyone! I have a quick (possibly newbie-ish) question about how people have been using [`BigQueryTargetConfigs]`(https://prefecthq.github.io/prefect-dbt/cli/configs/bigquery/#prefect_dbt.cli.configs.bigquery.BigQueryTargetConfigs).
  • a

    Aaron Gonzalez

    01/11/2023, 3:19 PM
    Newbie question: I've wondering how people have used the
    BigQueryTargetConfigs
    prefect-dbt block .
    a
    • 2
    • 32
  • s

    Stephen Herron

    01/20/2023, 11:49 AM
    How do I set my DbtCliProfile to use a SnowflakeTargetConfig? The UI only let’s you select TargetConfig and not a snowflake one
    👀 1
    a
    • 2
    • 4
  • r

    Raul Maldonado

    01/25/2023, 7:47 PM
    Just researching: • Does anyone have documentation or references on maxing out :dbt: dbt CORE capabilities before migrating to Prefect?
    d
    • 2
    • 6
  • f

    Farid

    02/06/2023, 10:31 PM
    Hi all! I'm trying to run a dbt library locally using dbt-cli in Prefect. I've noticed an error I tend to get is:
    ERROR: Runtime Error
      Credentials in profile "snowflake_dbt_transformations", target "dev" invalid: 'database' is a required propertyDefined profiles:
     - snowflake_dbt_transformations
    For more information on configuring profiles, please consult the dbt docs:
    
    <https://docs.getdbt.com/docs/configure-your-profile>
    Upon investigation, it seems like the dbtCliProfile passed to the dbtTrigger task gets saved in the
    ~/.dbt/profiles.yml
    first and then used inside the dbtCli shell task. However the problem is the method where it saves the dbtCliProfile into a yaml file does not save the
    TargetConfigs
    resulting in the profile file being incomplete. Has anyone had an experience with this before and know any workarounds?
    • 1
    • 3
  • d

    Donny Flynn

    02/09/2023, 5:45 PM
    Hi! I'm standing up dbt CLI on AWS Fargate using tasks from this template: https://github.com/anna-geller/dataflow-ops I have all of the flows deployed and scheduled using Prefect cloud, and I'm having no problems running the dbt run (it is using the project files from the S3 bucket/folder that's attached to the image), but I'm running into issues when running
    dbt docs generate
    as the Flow Run is outputting that
    Catalog written to /opt/prefect/target/catalog.json
    I checked S3 and it's definitely not there, and I'm guessing that the
    /opt
    is a directory that's tied to the ECS Task from the container? Is there a way that the dbt cli command can output the docs file (specifically index.html into the S3 bucket so we can host our dbt docs as a static site) Flow code is in the 🧵. I really appreciate any help or pointers 🙂
    a
    • 2
    • 4
  • o

    Oluremi Akinwale

    02/13/2023, 8:24 AM
    Hi everyone, please how can I run prefect 2.0 Agent within python code that contain the flow?
  • j

    Jaime Raldua Veuthey

    02/15/2023, 4:58 PM
    Hi, this might be a stupid question, but is it possible to schedule queries created with dbt inside Bigquery? We are looking to have views, ... created into Bigquery and would like to know if I can schedule when they run inside a flow. Atm we have them as scheduled queries that run at specific times. Thanks!
    s
    • 2
    • 2
  • a

    Andrew Huang

    02/16/2023, 10:46 PM
    Cross posting the announcement of prefect-dbt v0.3.0 https://prefect-community.slack.com/archives/CKNSX5WG3/p1676578412722889
    💯 5
    🔥 2
  • s

    Shaun Fender

    02/24/2023, 10:59 AM
    Hi all! Very much starting out trying to orchestrate our dbt commands using prefect but running into a few issues. Issues in images provided - can anyone guide me on why these flows aren't working? First image - no idea why it doesn't like that comma as I've lifted that snippet from prefect-dbt github page. Second image - I'm running a task from within a flow, but it seems to this I'm running a task from within a task?
    a
    • 2
    • 2
Powered by Linen
Title
s

Shaun Fender

02/24/2023, 10:59 AM
Hi all! Very much starting out trying to orchestrate our dbt commands using prefect but running into a few issues. Issues in images provided - can anyone guide me on why these flows aren't working? First image - no idea why it doesn't like that comma as I've lifted that snippet from prefect-dbt github page. Second image - I'm running a task from within a flow, but it seems to this I'm running a task from within a task?
a

Aaron Gonzalez

02/24/2023, 12:55 PM
The second image:
trigger_dbt_cli_command
is a task. It's complaining because you have nested that task inside other tasks like
debug_project
,
load_seeds
, etc.
s

Shaun Fender

02/24/2023, 1:14 PM
@Aaron Gonzalez Ah right I see, that makes sense, as I had this working with no issue. Thanks!
View count: 1