https://prefect.io logo
Powered by
Title
d

Dominik Wagner

06/28/2022, 10:10 AM
Prefect 2.0 (open-source) Hi! I’m currently trying to set up a simple flow to run dbt in the following way: 1. check 
dbt source freshness
a. If I get an error, fail the flow 2. run 
dbt build
I’ve figured out a way to fail the flow by scanning the result from PrefectFuture but somehow this doesn’t feel right (simplified code snippet in 🧵). Two questions 🙏: 1. Is there a “better” way to handle this? 2. I’d also like to add some kind of notification when the flow fails (email and/or slack) - I can’t find any integration within prefect itself, so would the recommended route be to send a webhook directly in the flow?
def dbt_flow(env: str = "dev"):
    logger = get_run_logger()

    freshness = shell_run_command(
        command=f"dbt source freshness -t {env}",
        return_all=True
    )

    result = freshness.result()
    for res in result:
        assert "ERROR STALE" not in res

    <http://logger.info|logger.info>(f"Source is fresh, continuing...")

    build = shell_run_command(
        command=f"dbt build -t {env}"
    )
b

Bianca Hoch

06/28/2022, 4:00 PM
Hello Dominik! In regards to your first question, let me do a little research on that and I'll get back to you. In regards to the second one, yes, I believe the best way to send an alert through email/slack would be to pass a webhook into the flow. Here is an example on how to do that.
If I understand what you are doing correctly, I'm not sure that failing the entire flow is really necessary. You can add an 
if
statement to your flow to determine whether or not to run 
dbt build
based on the result of the `check dbt source freshness.`Here is some documentation where you can find an example of using results from tasks within the context of a flow.
d

Dominik Wagner

06/28/2022, 7:34 PM
Thanks for the reply, Bianca! 🙌 Okay great, a webhook looks easy enough to implement and just plug into flows. That's a fair point about whether or not the flow should fail; though I think my question is more about whether parsing the result to look for the errors like that is a good idea at all! 🤔
b

Bianca Hoch

06/29/2022, 2:03 PM
Ah! Thank you for clarifying. I'm not terribly familiar with dbt, so I'll answer this the best of my capabilities. To my understanding, if you'd like to run 
dbt build
in response to the "ERROR STALE" error message specifically, then parsing the result to look for that error is a good idea.
:thank-you: 1
#prefect-communityJoin Slack
Powered by