Prefect 2 0 open source Hi I m currently trying to set up a Prefect Community #ask-community

I’ve figured out a way to fail the flow by scanning the result from PrefectFuture but somehow this doesn’t feel right (simplified code snippet in 🧵). Two questions 🙏: 1. Is there a “better” way to handle this? 2. I’d also like to add some kind of notification when the flow fails (email and/or slack) - I can’t find any integration within prefect itself, so would the recommended route be to send a webhook directly in the flow?

Dominik Wagner

06/28/2022, 10:10 AM

Copy code

def dbt_flow(env: str = "dev"):
    logger = get_run_logger()

    freshness = shell_run_command(
        command=f"dbt source freshness -t {env}",
        return_all=True
    )

    result = freshness.result()
    for res in result:
        assert "ERROR STALE" not in res

    <http://logger.info|logger.info>(f"Source is fresh, continuing...")

    build = shell_run_command(
        command=f"dbt build -t {env}"
    )

Bianca Hoch

06/28/2022, 4:00 PM

Hello Dominik! In regards to your first question, let me do a little research on that and I'll get back to you. In regards to the second one, yes, I believe the best way to send an alert through email/slack would be to pass a webhook into the flow. Here is an example on how to do that.

Bianca Hoch

06/28/2022, 5:02 PM

If I understand what you are doing correctly, I'm not sure that failing the entire flow is really necessary. You can add an

if

statement to your flow to determine whether or not to run

dbt build

based on the result of the `check dbt source freshness.`Here is some documentation where you can find an example of using results from tasks within the context of a flow.

Dominik Wagner

06/28/2022, 7:34 PM

Thanks for the reply, Bianca! 🙌 Okay great, a webhook looks easy enough to implement and just plug into flows. That's a fair point about whether or not the flow should fail; though I think my question is more about whether parsing the result to look for the errors like that is a good idea at all! 🤔

Bianca Hoch

06/29/2022, 2:03 PM

Ah! Thank you for clarifying. I'm not terribly familiar with dbt, so I'll answer this the best of my capabilities. To my understanding, if you'd like to run

dbt build

in response to the "ERROR STALE" error message specifically, then parsing the result to look for that error is a good idea.

🙏 1

2 Views

Open in Slack

Previous Next