Jovan Sakovic
01/29/2022, 11:30 AMFailed
state.
But, the DbtShellTask raises the FAIL signal with the message that first occurs Command failed with exit code 2
and the notifier doesn’t get to the Error that I actually want - dbt’s error.
Is there a way to push these other error messages to the notifier?Anna Geller
@task(trigger=all_finished)
def print_dbt_output(output):
logger = prefect.context.get("logger")
for line in output:
<http://logger.info|logger.info>(line)
Jovan Sakovic
01/29/2022, 1:54 PMany_failed
and then collects their output to a message that will be sent to Slack.
I guess this works fine if I use the SlackTask there, but now it got me wondering and trying to figure out how to use the state handler on this mapped task 🤔slack_handler = slack_notifier(only_states=[Failed])
@task(trigger=any_failed, state_handlers=[slack_handler])
def trigger_slack(output):
msg = "\n".join(output)
raise FAIL(msg)
with Flow('run-dbt') as flow:
dbt_deps = dbt(command="dbt deps", task_args={"name": "dbt dependencies"})
dbt_seed = dbt(command="dbt seed", upstream_tasks=[dbt_deps], task_args={"name": "dbt seed"})
dbt_run = dbt(command="dbt run", upstream_tasks=[dbt_seed], task_args={"name": "dbt run"})
trigger_slack.map([dbt_deps, dbt_seed, dbt_run],
task_args={"name": "dbt Fail Output"})
Anna Geller
any_failed
, then you only need to call it in your flow and specify that this task depends on the DBT tasks:
@task(trigger=any_failed)
def send_slack_alert_on_failure(output):
SlackTask(message=output).run()
# in Flow block:
send_slack_alert_on_failure(dbt_run)
send_slack_alert_on_failure(dbt_test)
So there is no need for mapping here imo and I think it’s cleaner to call it separately on each dbt task to explicitly show that on failure it sends an alert (and you can even name it to nicely visualize it in the DAG).
Btw, why do you run dbt_seed, dbt_run and dbt_deps in parallel? Does it make sense to start dbt_run unless we are sure the seed and deps were successful?
Here is a full gist you can use as a template: https://gist.github.com/1ee3a57076361fee4bb633ef86f7d989Jovan Sakovic
01/29/2022, 2:55 PMupstream_tasks
set up 🤔Kevin Kho
slack_notifier
. It’s just not as customizable. Yes it looks sequential to me.Jovan Sakovic
01/29/2022, 4:39 PMslack_notifier
apart - so now I’m manually sending a POST request with the similarly made payload (for now)👌PAnna Geller