Hi community 🙂, I’m having a strange bug with my flows on k8s: When a task fails, the flow goes into a failed state but I don’t see the error in the logs only some pod error message and a heartbeat related failure:

Pod prefect-job-9f127532-nnd2f failed.
	Container 'flow' state: terminated
		Exit Code:: 1
		Reason: Error

No heartbeat detected from the remote task; marking the run as failed.

But when I look at the pod logs, I can see the real error. In addition to that, we have no slack notification when the flow fails like that. Any idea how to overcome this ?

Hey @Thomas Opsomer! Are you on Prefect Cloud or Server? What was the real error you saw?

I’m on prefect cloud. This error is some custom error our application raises when something is wrong

I see. Have you seen Automations? You can configure an Automation to send a Slack message when the flow enters a Failed state

Yes I saw that, but I thought the

slack_notifier

state handler should already send failed notification

Can I see how you used it?

basically like this:

from prefect.utilities.notifications import slack_notifier

with Flow(
    name=f"flow-{username}",
    state_handlers=[set_flow_run_name, slack_notifier],
    storage=GCS(bucket),
    schedule=schedule,    run_config=get_k8s_flow_run_config(),
) as flow:
    #
    # ...

This seems like it should work. Will ask the team

alright, thanks very much Kevin 🙂 actually I’m wondering if the the notification issue and the strange error are related... because we also have missing slack notification on “paused” state... ^^

Just fyi, there's a bug with the slack notifier on

0.15.0

that is resolved in

0.15.1

Just found that the slack notification was not really missing, because the flow was not in a failed state, because there was a task with

trigger=manual_only

Oh glad you figured it out!