Running this version via docker:

Version:             2.6.4
API version:         0.8.2
Python version:      3.9.15
Git commit:          51e92dda
Built:               Thu, Oct 20, 2022 3:11 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         hosted

Example flow code:

from prefect import task, flow


@task
def fail_task():
    raise ValueError('Failing')


@flow(name='fail_flow')
def fail_flow():
    fail_task()

Webhook set up to go to MS Teams like so:

Triggering and when it hits the running state, I see that come across:

The flow run definitely hits the failed state, but no webhook notification:

on the Orion container, if I do

prefect config view

the following are set:

PREFECT_DEBUG_MODE='True' (from env)
PREFECT_LOGGING_LEVEL='DEBUG' (from env)
PREFECT_LOGGING_ORION_ENABLED='True' (from env)
PREFECT_LOGGING_SERVER_LEVEL='DEBUG' (from env)

but the std out logs from the container are only the HTTP logs — not sure if I’m doing something wrong here.

well, i should say, we have our own setup_logging shared task

the flow run itself has logs, just not sure how to see why the webhook isn’t firing at the orion level, particularly since i see it fire for

Running

but not

Failed

I’ll check quick..

When I add the call to our server startup code it hangs forever 😞

No clue why, must be something weird with Uvicorn. This will unfortunately not be a quick fix.

It looks like the issue is that Uvicorn takes control over all of the logging on startup

You can install it locally if you want to get the logs for that pesky failing notification, we’ll need to look into a proper fix

I was able to monkeypatch this into our current orion container via docker volume mount and i see python logs along side the uvicorn logs!

in a bizarre twist of fate, the state notifications for

Running

and

Failed

came through, so this could be a bit of MS Teams flakiness (rate limiting? something else?), however, the part that will be even more helpful going forward:

prefect-orion-1     | 01:25:21.075 | DEBUG   | urllib3.connectionpool - https://<redacted> HTTP/1.1" 200 None
prefect-orion-1     | 01:25:21.078 | INFO    | apprise - Sent MSTeams notification.

Thanks for the quick fix @Zanie — i’ll keep an eye out for the official change.