https://prefect.io logo
Title
t

Tim Ricablanca

11/09/2022, 11:55 PM
Hi! What’s the best way to see error logs or otherwise debug an MS Teams notification? Running Prefect 2.6.4 and I see notification webhooks coming across for flow
Running
states but not for
Failed
— details in thread:
1
Running this version via docker:
Version:             2.6.4
API version:         0.8.2
Python version:      3.9.15
Git commit:          51e92dda
Built:               Thu, Oct 20, 2022 3:11 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         hosted
Example flow code:
from prefect import task, flow


@task
def fail_task():
    raise ValueError('Failing')


@flow(name='fail_flow')
def fail_flow():
    fail_task()
Webhook set up to go to MS Teams like so:
Triggering and when it hits the running state, I see that come across:
The flow run definitely hits the failed state, but no webhook notification:
on the Orion container, if I do
prefect config view
the following are set:
PREFECT_DEBUG_MODE='True' (from env)
PREFECT_LOGGING_LEVEL='DEBUG' (from env)
PREFECT_LOGGING_ORION_ENABLED='True' (from env)
PREFECT_LOGGING_SERVER_LEVEL='DEBUG' (from env)
but the std out logs from the container are only the HTTP logs — not sure if I’m doing something wrong here.
z

Zanie

11/10/2022, 12:13 AM
Hm. We may be missing a call to
setup_logging
in the application start-up.
t

Tim Ricablanca

11/10/2022, 12:13 AM
oh i have that call in the code, i removed that when i copied and pasted it.
well, i should say, we have our own setup_logging shared task
the flow run itself has logs, just not sure how to see why the webhook isn’t firing at the orion level, particularly since i see it fire for
Running
but not
Failed
z

Zanie

11/10/2022, 12:17 AM
Yeah we may be missing this call on server-startup
I’ll check quick..
👍 1
:thank-you: 1
When I add the call to our server startup code it hangs forever 😞
No clue why, must be something weird with Uvicorn. This will unfortunately not be a quick fix.
t

Tim Ricablanca

11/10/2022, 12:22 AM
oh no! would this be the same in all 2.6 versions at this point?
z

Zanie

11/10/2022, 12:26 AM
I think it’d be the same in any version
It looks like the issue is that Uvicorn takes control over all of the logging on startup
t

Tim Ricablanca

11/10/2022, 12:29 AM
ah, that makes sense with what I see
z

Zanie

11/10/2022, 12:36 AM
You can install it locally if you want to get the logs for that pesky failing notification, we’ll need to look into a proper fix
t

Tim Ricablanca

11/10/2022, 12:38 AM
Oh this is awesome. Thank you. I'll try this out.
I was able to monkeypatch this into our current orion container via docker volume mount and i see python logs along side the uvicorn logs!
in a bizarre twist of fate, the state notifications for
Running
and
Failed
came through, so this could be a bit of MS Teams flakiness (rate limiting? something else?), however, the part that will be even more helpful going forward:
prefect-orion-1     | 01:25:21.075 | DEBUG   | urllib3.connectionpool - https://<redacted> HTTP/1.1" 200 None
prefect-orion-1     | 01:25:21.078 | INFO    | apprise - Sent MSTeams notification.
Thanks for the quick fix @Zanie — i’ll keep an eye out for the official change.