Channels
#prefect-community
Title
j
Jean-Michel Provencher
01/10/2023, 6:38 PMHello, out of curiosity, was there any changes related to shutdown signals for Flow runs in the last few releases? Unexpected exceptions that crashes flows are not longer sending exceptions to my Sentry integrations and I'm trying to understand why. When enabling Sentry debug logs, I see that the exception does not have time to be sent when the pod shuts down, and it was working 4 days ago.
z
Zanie
01/10/2023, 6:46 PM
I’d recommend taking a look at the release notes, but I don’t think so
There’s https://github.com/PrefectHQ/prefect/pull/7887 but it’s not released yet
j
Jean-Michel Provencher
01/10/2023, 6:48 PMyeah, what's weird is that it worked a few release ago, but yeah, it seems similar to that, like if the prefect engine would not respect the same behavior as what I see when running the python code locally.
Is this suppose to be included in the next release ?
z
Zanie
01/10/2023, 6:55 PM
No it’s on a feature branch
It may be in the next release, not sure yet
Is this with adhoc flow runs or deployed ones?
j
Jean-Michel Provencher
01/10/2023, 7:04 PMdeployed ones
I have created a ticket on my side to investigate more, but really what I see from my kubernetes run is that when an unexpected exception occurs, the process exits so quickly that Sentry does not receive the normal python shutdown signals.
And to be honest I'm not even sure this feature branch fixes that because I don't think an unexpected exception is the same as a sigterm when a cancelletion is called.
z
Zanie
01/10/2023, 7:22 PM
Are the runs being reported as CRASHED by Prefect?
j
Jean-Michel Provencher
01/10/2023, 7:22 PMThey're reported as failed
z
Zanie
01/10/2023, 7:22 PM
Oh.. we capture all exceptions in your code.
j
Jean-Michel Provencher
01/10/2023, 7:23 PMmessage has been deleted
z
Zanie
01/10/2023, 7:23 PM
What is Sentry hooking into? The interpreter is going to exit cleanly.
j
Jean-Michel Provencher
01/10/2023, 7:23 PMit's weird because I get a really different behaviour locally than in kubernetes
with the engine (not locally) locally I run as standard python commands
z
Zanie
01/10/2023, 7:23 PM
Well when you run a flow locally we reraise the exception when you retrieve the result
But when you’re running it via a deployment we don’t reraise the exception because the result isn’t retrieved
We exit the process cleanly to signal to the agent that the flow was orchestrated successfully, otherwise it’s a “crash”.
j
Jean-Michel Provencher
01/10/2023, 7:26 PMweird, because in Prefect 1 I was able to do so properly with the same code, and pretty sure last week too.
but Maybe last week was on a crashed run, I'll validate
I was able to capture an exception even on a failed state
z
Zanie
01/10/2023, 7:28 PM
Well Prefect 1 handles things entirely differently
I’m not sure how Sentry hooks in here
j
Jean-Michel Provencher
01/10/2023, 7:36 PMI'll try to get more information
normally it hooks on the error logs so this line would be sent
19:31:56.509 | ERROR | Flow run 'magic-wildebeest' - Finished in state Failed("Flow run encountered an exception. ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.\nA suitable version of pyarrow or fastparquet is required for parquet support.\nTrying to import the above resulted in these errors:\n - Missing optional dependency 'pyarrow'. pyarrow is required for parquet support. Use pip or conda to install pyarrow.\n - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.\n")z
Zanie
01/10/2023, 7:44 PM
Hm it might have something to do with the order of logging configuration if they’re attaching to errors logs
j
Jean-Michel Provencher
01/10/2023, 7:44 PMWhat do you mean
lol I just successfully captured an exception. The only thing that changed was the duration of my flow (it failed a little bit later)... really weird. I think I will try to understand more on my side why this happens. sorry for the questions.
Interesting enough, an exception caught in a task is always sent to sentry properly, but not inside the main flow.
I'll continue to understand more why.