https://prefect.io logo
j

Jean-Michel Provencher

01/10/2023, 6:38 PM
Hello, out of curiosity, was there any changes related to shutdown signals for Flow runs in the last few releases? Unexpected exceptions that crashes flows are not longer sending exceptions to my Sentry integrations and I'm trying to understand why. When enabling Sentry debug logs, I see that the exception does not have time to be sent when the pod shuts down, and it was working 4 days ago.
z

Zanie

01/10/2023, 6:46 PM
I’d recommend taking a look at the release notes, but I don’t think so
There’s https://github.com/PrefectHQ/prefect/pull/7887 but it’s not released yet
j

Jean-Michel Provencher

01/10/2023, 6:48 PM
yeah, what's weird is that it worked a few release ago, but yeah, it seems similar to that, like if the prefect engine would not respect the same behavior as what I see when running the python code locally.
Is this suppose to be included in the next release ?
z

Zanie

01/10/2023, 6:55 PM
No it’s on a feature branch
It may be in the next release, not sure yet
Is this with adhoc flow runs or deployed ones?
j

Jean-Michel Provencher

01/10/2023, 7:04 PM
deployed ones
I have created a ticket on my side to investigate more, but really what I see from my kubernetes run is that when an unexpected exception occurs, the process exits so quickly that Sentry does not receive the normal python shutdown signals.
And to be honest I'm not even sure this feature branch fixes that because I don't think an unexpected exception is the same as a sigterm when a cancelletion is called.
z

Zanie

01/10/2023, 7:22 PM
Are the runs being reported as CRASHED by Prefect?
j

Jean-Michel Provencher

01/10/2023, 7:22 PM
They're reported as failed
z

Zanie

01/10/2023, 7:22 PM
Oh.. we capture all exceptions in your code.
j

Jean-Michel Provencher

01/10/2023, 7:23 PM
message has been deleted
z

Zanie

01/10/2023, 7:23 PM
What is Sentry hooking into? The interpreter is going to exit cleanly.
j

Jean-Michel Provencher

01/10/2023, 7:23 PM
it's weird because I get a really different behaviour locally than in kubernetes
with the engine (not locally) locally I run as standard python commands
z

Zanie

01/10/2023, 7:23 PM
Well when you run a flow locally we reraise the exception when you retrieve the result
But when you’re running it via a deployment we don’t reraise the exception because the result isn’t retrieved
We exit the process cleanly to signal to the agent that the flow was orchestrated successfully, otherwise it’s a “crash”.
j

Jean-Michel Provencher

01/10/2023, 7:26 PM
weird, because in Prefect 1 I was able to do so properly with the same code, and pretty sure last week too.
but Maybe last week was on a crashed run, I'll validate
I was able to capture an exception even on a failed state
z

Zanie

01/10/2023, 7:28 PM
Well Prefect 1 handles things entirely differently
I’m not sure how Sentry hooks in here
j

Jean-Michel Provencher

01/10/2023, 7:36 PM
I'll try to get more information
normally it hooks on the error logs so this line would be sent
19:31:56.509 | ERROR   | Flow run 'magic-wildebeest' - Finished in state Failed("Flow run encountered an exception. ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.\nA suitable version of pyarrow or fastparquet is required for parquet support.\nTrying to import the above resulted in these errors:\n - Missing optional dependency 'pyarrow'. pyarrow is required for parquet support. Use pip or conda to install pyarrow.\n - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.\n")
For example, but it's not. I'll still try a few things on my side to investigate more.
z

Zanie

01/10/2023, 7:44 PM
Hm it might have something to do with the order of logging configuration if they’re attaching to errors logs
j

Jean-Michel Provencher

01/10/2023, 7:44 PM
What do you mean
lol I just successfully captured an exception. The only thing that changed was the duration of my flow (it failed a little bit later)... really weird. I think I will try to understand more on my side why this happens. sorry for the questions.
Interesting enough, an exception caught in a task is always sent to sentry properly, but not inside the main flow. I'll continue to understand more why.