https://prefect.io logo
Join Slack
Powered by
Hello, out of curiosity, was there any changes rel...
# ask-community
j
Hello, out of curiosity, was there any changes related to shutdown signals for Flow runs in the last few releases? Unexpected exceptions that crashes flows are not longer sending exceptions to my Sentry integrations and I'm trying to understand why. When enabling Sentry debug logs, I see that the exception does not have time to be sent when the pod shuts down, and it was working 4 days ago.
z
I’d recommend taking a look at the release notes, but I don’t think so
There’s https://github.com/PrefectHQ/prefect/pull/7887 but it’s not released yet
j
yeah, what's weird is that it worked a few release ago, but yeah, it seems similar to that, like if the prefect engine would not respect the same behavior as what I see when running the python code locally.
Is this suppose to be included in the next release ?
z
No it’s on a feature branch
It may be in the next release, not sure yet
Is this with adhoc flow runs or deployed ones?
j
deployed ones
I have created a ticket on my side to investigate more, but really what I see from my kubernetes run is that when an unexpected exception occurs, the process exits so quickly that Sentry does not receive the normal python shutdown signals.
And to be honest I'm not even sure this feature branch fixes that because I don't think an unexpected exception is the same as a sigterm when a cancelletion is called.
z
Are the runs being reported as CRASHED by Prefect?
j
They're reported as failed
z
Oh.. we capture all exceptions in your code.
j
message has been deleted
z
What is Sentry hooking into? The interpreter is going to exit cleanly.
j
it's weird because I get a really different behaviour locally than in kubernetes
with the engine (not locally) locally I run as standard python commands
z
Well when you run a flow locally we reraise the exception when you retrieve the result
But when you’re running it via a deployment we don’t reraise the exception because the result isn’t retrieved
We exit the process cleanly to signal to the agent that the flow was orchestrated successfully, otherwise it’s a “crash”.
j
weird, because in Prefect 1 I was able to do so properly with the same code, and pretty sure last week too.
but Maybe last week was on a crashed run, I'll validate
I was able to capture an exception even on a failed state
z
Well Prefect 1 handles things entirely differently
I’m not sure how Sentry hooks in here
j
I'll try to get more information
normally it hooks on the error logs so this line would be sent 
19:31:56.509 | ERROR   | Flow run 'magic-wildebeest' - Finished in state Failed("Flow run encountered an exception. ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.\nA suitable version of pyarrow or fastparquet is required for parquet support.\nTrying to import the above resulted in these errors:\n - Missing optional dependency 'pyarrow'. pyarrow is required for parquet support. Use pip or conda to install pyarrow.\n - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.\n")
For example, but it's not. I'll still try a few things on my side to investigate more.
z
Hm it might have something to do with the order of logging configuration if they’re attaching to errors logs
j
What do you mean
lol I just successfully captured an exception. The only thing that changed was the duration of my flow (it failed a little bit later)... really weird. I think I will try to understand more on my side why this happens. sorry for the questions.
Interesting enough, an exception caught in a task is always sent to sentry properly, but not inside the main flow. I'll continue to understand more why.
Open in Slack
PreviousNext
Powered by