Been receiving these error messages Weird Can anyone explain

Question

Been receiving these error messages Weird Can anyone explain ```Submission failed kubernetes client exceptions ApiException 500 Reason Internal Server Error HTTP response headers HTTPHeaderDict Audit Id 6d00e224 73e9 4d44 bfce d0589101f116 Cache Control no cache private Content Type application json Warning 299 unknown field spec template spec parallelism 299 unknown field spec template spec completions X Kubernetes Pf Flowschema Uid e8e112c2 396a 440d a000 51271ccde7dd X Kubernetes Pf Prioritylevel Uid a49f8ad5 8431 4216 90ae cd35c64b0736 Date Wed 10 May 2023 18 30 59 GMT Content Length 161 HTTP response body kind Status apiVersion v1 metadata status Failure message rpc error code = Unknown desc = exec try 500 database is locked code 500 ```

Nate · Answer

hi < Deceivious> are you running against Prefect Cloud

Deceivious · Answer

Nope

Deceivious · Answer

well atleast this issue isnt in cloud

Deceivious · Answer

we use both version

Zanie · Answer

Hm that looks like a Kubernetes internal error

Deceivious · Answer

Thanks ill scope it out tomorrow and if I discover something ill let you know if its related to prefect

Deceivious · Answer

Ummm < Nate> < Zanie> Unsure if this is an issue But the status is shown as Failed and not Crashed At this point when the pod itself failed to spawn the `on failure` hook will not get triggered because the pod hasnt started up Should the status be something else other than failure

Deceivious · Answer

image png

Deceivious · Answer

Is there any way of getting notified about such issues

Nate · Answer

we were just discussing this we think this should be a `Crashed` state this situation is a good example of why opening issue now

Deceivious · Answer

Thanks Please link the issue here as well so i can follow it

Deceivious · Answer

But the no notification issue will still exist i guess because there is no infrastucture to run `on failure` and `on crash` hook At which point the only way would be to create an automation

Zanie · Answer

Yeah automations are the only way to capture that kind of failure

Nate · Answer