https://prefect.io logo
Title
d

Deceivious

05/10/2023, 7:15 PM
Been receiving these error messages. Weird! Can anyone explain?
Submission failed. kubernetes.client.exceptions.ApiException: (500) Reason: Internal Server Error HTTP response headers: HTTPHeaderDict({'Audit-Id': '6d00e224-73e9-4d44-bfce-d0589101f116', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Warning': '299 - "unknown field \\"spec.template.spec.parallelism\\"", 299 - "unknown field \\"spec.template.spec.completions\\""', 'X-Kubernetes-Pf-Flowschema-Uid': 'e8e112c2-396a-440d-a000-51271ccde7dd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a49f8ad5-8431-4216-90ae-cd35c64b0736', 'Date': 'Wed, 10 May 2023 18:30:59 GMT', 'Content-Length': '161'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = exec (try: 500): database is locked","code":500}
n

Nate

05/10/2023, 7:24 PM
hi @Deceivious - are you running against Prefect Cloud?
d

Deceivious

05/10/2023, 7:25 PM
Nope
well atleast this issue isnt in cloud
we use both version
z

Zanie

05/10/2023, 7:45 PM
Hm that looks like a Kubernetes internal error?
d

Deceivious

05/10/2023, 7:53 PM
Thanks ill scope it out tomorrow and if I discover something ill let you know if its related to prefect
Ummm @Nate, @Zanie Unsure if this is an issue. But the status is shown as "Failed" and not "Crashed". At this point when the pod itself failed to spawn, , the
on_failure
hook will not get triggered because the pod hasnt started up. Should the status be something else other than failure?
image.png
Is there any way of getting notified about such issues?
n

Nate

05/10/2023, 9:35 PM
we were just discussing this, we think this should be a
Crashed
state - this situation is a good example of why opening issue now
d

Deceivious

05/10/2023, 9:35 PM
Thanks. Please link the issue here as well so i can follow it.
But the no notification issue will still exist i guess because there is no infrastucture to run
on_failure
and
on_crash
hook. At which point, the only way would be to create an automation ?
z

Zanie

05/10/2023, 9:39 PM
Yeah automations are the only way to capture that kind of failure