https://prefect.io logo
Title
h

Hamza Kazmi

12/20/2022, 1:22 PM
Hello everyone, I am not sure if this is correct place to ask this question, but today suddenly all of our prefect flows started failing and raise following error
(401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Audit-Id': '', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 20 Dec 2022 13:00:03 GMT', 'Content-Length': '129'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I have checked all keys and non of them is expired What could be other possible reason for this error ?
👀 1
1
@Bob LeBoeuf
b

Bob LeBoeuf

12/20/2022, 3:49 PM
@Hamza Kazmi I just sent a ticket over to support@prefect.io. they should be getting back to us shortly
r

Rob Freedy

12/20/2022, 4:00 PM
Hey @Hamza Kazmi and @Bob LeBoeuf!! We will have our support team respond in the support email thread to learn more about the issue.
:gratitude-thank-you: 1
b

Bob LeBoeuf

12/20/2022, 5:57 PM
@Rob Freedy https://meet.google.com/puk-cjgr-pxf If someone is available to join on your side, myself and two of our SRE engineers are on this call now investigating.
b

Bianca Hoch

12/20/2022, 7:22 PM
Hey team, marking this thread with a green check following our conversation from earlier
Notes from the meeting: It was noted that an upgrade to a newer version of Kubernetes (1.21) requires new brearer tokens (bound service account tokens) that were not utilized in the previous Kubernetes version that the flows were originally running with. The recommended troubleshooting steps were to: 1. Upgrade the agent to the latest 1.4.* version. 2. Delete the kubernetes pod and create a new one which produces an updated bearer token and cert.
Please feel free to add to/correct anything mentioned in this summary ^ 😄