Hello everyone,
I am not sure if this is correct p...
# ask-communityh
Hamza Kazmi
12/20/2022, 1:22 PMHello everyone,
I am not sure if this is correct place to ask this question, but today suddenly all of our prefect flows started failing and raise following error
(401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Audit-Id': '', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 20 Dec 2022 13:00:03 GMT', 'Content-Length': '129'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}👀 1
✅ 1
Hamza Kazmi
12/20/2022, 3:09 PM@Bob LeBoeuf
b
Bob LeBoeuf
12/20/2022, 3:49 PM@Hamza Kazmi I just sent a ticket over to support@prefect.io. they should be getting back to us shortly
r
Rob Freedy
12/20/2022, 4:00 PMHey @Hamza Kazmi and @Bob LeBoeuf!! We will have our support team respond in the support email thread to learn more about the issue.
gratitude thank you 1
b
Bob LeBoeuf
12/20/2022, 5:57 PM@Rob Freedy https://meet.google.com/puk-cjgr-pxf
If someone is available to join on your side, myself and two of our SRE engineers are on this call now investigating.
b
Bianca Hoch
12/20/2022, 7:22 PMHey team, marking this thread with a green check following our conversation from earlier
Bianca Hoch
12/20/2022, 7:26 PMNotes from the meeting: It was noted that an upgrade to a newer version of Kubernetes (1.21) requires new brearer tokens (bound service account tokens) that were not utilized in the previous Kubernetes version that the flows were originally running with. The recommended troubleshooting steps were to:
1. Upgrade the agent to the latest 1.4.* version.
2. Delete the kubernetes pod and create a new one which produces an updated bearer token and cert.
Bianca Hoch
12/20/2022, 7:27 PMPlease feel free to add to/correct anything mentioned in this summary ^ 😄
4 Views