https://prefect.io logo
Powered by
Title
j

Josefin Winberg

03/17/2023, 2:05 PM
Hello everyone! I am facing some issues in our deployment after the owner changed in our instance.. Right now I'm seeing this error message from the agent in the compute engine in GCP: "httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://api.prefect.cloud/api/accounts/8cba8c22-3359-43ea-845e-b88737a33ddd/workspaces/caa7e1bf-faa2-4ba5-baab-1987efa226f1/work_queues/name/prod' For more information check: https://httpstatuses.com/401 An exception occurred." Getting this when checking the /health endpoint (see image) However, I've updated the PREFECT_API_URL and the PREFECT_API_KEY. Locally flows run fine. Not sure what's causing this and how to fix it.. would anyone be able to point me in the right direction?
👀 1
✅ 1
b

Bianca Hoch

03/17/2023, 2:23 PM
Hey Josefin! We got your email as a support case as well. I'll respond here to assist with some troubleshooting steps.
Question, have you tried re-creating the agent using an account with the updated API key?
j

Josefin Winberg

03/17/2023, 2:23 PM
Ah great, thanks a lot!
I haven't re-created the agent, just updated the API key
However, I suspect that maybe the issue lays in the secret value in GCP? It seems that there is a header_token and a app_token that I believe needs to be re-generated from my account instead of the old owner?
I can't seem to find where to generate this though..
b

Bianca Hoch

03/17/2023, 2:28 PM
However, I suspect that maybe the issue lays in the secret value in GCP? It seems that there is a header_token and a app_token that I believe needs to be re-generated from my account instead of the old owner?
That could very well be a possibility.
As a first step, could you try restarting the agent to see if that changes anything?
j

Josefin Winberg

03/17/2023, 2:31 PM
I have already restarted the agent a couple of times.. didn't change anything unfortunately
🤔 1
Isn't there a way to generate new header_token and app_token?
b

Bianca Hoch

03/17/2023, 2:37 PM
taking a look now to see where/how to do that
j

Josefin Winberg

03/17/2023, 2:37 PM
Thanks a lot!
b

Bianca Hoch

03/17/2023, 2:41 PM
in the meantime, could you try running this in the environment where your agent is? 
curl -v -H 'Authorization: Bearer <replace with your api key>' <https://api.prefect.cloud/api/accounts/8cba8c22-3359-43ea-845e-b88737a33ddd/workspaces/caa7e1bf-faa2-4ba5-baab-1987efa226f1/work_queues/name/prod>
just to check the API key and rule out any issues with it
j

Josefin Winberg

03/17/2023, 2:43 PM
Connected, no issue!
b

Bianca Hoch

03/17/2023, 2:58 PM
Haven't been able to pinpoint where you can generate a new header_token and app_token...I'm wondering if you'll need to recreate the GCP service account and re-launch the agent?
I'm looking at the 
gcp-prefect
collection, and the first step is to create a service account key file.
j

Josefin Winberg

03/17/2023, 3:00 PM
Ah, I've looked there as well.. But the service account is not connected to my account so shouldn't be any issue there?
b

Bianca Hoch

03/17/2023, 3:09 PM
hmmm..good point
forgot to ask- in addition to updating the API key, did you also update the API URL?
j

Josefin Winberg

03/17/2023, 3:48 PM
Ah yes, I've updated the URL as well.
b

Bianca Hoch

03/17/2023, 3:59 PM
After talking a bit with the team, it may be worth it to start from scratch and configure the PREFECT_API_URL and PREFECT_API_KEY anew. And then restarting the agent after the fact.
c

Christopher Boyd

03/17/2023, 4:00 PM
Hi Josefin, 401 means a bad api key, 403 means a good api key without permissions (like you have a valid api key trying to access my workspace) and a 404 would indicate an invalid account / workspace
Here are examples of each: 
checkapikey 12345 (bad api key)
<https://api.prefect.cloud/api/accounts//workspaces/>
curl -i -H 'Authorization: Bearer 12345' <https://api.prefect.cloud/api/accounts//workspaces//health>
HTTP/2 401
date: Fri, 17 Mar 2023 15:22:07 GMT
server: istio-envoy
x-deny: ext-authz
content-length: 47
content-type: application/json
x-envoy-upstream-service-time: 5
via: 1.1 google
content-security-policy: default-src 'none'; object-src 'none'
x-content-type-options: nosniff
strict-transport-security: max-age=63072000; includeSubDomains; preload
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

{"detail":"Invalid authentication credentials"}%
Boyd
  31 minutes ago

versus good key, good workspace:

curl -i -H 'Authorization: Bearer <valid key>' <https://api.prefect.cloud/api/accounts///health>
HTTP/2 200
date: Fri, 17 Mar 2023 15:29:32 GMT
server: istio-envoy
content-length: 4
content-type: application/json
x-envoy-upstream-service-time: 62
via: 1.1 google
content-security-policy: default-src 'none'; object-src 'none'
x-content-type-options: nosniff
strict-transport-security: max-age=63072000; includeSubDomains; preload
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

true%
(edited)
Boyd
  30 minutes ago

versus 403 (good key, bad workspace):

curl -i -H 'Authorization: Bearer <valid key>' <https://api.prefect.cloud/api/accounts//workspaces//health>
HTTP/2 403
date: Fri, 17 Mar 2023 15:30:48 GMT
server: istio-envoy
content-length: 25
content-type: application/json
x-envoy-upstream-service-time: 30
via: 1.1 google
content-security-policy: default-src 'none'; object-src 'none'
x-content-type-options: nosniff
strict-transport-security: max-age=63072000; includeSubDomains; preload
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

{"detail":"Unauthorized"}%
j

Josefin Winberg

03/17/2023, 4:03 PM
Hi Christopher! Yes, that I know 🙂
I'm getting a 443 when running this from the agent as written before
@Bianca Hoch Okay, will try that later or after the weekend.
c

Christopher Boyd

03/17/2023, 4:08 PM
a 443?
j

Josefin Winberg

03/17/2023, 4:49 PM
Yeah from agent I get "Connected to api.prefect.cloud"
b

Bianca Hoch

03/22/2023, 2:18 PM
Hey Josefin! Just checking back to see if the connectivity issue was cleared up
j

Josefin Winberg

03/23/2023, 10:28 AM
Hi Bianca! Yes, it got solved after following your instructions, thanks you so much for your help.
⭐ 1
🙌 1
🚀 1
#prefect-communityJoin Slack
Powered by