https://prefect.io logo
Powered by
Title
a

Alexey Stoletny

09/28/2022, 3:26 PM
@Christopher Boyd the same issue again with flows crashing started to happen a few minutes ago: 
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/43db7ccd-9f39-41f2-8989-000b28747858/workspaces/cedd89e9-9f12-421e-a17b-94045c976a2a/block_types/e91e7544-7ecd-4fa6-b6d7-a53068cb67fc>'
Response: {'detail': 'protected block types cannot be updated.'}
For more information check: <https://httpstatuses.com/403>
✅ 1
k

Khuyen Tran

09/28/2022, 3:26 PM
Can you link to the previous thread of this issue? Also to keep the channel clean, do you mind putting the related questions in one thread?
a

Alexey Stoletny

09/28/2022, 3:27 PM
https://prefect-community.slack.com/archives/CL09KU1K7/p1664378789918679
okay sounds good
tagging @Anna Geller and @Zanie who helped with the last issue as well, just in case
this is affecting our production flow of data
z

Zanie

09/28/2022, 3:30 PM
Thanks for tagging! We’ll investigate.
:thank-you: 1
This should be fixed again.
👍 2
We attempted to deploy a long term fix and had to roll it back for an unrelated reason and it caused a brief change here. We’ll be implementing some changes to avoid this in the future.
a

Alexey Stoletny

09/28/2022, 3:47 PM
Appreciate it @Zanie
Just in case, we still have flows sometimes hanging for too long or failing because of a HTTP 500 on Prefect API, which is mostly mitigated via retries but still happens every few days; @Zanie I know you’re also investigating this, so figured I would ask
z

Zanie

09/28/2022, 3:50 PM
Which route are you getting a 500 on?
a

Alexey Stoletny

09/28/2022, 3:50 PM
randomly across a number, like getting a Secret block or even when the flow is being registered behind the scenes
the flows sometimes fail with HTTP 500
Christopher suggested we do retries while this is investigated, and while that fixes it “most of the time” as it is intermittent, it still occasionally happens
@Zanie here’s a link to that: https://prefect-community.slack.com/archives/CL09KU1K7/p1663092920489649?thread_ts=1663087056.534729&amp;cid=CL09KU1K7
sometimes it just times out and the agent becomes slow as it’s hanging in the API calls and flows become Late too
z

Zanie

09/28/2022, 3:53 PM
Thanks! I’m not working on Cloud performance, but I can forward this to the team that is.
I’m deep down this hole https://github.com/PrefectHQ/prefect/pull/6908 🙂
👍 2
a

Alexey Stoletny

09/28/2022, 4:40 PM
Understood, thank you @Zanie
#prefect-communityJoin Slack
Powered by