Hi, I'm running my flow on a kubernetes pod, I have self-hosted agent and worker. Now I see the error logs like this:

Worker 'KubernetesWorker 7763f002-66a8-475f-90cf-b40e048d3e02' submitting flow run '83edf78a-26ec-4f1a-870b-c82b96f25db2'
Creating Kubernetes job...
Job 'axiomatic-oyster-lcgft': Pod has status 'Pending'.
Completed submission of flow run '83edf78a-26ec-4f1a-870b-c82b96f25db2'
Job 'axiomatic-oyster-lcgft': Pod has status 'Running'.
Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/xxxx/workspaces/xxxx/block_schemas/>'
Response: {'detail': 'Workspace scopes missing: manage_blocks'}
For more information check: <https://httpstatuses.com/403>
Job 'axiomatic-oyster-lcgft': Job reached backoff limit.

I masked our account id and workspace id. Anyone has any ideas? Thanks.

Hi @Frost Ouyang it seems Kubernetes killed your worker because it failed repetitively, and that failed because it's hitting a 403 when trying to reach Prefect. Maybe double check your API key if correct, that you have the right account/workspace IDs - in case you haven't already

Thank you. I contacted prefect support today and figured out that it's because our role doesn't have enough access.

Aha, you guys are using the new ACL :)