https://prefect.io logo
j

Jeremy Hetzel

09/01/2023, 3:47 PM
I'm noticing this error in the past few minutes (from within an ACI worker). Is it just a transient server side error?
Copy code
prefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url '<https://api.prefect.cloud/api/accounts/16005bcf-76b3-4ba3-867f-e6f0b3faba97/workspaces/168ac782-a029-429a-968e-e14a6c0f6f06/block_documents/74d920bb-9410-4ef5-b8b8-fbace47b66e0?include_secrets=true>'
Response: {'exception_message': 'Service Unavailable'}
2
1
Full traceback:
Copy code
13:10:23.772 | INFO    | prefect.flow_runs.worker - Worker 'my-aci-worker' submitting flow run 'a8fbb97f-9c58-46cf-bdb6-9f3a7a3ea2b0'
13:10:41.663 | ERROR   | prefect.flow_runs.worker - Failed to submit flow run 'a8fbb97f-9c58-46cf-bdb6-9f3a7a3ea2b0' to infrastructure.
Traceback (most recent call last):
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/workers/base.py", line 832, in _submit_run_and_capture_errors
    submitted_event = self._emit_flow_run_submitted_event(configuration)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/workers/base.py", line 908, in _get_configuration
    values=deployment.infra_overrides or {},
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/workers/base.py", line 125, in from_template_and_values
    populated_configuration = await resolve_block_document_references(
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/utilities/templating.py", line 205, in resolve_block_document_references
    updated_value = await resolve_block_document_references(
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/utilities/templating.py", line 201, in resolve_block_document_references
    block_document = await client.read_block_document(block_document_id)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1317, in read_block_document
    response = await self._client.get(
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/client/base.py", line 280, in send
    response = PrefectResponse.from_httpx_response(response)
  File "/home/jhetzel/src/c-core-labs/holmes/.venv/lib/python3.10/site-packages/prefect/client/base.py", line 138, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url '<https://api.prefect.cloud/api/accounts/16005bcf-76b3-4ba3-867f-e6f0b3faba97/workspaces/168ac782-a029-429a-968e-e14a6c0f6f06/block_documents/74d920bb-9410-4ef5-b8b8-fbace47b66e0?include_secrets=true>'
Response: {'exception_message': 'Service Unavailable'}
For more information check: <https://httpstatuses.com/503>
13:10:41.668 | INFO    | prefect.flow_runs.worker - Completed submission of flow run 'a8fbb97f-9c58-46cf-bdb6-9f3a7a3ea2b0'
13:10:41.814 | INFO    | prefect.flow_runs.worker - Reported flow run 'a8fbb97f-9c58-46cf-bdb6-9f3a7a3ea2b0' as crashed: Flow run could not be submitted to infrastructure
s

Scott Henley

09/01/2023, 3:48 PM
Seeing the same thing. Just started about 20 min ago.
👍 1
l

Leonardo Perazzini

09/01/2023, 3:50 PM
@Nate can you help us ?
n

Nate

09/01/2023, 3:52 PM
hi all - looking
typingcat 3
j

Jeremy Hetzel

09/01/2023, 3:57 PM
@Nate, thanks, this looks resolved.
j

Jack P

09/01/2023, 3:58 PM
Just worked on a deployment for me in GCP too 😄
n

Nate

09/01/2023, 3:58 PM
i believe we've identified the cause of error - it should be resolved, but we should have a status update on the degradation that did occur momentarily.
🙌 1
❤️ 3
j

Jack P

09/01/2023, 3:58 PM
Thanks @Nate !
n

Nate

09/01/2023, 3:58 PM
thanks for the reports all - please let us know if anything is still not seeming right
🚀 1
b

Bianca Hoch

09/01/2023, 4:36 PM
Hello everyone! As Nate mentioned, there was a brief period of degraded performance which has been resolved. Please take a look at the Cloud status page for more details.
🙌 2