Hi there, I'm getting a new graphql client error a...
# ask-communityb
Billy McMonagle
05/10/2022, 5:44 PMHi there, I'm getting a new graphql client error and trying to figure out the cause. This is running on CI/CD and no related code has changed recently...
Billy McMonagle
05/10/2022, 5:45 PMI'm registering a number of flows, and this seems to be failing non-deterministically. The error is:
Copy code
[2022-05-10 17:38:40+0000] INFO - prefect.S3 | Uploading script /codebuild/output/src317307747/src/schedule.py to finance/2022-05-10t17-38-40-057549-00-00 in <bucket>
Traceback (most recent call last):
File "/codebuild/output/src317307747/src/datascience/register.py", line 110, in <module>
register_flow(flow)
File "/codebuild/output/src317307747/src/datascience/register.py", line 93, in register_flow
flow.register(
File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/core/flow.py", line 1727, in register
registered_flow = client.register(
File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 1244, in register
self.graphql(
File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_schedule_active'], 'message': 'unable to perform operation on <TCPTransport closed=True reading=False 0x555efa0e7420>; the handler is closed', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]Billy McMonagle
05/10/2022, 5:45 PMrunning prefect version 0.15.13
k
Kevin Kho
05/10/2022, 5:54 PM
This is a weird error. It looks like the API call is just failing. For these flows, what happens if you register without the schedule and then turn it on in the UI. Does that fail as well?
b
Billy McMonagle
05/10/2022, 5:55 PMI'll see.
Billy McMonagle
05/10/2022, 6:14 PMNo luck, different error. Again, some flows registered successfully, but:
Copy code
[2022-05-10 18:13:13+0000] INFO - prefect.S3 | Uploading script /codebuild/output/src944635918/src/schedule.py to path/2022-05-10t18-13-13-230406-00-00 in <bucket>
Traceback (most recent call last):
File "/codebuild/output/src944635918/src/datascience/register.py", line 110, in <module>
register_flow(flow)
File "/codebuild/output/src944635918/src/datascience/register.py", line 93, in register_flow
flow.register(
File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/core/flow.py", line 1727, in register
registered_flow = client.register(
File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 1176, in register
res = self.graphql(
File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Unable to complete operation', 'extensions': {'code': 'API_ERROR'}}]a
Andrew Lawlor
05/10/2022, 6:17 PMim also seeing this today
š 1
k
Kevin Kho
05/10/2022, 6:18 PM
Thanks for chiming in Andrew. Will elevate this
b
Billy McMonagle
05/10/2022, 6:19 PMThanks for the +1 Andrew and thank you @Kevin Kho for escalating. If there is an additional or alternative support route I should take let me know.
k
Kevin Kho
05/10/2022, 6:21 PM
I left a message. May be slow to respond. We have an onsite also.
b
Billy McMonagle
05/10/2022, 6:25 PMUnderstood, thanks for the context š
a
Anna Geller
05/10/2022, 6:28 PM
I checked with the infra team and it all should be resolved now - could you try again now and report back?
b
Billy McMonagle
05/10/2022, 6:32 PM@Anna Geller will do!
š 1
a
Andrew Lawlor
05/10/2022, 6:36 PMworked for me. thank you
š 1
b
Billy McMonagle
05/10/2022, 6:37 PMWe're green too. Thanks a lot
š 1
b
Ben Ayers-Glassey
05/10/2022, 6:38 PMI was having this error earlier as well:
Copy code
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}]Ben Ayers-Glassey
05/10/2022, 6:39 PMFor instance, here is the output of
prefect run Copy code
Looking up flow metadata... Done
Creating run for flow 'bag_wildfire_perimeters_log'... Done
āāā Name: axiomatic-kestrel
āāā UUID: ca7b3443-26e8-4864-a01f-9a8c4e23cbd0
āāā Labels: []
āāā Parameters: {'shapefile_url': '<http://opendata.arcgis.com/api/v3/datasets/6bd5dd4e93154f6fae0de1f2c82d95bf_0/downloads/data?format=shp&spatialRefId=4326>'}
āāā Context: {}
āāā URL: <https://cloud.prefect.io/zesty-ai/flow-run/ca7b3443-26e8-4864-a01f-9a8c4e23cbd0>
Watching flow run execution...
āāā 11:32:35 | INFO | Entered state <Scheduled>: Flow run scheduled.
āāā 11:32:39 | INFO | Entered state <Submitted>: Submitted for execution
āāā 11:33:21 | INFO | Entered state <Running>: Running flow.
āāā 11:34:31 | INFO | Entered state <Success>: All reference tasks succeeded.
Flow run succeeded!Ben Ayers-Glassey
05/10/2022, 6:40 PMI also can't see logs in the UI, for any flows -- old, new, or currently running. E.g.
https://cloud.prefect.io/zesty-ai/flow-run/ca7b3443-26e8-4864-a01f-9a8c4e23cbd0?logs
No logs found. Try expanding or resetting your search.
Click here to retrieve archived logs - please allow up to 30 seconds for the retrieval to take effect....if I click there, it says "Retrieving archived logs from glacial storage......" for a while. ...and then eventually just says:
Sorry, we couldn't get the logs from glacial storage... Retry?
a
Andrew Lawlor
05/10/2022, 7:05 PMi also cant see logs actually
a
Anna Geller
05/11/2022, 1:35 PM
@Ben Ayers-Glassey and @Andrew Lawlor you are both right and this announcement explains why
b
Ben Ayers-Glassey
05/11/2022, 3:15 PMThanks for the update!
š 1
9 Views