https://prefect.io logo
Title
b

Billy McMonagle

05/10/2022, 5:44 PM
Hi there, I'm getting a new graphql client error and trying to figure out the cause. This is running on CI/CD and no related code has changed recently...
I'm registering a number of flows, and this seems to be failing non-deterministically. The error is:
[2022-05-10 17:38:40+0000] INFO - prefect.S3 | Uploading script /codebuild/output/src317307747/src/schedule.py to finance/2022-05-10t17-38-40-057549-00-00 in <bucket>
Traceback (most recent call last):
  File "/codebuild/output/src317307747/src/datascience/register.py", line 110, in <module>
    register_flow(flow)
  File "/codebuild/output/src317307747/src/datascience/register.py", line 93, in register_flow
    flow.register(
  File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/core/flow.py", line 1727, in register
    registered_flow = client.register(
  File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 1244, in register
    self.graphql(
  File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_schedule_active'], 'message': 'unable to perform operation on <TCPTransport closed=True reading=False 0x555efa0e7420>; the handler is closed', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
running prefect version 0.15.13
k

Kevin Kho

05/10/2022, 5:54 PM
This is a weird error. It looks like the API call is just failing. For these flows, what happens if you register without the schedule and then turn it on in the UI. Does that fail as well?
b

Billy McMonagle

05/10/2022, 5:55 PM
I'll see.
No luck, different error. Again, some flows registered successfully, but:
[2022-05-10 18:13:13+0000] INFO - prefect.S3 | Uploading script /codebuild/output/src944635918/src/schedule.py to path/2022-05-10t18-13-13-230406-00-00 in <bucket>
Traceback (most recent call last):
  File "/codebuild/output/src944635918/src/datascience/register.py", line 110, in <module>
    register_flow(flow)
  File "/codebuild/output/src944635918/src/datascience/register.py", line 93, in register_flow
    flow.register(
  File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/core/flow.py", line 1727, in register
    registered_flow = client.register(
  File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 1176, in register
    res = self.graphql(
  File "/root/.pyenv/versions/3.9.5/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Unable to complete operation', 'extensions': {'code': 'API_ERROR'}}]
a

Andrew Lawlor

05/10/2022, 6:17 PM
im also seeing this today
šŸ‘€ 1
k

Kevin Kho

05/10/2022, 6:18 PM
Thanks for chiming in Andrew. Will elevate this
b

Billy McMonagle

05/10/2022, 6:19 PM
Thanks for the +1 Andrew and thank you @Kevin Kho for escalating. If there is an additional or alternative support route I should take let me know.
k

Kevin Kho

05/10/2022, 6:21 PM
I left a message. May be slow to respond. We have an onsite also.
b

Billy McMonagle

05/10/2022, 6:25 PM
Understood, thanks for the context šŸ™‚
a

Anna Geller

05/10/2022, 6:28 PM
I checked with the infra team and it all should be resolved now - could you try again now and report back?
b

Billy McMonagle

05/10/2022, 6:32 PM
@Anna Geller will do!
šŸ‘ 1
a

Andrew Lawlor

05/10/2022, 6:36 PM
worked for me. thank you
:thank-you: 1
b

Billy McMonagle

05/10/2022, 6:37 PM
We're green too. Thanks a lot
šŸ™Œ 1
b

Ben Ayers-Glassey

05/10/2022, 6:38 PM
I was having this error earlier as well:
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}]
...it's now fixed in the sense that I can run flows, however I don't seem to be able to see their logs.
For instance, here is the output of
prefect run
with `--watch`:
Looking up flow metadata... Done
Creating run for flow 'bag_wildfire_perimeters_log'... Done
ā””ā”€ā”€ Name: axiomatic-kestrel
ā””ā”€ā”€ UUID: ca7b3443-26e8-4864-a01f-9a8c4e23cbd0
ā””ā”€ā”€ Labels: []
ā””ā”€ā”€ Parameters: {'shapefile_url': '<http://opendata.arcgis.com/api/v3/datasets/6bd5dd4e93154f6fae0de1f2c82d95bf_0/downloads/data?format=shp&spatialRefId=4326>'}
ā””ā”€ā”€ Context: {}
ā””ā”€ā”€ URL: <https://cloud.prefect.io/zesty-ai/flow-run/ca7b3443-26e8-4864-a01f-9a8c4e23cbd0>
Watching flow run execution...
ā””ā”€ā”€ 11:32:35 | INFO    | Entered state <Scheduled>: Flow run scheduled.
ā””ā”€ā”€ 11:32:39 | INFO    | Entered state <Submitted>: Submitted for execution
ā””ā”€ā”€ 11:33:21 | INFO    | Entered state <Running>: Running flow.
ā””ā”€ā”€ 11:34:31 | INFO    | Entered state <Success>: All reference tasks succeeded.
Flow run succeeded!
...none of the logging my tasks do is showing up, nor is it even logging which tasks are running.
I also can't see logs in the UI, for any flows -- old, new, or currently running. E.g. https://cloud.prefect.io/zesty-ai/flow-run/ca7b3443-26e8-4864-a01f-9a8c4e23cbd0?logs
No logs found. Try expanding or resetting your search.
Click here to retrieve archived logs - please allow up to 30 seconds for the retrieval to take effect.
...if I click there, it says "Retrieving archived logs from glacial storage......" for a while. ...and then eventually just says:
Sorry, we couldn't get the logs from glacial storage... Retry?
a

Andrew Lawlor

05/10/2022, 7:05 PM
i also cant see logs actually
a

Anna Geller

05/11/2022, 1:35 PM
@Ben Ayers-Glassey and @Andrew Lawlor you are both right and this announcement explains why
b

Ben Ayers-Glassey

05/11/2022, 3:15 PM
Thanks for the update!
šŸ‘ 1