Tomas Moreno
05/12/2023, 1:31 PMPREFECT__CLOUD__REQUEST_TIMEOUT=60
in our CICD env, but I see timeouts happening in 14s sometimes. anyone have any tips on what could be going on?Traceback (most recent call last):
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 670, in <module>
main()
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 664, in main
create_proj_and_register_flows(flows, args)
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 300, in create_proj_and_register_flows
register_flow(flow, flow_file, args)
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 409, in register_flow
flow.register(
File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/core/flow.py", line 1727, in register
registered_flow = client.register(
File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/client/client.py", line 1176, in register
res = self.graphql(
File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
Bianca Hoch
05/15/2023, 6:43 PMTomas Moreno
05/15/2023, 7:59 PMBianca Hoch
05/15/2023, 8:02 PMTomas Moreno
05/16/2023, 2:06 PM[2021-01-14 22:00:00.000] ERROR --- [{'path': ['create_flow_from_compressed_string'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
Traceback (most recent call last):
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 681, in <module>
main()
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 675, in main
create_proj_and_register_flows(flows, args)
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 302, in create_proj_and_register_flows
register_flow(flow, flow_file, args)
File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 419, in register_flow
flow.register(
File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/core/flow.py", line 1727, in register
registered_flow = client.register(
File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/client/client.py", line 1176, in register
res = self.graphql(
File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
Bianca Hoch
05/16/2023, 2:14 PMTomas Moreno
05/16/2023, 2:23 PMBianca Hoch
05/16/2023, 7:02 PMTomas Moreno
05/16/2023, 7:09 PMScott Aefsky
05/17/2023, 7:53 PMTraceback (most recent call last):
File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/cli/build_register.py", line 475, in build_and_register
flow_id, flow_version, is_new = register_serialized_flow(
File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/cli/build_register.py", line 399, in register_serialized_flow
res = client.graphql(
File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
Bianca Hoch
05/17/2023, 8:21 PMBeizhen
05/19/2023, 12:44 PM[2023-05-19 11:56:17,686] ERROR - agent | Failed to query for ready flow runs
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 320, in _submit_deploy_flow_run_jobs
flow_run_ids = self._get_ready_flow_runs()
File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 571, in _get_ready_flow_runs
result = self.client.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['get_runs_in_queue'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
agent run on prefect 1.2.0-python3.8
I could see from the log that we have at least had this problem since 2023-04-19. Would be so nice to resolve this. Please let me know if I need to provide more info. Thanks.Bianca Hoch
05/24/2023, 9:41 PMBeizhen
05/25/2023, 10:01 AM