John-Craig Borman
03/23/2023, 3:38 PMServer error '500 Internal Server Error' for url
in an orchestration flow (run in Prefect Cloud) that is triggering many run_deployment
calls. Is this likely an API limit being tripped?Bianca Hoch
03/23/2023, 5:22 PMJohn-Craig Borman
03/23/2023, 5:27 PMEncountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 643, in orchestrate_flow_run
result = await flow_call()
File "/.../flows/orchestrator.py", line 54, in orchestrator
await utils.run_deployments_throttled(deployment_params, timeout=0)
File "/.../flows/utils.py", line 75, in run_deployments_throttled
return await aiometer.run_all(jobs, max_per_second=max_calls_per_second)
File "/usr/local/lib/python3.10/dist-packages/aiometer/_impl/run_all.py", line 18, in run_all
async with amap(
File "/usr/lib/python3.10/contextlib.py", line 217, in __aexit__
await self.gen.athrow(typ, value, traceback)
File "/usr/local/lib/python3.10/dist-packages/aiometer/_impl/amap.py", line 69, in _amap
async with anyio.create_task_group() as task_group:
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.10/dist-packages/aiometer/_impl/amap.py", line 74, in sender
await run_on_each(
File "/usr/local/lib/python3.10/dist-packages/aiometer/_impl/run_on_each.py", line 52, in run_on_each
async with anyio.create_task_group() as task_group:
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.10/dist-packages/aiometer/_impl/run_on_each.py", line 19, in _worker
result = await async_fn(value)
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments.py", line 123, in run_deployment
flow_run = await client.create_flow_run_from_deployment(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/orion.py", line 433, in create_flow_run_from_deployment
response = await <http://self._client.post|self._client.post>(
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1848, in post
return await self.request(
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 253, in send
response.raise_for_status()
File "/usr/local/lib/python3.10/dist-packages/httpx/_models.py", line 749, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/60b38cc6-c00d-4e61-8757-f9a7fa6b6f90/workspaces/444e91be-a3d5-416b-81a7-0361db5c01d1/deployments/b71ae396-6d0c-4a94-8c38-82f07803d108/create_flow_run>'
For more information check: <https://httpstatuses.com/500>
aiometer
to throttle requests to a max of 5 per second (as I thought this was an API limit issue), but now I'm not so sureBianca Hoch
03/23/2023, 9:38 PMprefect version
in your terminal.John-Craig Borman
03/24/2023, 2:49 PMBianca Hoch
03/24/2023, 3:22 PMrun_deployment
?John-Craig Borman
03/24/2023, 3:25 PMrun_deployment(name=..., parameters={'key': value}, flow_run_name=value, timeout=0)
Bianca Hoch
03/24/2023, 5:34 PMJohn-Craig Borman
03/24/2023, 6:11 PMMatt Kizaric
03/24/2023, 8:12 PMprefect.orion.services
) that have to query the database. These run on a loop that has a configurable timeout in most cases. I was able to bump PREFECT_ORION_SERVICES_PAUSE_EXPIRATIONS_LOOP_SECONDS
to solve one exception. I don't use cloud, but if you can configure that it might help a bit
• There was one Loop for notifications that has a non-configurable timeout (FlowRunNotifications). This has become my new bottleneck, though it might be more due to postgres being under-provisioned.
Not sure if this helps much, but just some details I was able to pull out of my local testingJohn-Craig Borman
03/24/2023, 8:15 PMScott Walsh
04/25/2023, 5:25 PM