Giuliano Mega01/20/2023, 11:44 AM
which I guess isn't my fault as the Prefect agent is in control of rate limiting calls to the logging API. Looking into Prefect Cloud's UI it indeed seems to be dropping logs, which I don't think was happening before. Is this a known issue? Anything I can do to mitigate?
--- Orion logging error --- Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/logging/handlers.py", line 151, in send_logs await client.create_logs(self._pending_logs) File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 1830, in create_logs await <http://self._client.post|self._client.post>(f"/logs/", json=serialized_logs) File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post return await self.request( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 253, in send response.raise_for_status() File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 745, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/[account]/workspaces/[workspace]/logs/>' For more information check: <https://httpstatuses.com/429> Worker information: Approximate queue length: 0 Pending log batch length: 458 Pending log batch size: 164880 The log worker has tried to send these logs 4 times and will now drop them.
Christopher Boyd01/20/2023, 3:20 PM
Giuliano Mega01/20/2023, 5:11 PM