Alissa Wang
04/10/2023, 3:29 PMcreate_task_run
error trying to run a task in a for loop, wondering if anyone has run into this before (error in thread):Crash details:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1669, in report_flow_run_crashes
yield
File "/usr/local/lib/python3.10/contextlib.py", line 697, in __aexit__
cb_suppress = await cb(*exc_details)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1168, in create_task_run_then_submit
task_run = await create_task_run(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1213, in create_task_run
task_run = await flow_run_context.client.create_task_run(
File "/usr/local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1833, in create_task_run
response = await <http://self._client.post|self._client.post>(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 227, in send
response = await self._send_with_retry(
File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 187, in _send_with_retry
response = await request()
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
raise exc
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
await self._send_request_body(request=request, stream_id=stream_id)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 220, in _send_request_body
self._h2_state.end_stream(stream_id)
File "/usr/local/lib/python3.10/site-packages/h2/connection.py", line 883, in end_stream
frames = self.streams[stream_id].end_stream()
KeyError: 67
results = []
for data in data_array:
results.append(
task_name.submit(
data=data
)
)
Zanie
04/10/2023, 8:38 PMPREFECT_API_ENABLE_HTTP2=False
vholmer
04/26/2023, 1:29 PMERROR | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
File "/root/.pyenv/versions/3.8.9/lib/python3.8/site-packages/h2/connection.py", line 224, in process_input
During handling of the above exception, another exception occurred:
File "/root/.pyenv/versions/3.8.9/lib/python3.8/site-packages/httpcore/_async/http2.py", line 116, in handle_async_request
This happens intermittently for our agents which leads them to restart, and sometimes our jobs end up being stuck in "Late" for hours due to this! Any idea what this could be due to? Perhaps this warrants a separate issue in Github...Zanie
04/26/2023, 2:51 PMvholmer
04/26/2023, 3:07 PMZanie
04/26/2023, 3:09 PMvholmer
04/26/2023, 3:09 PMZanie
04/26/2023, 3:10 PM