https://prefect.io logo
Powered by
Title
a

Alissa Wang

04/10/2023, 3:29 PM
Hi everyone! Running into a 
create_task_run
error trying to run a task in a for loop, wondering if anyone has run into this before (error in thread):
Crash details:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1669, in report_flow_run_crashes
    yield
  File "/usr/local/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1168, in create_task_run_then_submit
    task_run = await create_task_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1213, in create_task_run
    task_run = await flow_run_context.client.create_task_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1833, in create_task_run
    response = await <http://self._client.post|self._client.post>(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 227, in send
    response = await self._send_with_retry(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_body(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 220, in _send_request_body
    self._h2_state.end_stream(stream_id)
  File "/usr/local/lib/python3.10/site-packages/h2/connection.py", line 883, in end_stream
    frames = self.streams[stream_id].end_stream()
KeyError: 67
The task is just being called like so: 
results = []
for data in data_array:
    results.append(
        task_name.submit(
            data=data
        )
    )
This also doesn’t happen consistently, but there’s no obvious difference between runs where this error gets thrown and when it doesn’t
z

Zanie

04/10/2023, 8:38 PM
That’s really weird. Looks like an HTTP/2 bug
You can disable HTTP/2 with a setting 
PREFECT_API_ENABLE_HTTP2=False
Would you mind opening an issue so we can track it with the upstream http library?
v

vholmer

04/26/2023, 1:29 PM
@Zanie I'm seeing a similar crash in h2 but from the prefect agent after upgrading to 2.10.5: 
ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
File "/root/.pyenv/versions/3.8.9/lib/python3.8/site-packages/h2/connection.py", line 224, in process_input
During handling of the above exception, another exception occurred:
File "/root/.pyenv/versions/3.8.9/lib/python3.8/site-packages/httpcore/_async/http2.py", line 116, in handle_async_request
This happens intermittently for our agents which leads them to restart, and sometimes our jobs end up being stuck in "Late" for hours due to this! Any idea what this could be due to? Perhaps this warrants a separate issue in Github...
z

Zanie

04/26/2023, 2:51 PM
We think that’s an upstream bug and added the ability to turn off HTTP2 if affected https://github.com/PrefectHQ/prefect/issues/7442
It looks like the same issue
v

vholmer

04/26/2023, 3:07 PM
@Zanie Thanks, I'll try disabling that for our agents! I think it's noteworthy that we went from having like 3 crashes per month to ~20 per day after going from 2.8.6 to 2.10.5
So perhaps that could indicate that it might not just be an upstream issue?
z

Zanie

04/26/2023, 3:09 PM
I think we may just be using the client more in the newer agent versions? I’m not sure
✅ 1
v

vholmer

04/26/2023, 3:09 PM
@Zanie Ah, that makes sense. Thanks for the quick replies 🙂
z

Zanie

04/26/2023, 3:10 PM
It’s definitely an upstream issue unfortunately. If you could provide httpx debug logs as described in https://github.com/PrefectHQ/prefect/issues/7442#issuecomment-1515424020 that’d be really helpful for us to provide to them
👍 1
2 Views
#prefect-communityJoin Slack
Powered by