https://prefect.io logo
Join Slack
Powered by
Hi everyone! Running into a `create_task_run` erro...
# ask-community
a
Hi everyone! Running into a 
create_task_run
error trying to run a task in a for loop, wondering if anyone has run into this before (error in thread):
Copy code
Crash details:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1669, in report_flow_run_crashes
    yield
  File "/usr/local/lib/python3.10/contextlib.py", line 697, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1168, in create_task_run_then_submit
    task_run = await create_task_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1213, in create_task_run
    task_run = await flow_run_context.client.create_task_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1833, in create_task_run
    response = await <http://self._client.post|self._client.post>(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 227, in send
    response = await self._send_with_retry(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_body(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 220, in _send_request_body
    self._h2_state.end_stream(stream_id)
  File "/usr/local/lib/python3.10/site-packages/h2/connection.py", line 883, in end_stream
    frames = self.streams[stream_id].end_stream()
KeyError: 67
The task is just being called like so:
Copy code
results = []
for data in data_array:
    results.append(
        task_name.submit(
            data=data
        )
    )
This also doesn’t happen consistently, but there’s no obvious difference between runs where this error gets thrown and when it doesn’t
z
That’s really weird. Looks like an HTTP/2 bug
You can disable HTTP/2 with a setting 
PREFECT_API_ENABLE_HTTP2=False
Would you mind opening an issue so we can track it with the upstream http library?
v
@Zanie I'm seeing a similar crash in h2 but from the prefect agent after upgrading to 2.10.5:
Copy code
ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
File "/root/.pyenv/versions/3.8.9/lib/python3.8/site-packages/h2/connection.py", line 224, in process_input
During handling of the above exception, another exception occurred:
File "/root/.pyenv/versions/3.8.9/lib/python3.8/site-packages/httpcore/_async/http2.py", line 116, in handle_async_request
This happens intermittently for our agents which leads them to restart, and sometimes our jobs end up being stuck in "Late" for hours due to this! Any idea what this could be due to? Perhaps this warrants a separate issue in Github...
z
We think that’s an upstream bug and added the ability to turn off HTTP2 if affected https://github.com/PrefectHQ/prefect/issues/7442
It looks like the same issue
v
@Zanie Thanks, I'll try disabling that for our agents! I think it's noteworthy that we went from having like 3 crashes per month to ~20 per day after going from 2.8.6 to 2.10.5
So perhaps that could indicate that it might not just be an upstream issue?
z
I think we may just be using the client more in the newer agent versions? I’m not sure
✅ 1
v
@Zanie Ah, that makes sense. Thanks for the quick replies 🙂
z
It’s definitely an upstream issue unfortunately. If you could provide httpx debug logs as described in https://github.com/PrefectHQ/prefect/issues/7442#issuecomment-1515424020 that’d be really helpful for us to provide to them
👍 1
4 Views
Open in Slack
PreviousNext
Powered by