https://prefect.io logo
Join Slack
Powered by
I know this stack is huge, but I'm completely at a...
# prefect-cloud
g
I know this stack is huge, but I'm completely at a loss why my agent is crashing .. here is the traceback:
Copy code
12:51:35.373 | INFO    | prefect.runner - Pausing schedules for all deployments...
12:52:40.797 | INFO    | prefect.runner - All deployment schedules have been paused!
Traceback (most recent call last):
  File "/app/scripts/jobs.py", line 140, in <module>
Failed the last 3 attempts. Please check your environment and configuration.
Examples of recent errors:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 106, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 203, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions

d
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
    serve(
  File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 142, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/services.py", line 53, in critical_service_loop
    await workload()
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 501, in _get_and_submit_flow_runs
    runs_response = await self._get_scheduled_flow_runs()
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 614, in _get_scheduled_flow_runs
    scheduled_flow_runs = await self._client.read_flow_runs(
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/orchestration.py", line 1778, in read_flow_runs
    response = await <http://self._client.post|self._client.post>("/flow_runs/filter", json=body)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 256, in send
    response = await self._send_with_retry(
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 190, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
    return call()
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 382, in __call__
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
    return self.result()
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 282, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 989, in serve
    await runner.start()
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 283, in start
    async with self._loops_task_group as tg:
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/services.py", line 121, in critical_service_loop
    raise RuntimeError("Service exceeded error threshold.")
RuntimeError: Service exceeded error threshold.
✅ 2
c
Does this happen shortly after calling 
.serve
? were there any errors that you saw leading up to this?
g
No, just during the normal course of running a flow scheduled every 30 minutes (backup of a DB)
here are the preceding log messages
Copy code
11:23:41.654 | INFO    | Flow run 'sassy-wildebeest' - Downloading flow code from storage at '/app'
11:23:46.133 | INFO    | Flow run 'sassy-wildebeest' - Backing up database
11:23:46.135 | INFO    | Flow run 'sassy-wildebeest' - About to dump mongo DB using command
11:23:47.279 | INFO    | Flow run 'sassy-wildebeest' - Finished in state Completed()
11:23:49.686 | INFO    | prefect.flow_runs.runner - Process 396 exited cleanly.
11:53:41.430 | INFO    | prefect.flow_runs.runner - Runner 'runner-d736fb2b-970c-4d98-a12b-bc5a6f9b0bc5' submitting flow run '492c4242-da7a-41f8-88cc-76b9b3ed6a9b'
11:53:41.589 | INFO    | prefect.flow_runs.runner - Opening process...
11:53:41.596 | INFO    | prefect.flow_runs.runner - Completed submission of flow run '492c4242-da7a-41f8-88cc-76b9b3ed6a9b'
/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
11:53:43.844 | INFO    | Flow run 'enormous-shrew' - Downloading flow code from storage at '/app'
11:53:47.882 | INFO    | Flow run 'enormous-shrew' - Backing up database
11:53:47.883 | INFO    | Flow run 'enormous-shrew' - About to dump mongo DB using command
11:53:49.028 | INFO    | Flow run 'enormous-shrew' - Finished in state Completed()
11:53:51.454 | INFO    | prefect.flow_runs.runner - Process 417 exited cleanly.
12:23:46.918 | INFO    | prefect.flow_runs.runner - Runner 'runner-d736fb2b-970c-4d98-a12b-bc5a6f9b0bc5' submitting flow run 'c9c5faee-2272-45f7-9162-a587911119ee'
12:23:47.081 | INFO    | prefect.flow_runs.runner - Opening process...
12:23:47.083 | INFO    | prefect.flow_runs.runner - Completed submission of flow run 'c9c5faee-2272-45f7-9162-a587911119ee'
/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
12:23:49.646 | INFO    | Flow run 'military-python' - Downloading flow code from storage at '/app'
12:23:54.008 | INFO    | Flow run 'military-python' - Backing up database
12:23:54.011 | INFO    | Flow run 'military-python' - About to dump mongo DB using command
12:23:55.025 | INFO    | Flow run 'military-python' - Finished in state Completed()
12:23:57.450 | INFO    | prefect.flow_runs.runner - Process 438 exited cleanly.
c
Gotcha, ok I think I know what's going on -- can you upgrade 
httpcore
in your environment?
Copy code
pip install -U httpcore
g
That might take a bit of experimentation since it's a dependent package from my poetry env
c
gotcha, that's fair -- there's an alternative route in the meantime
if you set 
PREFECT_API_ENABLE_HTTP2=False
in your environment, that should also resolve the issue; longer term i recommend upgrading the underlying library, but I totally understand that you'll need to approach that with more care
g
Fantastic, yeah I just dumped the poetry dependency tree for httpcore, and it goes all the way up to a python package that I maintain.. So, I'll have to go back a bit, update that package, then bump my docker container's version, get the new httpcore, and push a new docker image 🙂 In the mean time, I'll set that ENV var.
😅 1
Thank you
👍 1
Side note, this gives me another push to move to workers instead of one long running 
serve().
💯 1
code crashes, at least it won't crash my agent
@Chris White.. FYI, I upgraded httpcore and it still crashed. Same backtrace. I'll add the env var: 
PREFECT_API_ENABLE_HTTP2=False
next
🧐 1
Also, (FYI) this was also with a worker started with 
prefect worker start --pool 'pool-1'
AND, just implementing a simple demo script
Copy code
import httpx
from prefect import flow


@flow(log_prints=True)
def get_repo_info(repo_name: str = "PrefectHQ/prefect"):
    """ Demo flow """
    url = f"<https://api.github.com/repos/{repo_name}>"
    response = httpx.get(url)
    response.raise_for_status()
    repo = response.json()
    print(f"{repo_name} repository statistics 🤓:")
    print(f"Stars 🌠 : {repo['stargazers_count']}")
    print(f"Forks 🍴 : {repo['forks_count']}")
Copy code
Traceback (most recent call last):
  File "/Users/johnsturgeon/.pyenv/versions/3.10.5/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
Related???
c
Yup it looks to me like the exact same issue - I'm surprised httpcore upgrade didn't work
g
There could be some strange packaging issue on my local machine. My prod server has not crashed since upgrading httpcore
c
ah that's great to hear, and even better that prod is the healthy environment 😄
6 Views
Open in Slack
PreviousNext
Powered by