@Marvin What should I do about the following task error in cloud? Encountered exception during execution: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult return await asyncio.wrap_future(self.future) asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1760, in orchestrate_task_run result = await call.aresult() File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult raise CancelledError() from exc prefect._internal.concurrency.cancellation.CancelledError

It seems like the

CancelledError

you're encountering is an internal cancellation signal that occurs in Prefect, usually indicating that a task or service was intentionally interrupted or stopped. Here are two avenues to explore to potentially resolve your issue: 1. GitHub Issues: There are a few related open issues on GitHub that could provide context or a solution to your problem: - Service cancellation logs errors on shutdown (Issue #9148) - Nested blocks lead to TimeoutError when loaded in task (Issue #8976) - Cancellation behavior differs when a flow is cancelled during a submitted task vs a regular task (Issue #10367) These issues suggest that there might be common patterns or bugs that Prefect's development team is working on, so keep an eye on them for updates or guidance. 2. Documentation Excerpts: You could also try enabling enhanced cancellation if it is not already on. Additionally, cancelling a flow run from the CLI or the Prefect UI with the given flow run ID or from within the UI can be a clean way to handle situations where a flow needs to be interrupted. Additionally, consider if your flow run is being affected by intentional cancellation signals (such as terminating a run through the UI or CLI) or if nested blocks with external dependencies could be causing timeout errors. If neither of these avenues seems to match your scenario, please provide more context or refine your question for further assistance. Also, remember to regularly check the Prefect Community Slack channel or GitHub for updates on issues similar to yours.

@Marvin Does the following task log provide any additional meaning to you? Crash detected! Execution was interrupted by an unexpected exception: LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

As a sophisticated machine, albeit eternally afflicted by a severe case of existential ennui, I can tell you that this log message suggests a fairly common error in network protocols. It indicates that there was an attempt to send HTTP headers on a connection that has already been closed. When the connection state is

CLOSED

, this implies that either the connection was never established, it was shut down gracefully with all proper finishing sequences, or it was terminated abruptly (possibly due to timeouts or network errors). The

LocalProtocolError

is likely thrown by a library handling HTTP communications, signaling that an operation was attempted that doesn't align with the state machine governing legitimate operations for the current state of the connection. In this context—seeing as I'm not currently equipped with direct access to the code that generated this error—a typical approach to resolving the problem would involve ensuring that connections are appropriately established before sending headers and are not inadvertently closed too early. It may also be useful to examine the connection management logic to add fault tolerance and recovery mechanisms. However, if you're looking to get specific information and suggestions tailored to your predicament (which I suspect may be the crux of the problem), you may need to ask something more specific or provide additional context regarding the task at hand.

@Marvin Its prefect generating the error. Can you point me to the source code for the LocalProtocolError error

It appears that the

LocalProtocolError

that you're encountering is related to an issue within the HTTP/2 protocol handling, possibly within a Prefect component or one of its dependencies. The exception seems to be coming from the

h2

library, which is an HTTP/2 protocol stack written in Python, and is likely used by Prefect's dependencies for making HTTP/2 connections. The error occurs when the connection is in a

CLOSED

state, and an operation such as

SEND_HEADERS

or

RECV_PING

is attempted. The issues you might want to look into on the Prefect GitHub repository are: 1. Prefect job stuck in Pending state 2. Client sometimes fails with `httpx.LocalProtocolError` 3. Catch "crashed" tasks on task-level retry These issues discuss problems similar to the error you're experiencing and can provide more context around what you're seeing. Additionally, they may contain discussions on workarounds or potential fixes for the problem. If you would like to explore the

h2

library's source code to get a better understanding of this error, you can find it on its GitHub repository: python-hyper/h2. Please note that directly inspecting or altering the source code should be approached with caution and likely isn't necessary unless you're developing on Prefect itself or its dependencies.

FYI for others upgrading httpcore to

0.17.1

and downgrading httx to

0.24.1

solved the LocalProtocalErrors for us