@Marvin have you seen an error like this?

Task run received abort during orchestration: This run is in a PENDING state and cannot transition to a PENDING state

I am starting a task from a failure hook:

def cleanup_hook(flow: Flow, flow_run: FlowRun, state: State):
    execution_state: FlowCompletionState = state.data.get()
    logger = get_run_logger()
    logger.setLevel(logging.DEBUG)

    <http://logger.info|logger.info>(execution_state.__dict__)
    cleanup_state: State = cleanup(flow_run, execution_state, return_state=True)
    <http://logger.info|logger.info>(cleanup_state.message)

The error message you encountered,

Task run received abort during orchestration: This run is in a PENDING state and cannot transition to a PENDING state

, typically indicates that there is an issue with the state transition of the task run within Prefect. This can occur for a variety of reasons and has been discussed in both Prefect 2.x and 3.x documentation and GitHub issues. ### Possible Causes and Solutions: 1. Incorrect State Transition: Prefect tasks have defined state transitions, and certain transitions are not allowed. For example, moving from

PENDING

to

PENDING

or from

RUNNING

to

RUNNING

. If there is an attempt to transition a task to a state it is already in, this error can occur. - In Prefect 2.x, state management for tasks includes types such as

PENDING

,

RUNNING

, etc., and the allowed transitions are well-defined. - Reference for Prefect 2.x State Management - In Prefect 3.x, the state management works similarly with a focus on preventing invalid state transitions. - Prefect 3.x Manage States 2. Hook and Task Interaction: If you are using hooks to manage task states, ensure that the state transitions defined in the hooks are valid and make sense in the given context. The error might be occurring due to invalid state transition logic in the

cleanup_hook

method. 3. Parallel or Nested Task Runs: There are documented issues when tasks attempt to run other tasks directly or in nested manners, especially in an asynchronous context. - GitHub Issue Example 4. Infrastructure or Network Issues: Sometimes intermittent network issues or infrastructure failures (e.g., spot instance termination in AWS) can cause state management errors. - Related GitHub Issue ### Suggested Steps: - Review Hook Logic: Ensure that the state transitions in the

cleanup_hook

are valid. - Avoid Nested Task Submission: Ensure tasks are not being submitted in a nested manner outside of a flow context. - Retry Mechanism: Implement retry logic or state checking mechanisms to ensure tasks are not repeatedly attempting invalid state transitions. If you are consistently facing this issue and the suggested steps do not resolve it, please provide additional context or code snippets for more in-depth debugging. For more detailed guidance, refer to the Prefect documentation and related GitHub issues.