Hey, is there a theoretical limit to the number of child tasks created from a mapped task? The server gets really clogged up if I create >1000 child tasks & cancel the run. Each task fails in the server output with something like the following, which is expected but it takes about 5 minutes for 1000 child tasks to cancel.

graphql_1    | ERROR:    State update failed for task run ID 191a019d-4e0a-4692-a05a-823659c3f535: provided a running state but associated flow run 635f983d-8082-49c3-9dcd-af694e400ced is not in a running state.
graphql_1    | 
graphql_1    | GraphQL request:6:7
graphql_1    | 5 |       status
graphql_1    | 6 |       id
graphql_1    |   |       ^
graphql_1    | 7 |     }
graphql_1    | Traceback (most recent call last):
graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 668, in complete_value_catching_error
graphql_1    |     return_type, field_nodes, info, path, result
graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
graphql_1    |     raise result
graphql_1    |   File "/prefect-server/src/prefect_server/graphql/states.py", line 73, in set_state
graphql_1    |     task_run_id=state_input["task_run_id"], state=state,
graphql_1    |   File "/prefect-server/src/prefect_server/api/states.py", line 91, in set_task_run_state
graphql_1    |     f"State update failed for task run ID {task_run_id}: provided "
graphql_1    | ValueError: State update failed for task run ID 191a019d-4e0a-4692-a05a-823659c3f535: provided a running state but associated flow run 635f983d-8082-49c3-9dcd-af694e400ced is not in a running state.
graphql_1    | 
graphql_1    | The above exception was the direct cause of the following exception:
graphql_1    | 
graphql_1    | Traceback (most recent call last):
graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 668, in complete_value_catching_error
graphql_1    |     return_type, field_nodes, info, path, result
graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
graphql_1    |     raise result
graphql_1    |   File "/prefect-server/src/prefect_server/graphql/states.py", line 73, in set_state
graphql_1    |     task_run_id=state_input["task_run_id"], state=state,
graphql_1    |   File "/prefect-server/src/prefect_server/api/states.py", line 91, in set_task_run_state
graphql_1    |     f"State update failed for task run ID {task_run_id}: provided "
graphql_1    | graphql.error.graphql_error.GraphQLError: State update failed for task run ID 191a019d-4e0a-4692-a05a-823659c3f535: provided a running state but associated flow run 635f983d-8082-49c3-9dcd-af694e400ced is not in a running state.
graphql_1    | 
graphql_1    | GraphQL request:6:7
graphql_1    | 5 |       status
graphql_1    | 6 |       id
graphql_1    |   |       ^
graphql_1    | 7 |     }

Hi @Howard Cornwell there are actually some cancellation API changes coming down the pipe that I feel have the opportunity to help resolve the issues seen here! Going to cc @Jim Crist-Harif for visibility. For some context to the output above what I believe is happening is you are marking the flow run as cancelled, which then updates each task run’s state to the same. In the current iteration of cancellation all of those tasks still have to make an attempt to start to verify their current state and it’s raising this issue because on the attempt to start in

Running

it is actually finding that it is

Cancelled

.

Yup, that makes sense of what I thought was happening. I’ll keep the volume of mapped tasks down for now Thanks for the response.