Thread
#prefect-community
    h

    Howard Cornwell

    2 years ago
    Hey, is there a theoretical limit to the number of child tasks created from a mapped task? The server gets really clogged up if I create >1000 child tasks & cancel the run. Each task fails in the server output with something like the following, which is expected but it takes about 5 minutes for 1000 child tasks to cancel.
    graphql_1    | ERROR:    State update failed for task run ID 191a019d-4e0a-4692-a05a-823659c3f535: provided a running state but associated flow run 635f983d-8082-49c3-9dcd-af694e400ced is not in a running state.
    graphql_1    | 
    graphql_1    | GraphQL request:6:7
    graphql_1    | 5 |       status
    graphql_1    | 6 |       id
    graphql_1    |   |       ^
    graphql_1    | 7 |     }
    graphql_1    | Traceback (most recent call last):
    graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 668, in complete_value_catching_error
    graphql_1    |     return_type, field_nodes, info, path, result
    graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
    graphql_1    |     raise result
    graphql_1    |   File "/prefect-server/src/prefect_server/graphql/states.py", line 73, in set_state
    graphql_1    |     task_run_id=state_input["task_run_id"], state=state,
    graphql_1    |   File "/prefect-server/src/prefect_server/api/states.py", line 91, in set_task_run_state
    graphql_1    |     f"State update failed for task run ID {task_run_id}: provided "
    graphql_1    | ValueError: State update failed for task run ID 191a019d-4e0a-4692-a05a-823659c3f535: provided a running state but associated flow run 635f983d-8082-49c3-9dcd-af694e400ced is not in a running state.
    graphql_1    | 
    graphql_1    | The above exception was the direct cause of the following exception:
    graphql_1    | 
    graphql_1    | Traceback (most recent call last):
    graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 668, in complete_value_catching_error
    graphql_1    |     return_type, field_nodes, info, path, result
    graphql_1    |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
    graphql_1    |     raise result
    graphql_1    |   File "/prefect-server/src/prefect_server/graphql/states.py", line 73, in set_state
    graphql_1    |     task_run_id=state_input["task_run_id"], state=state,
    graphql_1    |   File "/prefect-server/src/prefect_server/api/states.py", line 91, in set_task_run_state
    graphql_1    |     f"State update failed for task run ID {task_run_id}: provided "
    graphql_1    | graphql.error.graphql_error.GraphQLError: State update failed for task run ID 191a019d-4e0a-4692-a05a-823659c3f535: provided a running state but associated flow run 635f983d-8082-49c3-9dcd-af694e400ced is not in a running state.
    graphql_1    | 
    graphql_1    | GraphQL request:6:7
    graphql_1    | 5 |       status
    graphql_1    | 6 |       id
    graphql_1    |   |       ^
    graphql_1    | 7 |     }
    j

    josh

    2 years ago
    Hi @Howard Cornwell there are actually some cancellation API changes coming down the pipe that I feel have the opportunity to help resolve the issues seen here! Going to cc @Jim Crist-Harif for visibility. For some context to the output above what I believe is happening is you are marking the flow run as cancelled, which then updates each task run’s state to the same. In the current iteration of cancellation all of those tasks still have to make an attempt to start to verify their current state and it’s raising this issue because on the attempt to start in
    Running
    it is actually finding that it is
    Cancelled
    .
    h

    Howard Cornwell

    2 years ago
    Yup, that makes sense of what I thought was happening. I’ll keep the volume of mapped tasks down for now Thanks for the response.