Amanda Wee
05/13/2021, 1:14 AM12:28:38: Task 'get_vend_outlet[0]': Starting task run...
12:28:46: Error getting flow run info (...) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: <http://internal-load-balancer.amazonaws.com:80/graphql>
12:28:59: Task 'get_vend_outlet[0]': Finished task run for task with final state: 'Success'
(... some successful task runs ...)
12:29:58: Task 'setup_retailer_outlet_register_dataframe[0]': Starting task run...
12:37:39: No heartbeat detected from the remote task; marking the run as failed.
12:40:38: Rescheduled by a Lazarus process. This is attempt 1.
12:40:43: Flow run is no longer in a running state; the current state is: <Scheduled: "Rescheduled by a Lazarus process.">
(... flow run restarts ...)
(... some successful task runs ...)
12:40:50: Task 'setup_retailer_outlet_register_dataframe[0]': Starting task run...
12:40:50: Task 'setup_retailer_outlet_register_dataframe[0]': Finished task run for task with final state: 'Failed'
(.. some successful task runs and also trigger failed task runs ...)
12:41:26: Flow run FAILED: some reference tasks failed.
12:41:47: Flow run is no longer in a running state; the current state is: <Failed: "Some reference tasks failed.">
12:48:43: Task 'get_vend_register_open_sequence[0]': Starting task run...
12:48:43: prefect.utilities.exceptions.ClientError: [{'message': 'State update failed for task run ID 168abb5c-a7af-4ec7-b221-81637152cf24: provided a running state but associated flow run 9d271ae3-58bf-4501-8a84-036c8e24fa6d is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 168abb5c-a7af-4ec7-b221-81637152cf24: provided a running state but associated flow run 9d271ae3-58bf-4501-8a84-036c8e24fa6d is not in a running state.'}}}]
12:48:43: Task 'get_vend_register_open_sequence[0]': Finished task run for task with final state: 'ClientFailed'
(... same error for next task ...)
(... some task runs with Finished task run for task with final state: 'Pending' ...)
12:48:45: Flow run is no longer in a running state; the current state is: <Failed: "Some reference tasks failed.">
12:48:45: Flow run RUNNING: terminal tasks are incomplete.
12:48:45: Heartbeat process died with exit code 1
(end)
Weirdly, if I look at setup_retailer_outlet_register_dataframe[0]
, it says it succeeded (and so does the parent task before the mapping). So I'm confused.Kevin Kho
05/13/2021, 2:48 PMmerlin
05/13/2021, 4:17 PMAmanda Wee
05/20/2021, 2:20 AMKevin Kho
05/20/2021, 2:24 AMAmanda Wee
05/20/2021, 3:57 AMKevin Kho
05/20/2021, 12:33 PM