Argemiro Neto
10/08/2019, 5:32 PM[2019-10-08 17:26:55,377] ERROR - prefect.TaskRunner | Unexpected error: TooManyRequestsException('An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 869, in get_task_run_state
self.task.run, timeout=self.task.timeout, **raw_inputs
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 79, in timeout_handler
return fn(*args, **kwargs)
File "/Users/aneto/Documents/github/data-platform/scheduler/main.py", line 244, in run_sync_loads
Payload=json.dumps(load)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.
The function has a 15min timeout and it is being called as:
client = boto3.client('lambda')
sync_results = client.invoke(
FunctionName='myfunction',
Payload=json.dumps(load)
)
The code following this invocation is not being called. I noticed that even before the Prefect retry the function is being called more than once.Chris White
10/08/2019, 5:38 PMArgemiro Neto
10/08/2019, 5:40 PMZachary Hughes
10/08/2019, 5:51 PMArgemiro Neto
10/08/2019, 6:23 PM