Hello team! I'm facing a weird error that I'm not sure if related to Prefect or not: I'm invoking a lambda inside a task and getting the following error:

[2019-10-08 17:26:55,377] ERROR - prefect.TaskRunner | Unexpected error: TooManyRequestsException('An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 869, in get_task_run_state
    self.task.run, timeout=self.task.timeout, **raw_inputs
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 79, in timeout_handler
    return fn(*args, **kwargs)
  File "/Users/aneto/Documents/github/data-platform/scheduler/main.py", line 244, in run_sync_loads
    Payload=json.dumps(load)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.

The function has a 15min timeout and it is being called as:

client = boto3.client('lambda')
sync_results = client.invoke(
            FunctionName='myfunction',
            Payload=json.dumps(load)
        )

The code following this invocation is not being called. I noticed that even before the Prefect retry the function is being called more than once.

This appears to be a boto3 error being raised --> are you able to run the boto3 code on its own without Prefect?

yes, I'll do that!

Hi Argemiro! My immediate thought in looking at this is that the issue might arise from calling boto3 from mapped tasks (I've run into a similar issue). Not sure what your task looks like, but worth a shot!

Solved! The problem was the default SDK timeout of 60s while the function is taking around 10min to complete. The code was fixed based on this doc: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/