https://prefect.io logo
Title
a

Argemiro Neto

10/08/2019, 5:32 PM
Hello team! I'm facing a weird error that I'm not sure if related to Prefect or not: I'm invoking a lambda inside a task and getting the following error:
[2019-10-08 17:26:55,377] ERROR - prefect.TaskRunner | Unexpected error: TooManyRequestsException('An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 869, in get_task_run_state
    self.task.run, timeout=self.task.timeout, **raw_inputs
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 79, in timeout_handler
    return fn(*args, **kwargs)
  File "/Users/aneto/Documents/github/data-platform/scheduler/main.py", line 244, in run_sync_loads
    Payload=json.dumps(load)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.
The function has a 15min timeout and it is being called as:
client = boto3.client('lambda')
sync_results = client.invoke(
            FunctionName='myfunction',
            Payload=json.dumps(load)
        )
The code following this invocation is not being called. I noticed that even before the Prefect retry the function is being called more than once.
c

Chris White

10/08/2019, 5:38 PM
This appears to be a boto3 error being raised --> are you able to run the boto3 code on its own without Prefect?
👍 1
a

Argemiro Neto

10/08/2019, 5:40 PM
yes, I'll do that!
👍 1
z

Zachary Hughes

10/08/2019, 5:51 PM
Hi Argemiro! My immediate thought in looking at this is that the issue might arise from calling boto3 from mapped tasks (I've run into a similar issue). Not sure what your task looks like, but worth a shot!
👍 1
a

Argemiro Neto

10/08/2019, 6:23 PM
Solved! The problem was the default SDK timeout of 60s while the function is taking around 10min to complete. The code was fixed based on this doc: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/
💯 3