https://prefect.io logo
Join Slack
Powered by
Hi everyone! I am invoking an AWS Lambda function ...
# ask-community
a
Hi everyone! I am invoking an AWS Lambda function from prefect using boto3 sdk. It runs for 7 mins roughly. It is invoked synchronously but prefect never gets the response message back from the lambda, even if the lambda finishes and the task fails throwing a ReadTimeout error. I increased the timeout to 15 mins but still the same result. This happens randomly, sometimes it finishes sometimes not. Does this have something to do with how prefect handles a task? How to solve this?
n
hi @Abhishek Mitra could you share the code you're using to invoke the lambda and how it interacts with prefect? > sometimes it finishes sometimes not. do you mean the lambda or the prefect task/flow wrapping the invocation?
a
Copy code
from prefect import task, flow,get_run_logger
import json
import botocore.config

cfg = botocore.config.Config(retries={'max_attempts': 0}, read_timeout=900, connect_timeout=900,tcp_keepalive=True)

@task(log_prints=True,timeout_seconds=3000)
def lambda_call(aws_credentials,func_name,payload_bytes,log_dict):
    
    payload = { "body" : json.dumps(payload_bytes)}
    payload_byte = bytes(json.dumps(payload), encoding='utf8')
    lambdas = aws_credentials.get_boto3_session().client("lambda",config=cfg)
    response = lambdas.invoke(
    FunctionName=func_name,
    InvocationType="RequestResponse",
    Payload=payload_byte
)
    
    lambda_payload = response['Payload']
    lambda_text = lambda_payload.read()
    my_json = lambda_text.decode('utf8').replace("'", '"')
    data = json.loads(my_json)
    if data['statusCode'] == 200:
        return "Success"
    else:
        err = json.loads(data['body'])      
        raise Exception(err['error_message'])


@flow()
def main_flow():
    try: 
        lambda_call(aws_creds, lambda_func_name, payload, log_dict)
    
    except Exception:
        raise


if __name__=='__main__':
    main_flow()
@Nate the lambda always finishes but the prefect task sometimes runs successfully sometimes it throws the timeout error
n
where in the above code would you expect to hear back from the lambda? it seems (like this PR) that you're just kicking it off, not polling for status
d
@Abhishek Mitra It seems that all you are using the Lambda response for is to check for errors and log them. There are two ways for you to prevent timeouts: 1. Extend the task timeout to 15 minutes (lambda max) 2. Use the Event invocation type The latter means you can’t log the precise error, but you may be able to find it in CloudWatch or elsewhere in AWS
🙏 1
a
Thanks @Dominic Tarro I have already tried by extending the timeout to 15 mins here
Copy code
cfg = botocore.config.Config(retries={'max_attempts': 0}, read_timeout=900, connect_timeout=900,tcp_keepalive=True)
but it still times out, The lambda finishes as I mentioned but the response doesn't reach prefect. I might have to go for the 2nd option.
Thanks @Nate for the link. Is the lambda support coming to prefect @Dominic Tarro?
d
@Abhishek Mitra The feature is currently awaiting review. I can’t speak to when it will be released. Just to rule out potential sources of error, could you see what 
tcp_keepalive=False
does? I don’t know if this is what’s going on, but there may be a break in communication between the Lambda and client and the Keep-Alive is purging the connection without the client knowing, causing it to wait until the timeout. Worth a try.
👁️ 1
👍 1
n
I've just approved the PR, I'd be able to cut a release and get that out provided main is otherwise gtg (it should be). I can update the release notes and check that out and update here once released - thanks again for the contribution @Dominic Tarro catjam
🙌 2
a
@Dominic Tarro I'll check that out and let u know.
Thanks @Nate. This would be very helpful.
n
prefect-aws
v0.4.7 has been released, here are the docs for the new `LambdaFunction` block the new block-type will be made available in Prefect Cloud (auto registered your workspace) next time we push out changes (which happens rather often) if you want it before then, you're free to 
prefect block resigter -m prefect_aws
after installing the new library thanks again @Dominic Tarro for the contribution 💙 - let us know if anything looks off
🙌 1
👀
description looks a little odd but we can PR that later 🙂
d
Oh yeah I just copied it from another block and added slight context
a
@Dominic Tarro it is still getting read timeout error with 
tcp_keepalive=False
d
Hmm. I don’t have a good answer then. A few things I would try 1. Try with the newly released 
LambdaFunction
block 2. Try without passing a boto config 3. Try updating boto version If those don’t work there’s not much else I can do from here
👍 1
20 Views
Open in Slack
PreviousNext
Powered by