https://prefect.io logo
Join Slack
Powered by
Hi all - quick question for you - we’re running a ...
# ask-community
b
Hi all - quick question for you - we’re running a flow which is running a lambda function. Works fine in test with serverless framework, but on AWS itself, the .run() function is simply freezing and then never returning. It doesn’t seem to ever timeout, and we’re not getting an error message back. Does anyone have experience of similar issues - we’re assuming it’s a timeout due to permissions and checking IAM roles, security groups and so on, but theoretically these are set correctly. Are there any obvious gotchas or flags we could use to debug this wrapped function?
k
Hi @Ben Collier, any CloudWatch logs that show it at least got triggered? You can run your flwo in debug mode for more info. Most common gotcha is instantiating a boto3 Client outside tasks and the flow, so it doesn’t get respected during runtime if you have timeout configurations or stuff like that
b
So yes, it’s already in debug mode, and we see no logs at all from the command after it’s called. LambdaInvoke() runs fine, but it’s when we call .run() on the returned client that everything grinds to a halt.
It’s all instantiated inside the task.
(thanks for the quick response btw)
One question - we’re doing a pip install -r on the packages imported by pip install “prefect[aws]” originally, but does pip install “prefect[aws]” do something special which means we need to specifically call it when deploying?
Does it set a flag somewhere or something?
k
pip install prefect[aws]
just installs boto3 as an extra (and maybe other stuff). Nothing special. Could you explain more on the 
.run()
on the returned Client? Are you returning the client from one task and passing it on to the next?
b
Copy code
lambda_client = LambdaInvoke(function_name=f"name")
lambda_client.run(payload=json.dumps(
        {
            "ids": some_ids,
        }))
The first line of code works - the second freezes.
It works fine locally.
But that involves setting
Copy code
boto_kwargs=dict(endpoint_url=LAMBDA_ENDPOINT)
in the Invoke call.
It’s all inside the one task.
k
Gotcha, Yeah this looks good to me. What agent are you using? Does local mean 
flow.run()
? Or do you also see this when you use local agent on your laptop and try it as a registered flow from Prefect Cloud/Server?
b
So, local is serverless framework running locally, with a prefect local agent and a flow installed alongside it, triggered from the prefect cloud. That works. The online version (for now) is the same thing but with the agent dockerised, and calling actual AWS lambda. My guess is that the problem is just a timeout, but I’m surprised that the function call never returns and we never see a timeout error.
It’s pretty odd though. Most of the reasons we’d expect a timeout are covered, and we’re not seeing a permissions problem either.
k
I see. Could it be that machine doesn’t have permissions to sent those outbound requests? I assume you tried the “local run” on that machine too?
Open in Slack
PreviousNext
Powered by