https://prefect.io logo
Title
h

Hedgar

10/03/2022, 3:08 PM
@Anna Geller I’m still working on implementing the aws-lambda-serverless-flow template example. On a closer look at the repo, I can see that you didn’t build deployment with prefect(no deployment scripts). Was this intentional or an omission? My attempt to build and apply deployment led to this error:`File "/home/ec2-user/.servy/lib/python3.9/site-packages/botocore/handlers.py", line 278, in validate_bucket_name` raise ParamValidationError(report=error_msg) botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid bucket name “s3:“: Bucket name must match the regex “^[a-zA-Z0-9.\-_]{1,255}$” or be an ARN matching the regex “^arn:(aws).:(s3|s3-object-lambda):[a-z\-0-9]:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*😒3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$”
1
a

Anna Geller

10/03/2022, 4:49 PM
Lambda is what deploys and runs those flows, not Prefect. Check out https://github.com/anna-geller/prefect-aws-lambda/blob/master/serverless.yml and the accompanying Blog post to get a better picture of how this works
h

Hedgar

10/03/2022, 6:53 PM
@Anna Geller I have deployed successfully but no data is being written to my s3 bucket and I cant see anything on my ui
a

Anna Geller

10/03/2022, 7:27 PM
What UI - do you mean flow run from Lambda is not recorded in the Prefect backend? If so, then you didn't set the PREFECT_API_URL of your Prefect Cloud workspace
h

Hedgar

10/04/2022, 11:17 AM
@Anna Geller Got it! Thanks. This literally means I can safely do away with the “stop and start ec2 schedule” to run my flow, I had under prefect 1. Wow!!👏
a

Anna Geller

10/04/2022, 12:55 PM
Yeah exactly 💯
h

Hedgar

10/04/2022, 4:45 PM
@Anna Geller I however observed that in the Dockerfile wherein we are pulling from
public.ecr.aws/lambda/python:3.9
doesn’t allow installation from requirement.txt file. I have tried by copying the
requirement.txt
file and running
pip install -r requirement.txt
but alway end with error that “no such file found”.-cant’ build. What do you think could be wrong? Although I observed in the template you install the packages directly.
a

Anna Geller

10/04/2022, 6:56 PM
try:
ADD requirements.txt .
you can also list packages this way:
RUN pip install prefect requests pandas package1 package2 package3 package4
h

Hedgar

10/04/2022, 7:39 PM
@Anna Geller ok. meanwhile last question on this serverless with lambda. I have a directory :
engineroom
inside it I have several
.py
files each with a task function. also inside this directory I have
engineroom.py
in this py file I imported the various task function which I called under a flow e.g
from nnfm import get_nnfm_ideas
from sofia import get_sofia_ideas
@flow(name="run all ideas")
def gather_all_ideas():
        get_nnfm_ideas()
        get_sofia_ideas()
#def handler(event,context):
      gather_all_ideas()
if __name__ == "__main__":
    gather_all_ideas()
I was able to run locally successfully by doing:
python engineroom/engineroom.py
My challenge is how to represent in
Dockerfile
and the
serverless.yml
file to run “serverlessly” I have tried but all I get is “module engineroom not found” sorry for taking your time!
a

Anna Geller

10/04/2022, 9:06 PM
You would need to copy the whole directory and add it to Pythonpath or build a package within your image