How to connect EFS to AWS Lambda used together with Prefect?

Question

I have prefect agent running on Lambda that connected to a VPC at the moment I allow all traffic from everywhere for the inbound Security group setting but this is not ideal besides port 4200 might I know what other ports I need to allow to get Prefect running on Lambda with a VPC without issue Thank you

Viet Nguyen · Answer

< Anna Geller> long time no chat

Viet Nguyen · Answer

hope you re doing well slightly smiling face

Anna Geller · Answer

I hope you re well too wave

Anna Geller · Answer

this repo template may be a good way to get started there s a link to a blog post in readme <https github com anna geller prefect aws lambda>

Viet Nguyen · Answer

yeah I got it worked perfectly 10mins ago when the Lambda func is not connected to a VPC

Viet Nguyen · Answer

but since I need to use AWS EFS for the Lambda func I need to hook it to a VPC

Viet Nguyen · Answer

and that requires security group settings

Viet Nguyen · Answer

``` ERROR RuntimeError Cannot create flow run Failed to reach API at <https api prefect cloud api accounts>```

Viet Nguyen · Answer

and it used to be successful running like this

Viet Nguyen · Answer

logs from Cloudwatch

Viet Nguyen · Answer

code and env variables for the Lambda func stay the same

Viet Nguyen · Answer

just adding a VPC to Lambda

Viet Nguyen · Answer

so I wonder which ports Prefect running in the background consumes I don t want to open all ports

Viet Nguyen · Answer

and the Lambda func has too hook to an EFS because multiple Lambda calls that writing to same Zarr stor require a Process synchronizer for write consistency

Viet Nguyen · Answer

thanks < Anna Geller>

Anna Geller · Answer

sorry I can t help with EFS and I m not sure if I would recommend that with Lambda it adds so much complexity why not use S3

Viet Nguyen · Answer

S3 still a destination for Zarr EFS for sync files

Viet Nguyen · Answer

multiple Lambda invocations trying to write data to the Same Zarr store on `S3` needs to use `zarr ProcessSycnchronizer ` so that different invocations wont lapse on each other causing data corruptions to the Zarr store on `S3`

Viet Nguyen · Answer

anyway I figured it out

Viet Nguyen · Answer

all I needed to know was which ports Prefect API and agent using not asking how to use Prefect with Lambda or use Prefect with Lambda EFS combination so I think it s a miscommunication slightly smiling face

Anna Geller · Answer

oh wow can you share how you did it I m curious

Anna Geller · Answer

it s just more that you know more about it than me I wouldn t honestly know how to use self hosted Prefect with Lambda I only used it with Cloud anyway great to hear you figured it out and if you want to share how you did it that could be useful to others raised hands

Viet Nguyen · Answer

I dont host Prefect too I use Prefect cloud

Viet Nguyen · Answer

hang on I have some diagram

Viet Nguyen · Answer

so this is a high level logics <https zarr readthedocs io en stable api sync html> gt Path to a directory on a file system that is shared by all processes N B this should be a different path to where you store the array

Viet Nguyen · Answer

and this is the a bit more details logics

Viet Nguyen · Answer

EFS solves this `Path to a directory on a file system that is shared by all processes` on a distributed system like Lambda

Viet Nguyen · Answer

to maintain the accuracy of the S3 source data store when multiple invocations trying to update the same Zarr store at the same time

Viet Nguyen · Answer

so it s not self hosted Prefect flows run showing up in your cloud look at the address bar

Viet Nguyen · Answer

reupload first pic high level diagram

Viet Nguyen · Answer

< Anna Geller>

Viet Nguyen · Answer

because there are API connections between the Lambda func with Prefect cloud with a VPC hooked in a a Security group attached required if attach EFS to Lambda func it needs to be granted access so that the 2 things can talk to each other e g the Lambda can access Prefect cloud API does that make sense to you

Anna Geller · Answer

so Lambda is needed so that some flow runs are triggered event driven when new data arrives but this Lambda needs access to EFS which host numpy arrayed serialized with Zarr did I get it So it looks like your Lambda is doing all the work and Prefect is used to observe Lambda only

Anna Geller · Answer

Thanks so much for sharing Appreciate it

Viet Nguyen · Answer

no worries besides observing I have some state management logics within the flows and tasks also use Prefect notification on tagged runs to me Lambda is just like an agent we process only 1 file per invocation so Lambda has enough computing resource and timeout of 15 min is not a problem gt EFS which host numpy arrayed serialized with Zarr no for this EFS only host sync files that multi Lambda invocations access to this makes sure they write to S3 appropriately arrays stored on S3 This pipeline to manage data update only new files added occasionally or ingested file receiving an update the actual Zarr store holding aggregated data from hundred thousands of NetCDF files run separately by scheduler

Viet Nguyen · Answer

The pre generated big Zarr store creation pipelines run on ECS Fargate cluster and using Dask slightly smiling face those Lambda pipelines running without Dask just 1 file not a big deal

Viet Nguyen · Answer

gt sync files Process Synchronizer specifically

Viet Nguyen · Answer

realised that I didn t answer you how I fixed the problem when adding a VPC to Lambda gt When you connect a function to a VPC in your account it does not have access to the internet unless your VPC provides access To give your function access to the internet route outbound traffic to a NAT gateway in a public subnet I need to update the route table to allow Lambda accessing internet Ports weren t a problem < Anna Geller>