Viet Nguyen
10/27/2022, 2:50 PMAnna Geller
10/27/2022, 3:44 PMViet Nguyen
10/27/2022, 3:53 PM[ERROR] RuntimeError: Cannot create flow run. Failed to reach API at <https://api.prefect.cloud/api/accounts>
Anna Geller
10/27/2022, 7:21 PMViet Nguyen
10/27/2022, 10:55 PMS3
, needs to use zarr.ProcessSycnchronizer()
so that different invocations wont lapse on each other, causing data corruptions to the Zarr store on S3
Anna Geller
10/27/2022, 11:37 PMViet Nguyen
10/27/2022, 11:41 PMPath to a directory on a file system that is shared by all processes. N.B., this should be a different path to where you store the array.
Path to a directory on a file system that is shared by all processes
on a distributed system like LambdaAnna Geller
10/28/2022, 2:19 AMViet Nguyen
10/28/2022, 2:26 AMEFS which host numpy arrayed serialized with Zarrno for this, EFS only host sync files that multi Lambda invocations access to, this makes sure they write to S3 appropriately, arrays stored on S3. This pipeline to manage data update only (new files added occasionally, or ingested file receiving an update), the actual Zarr store holding aggregated data from hundred thousands of NetCDF files run separately by scheduler.
sync filesProcess Synchronizer specifically
When you connect a function to a VPC in your account, it does not have access to the internet unless your VPC provides access. To give your function access to the internet, route outbound traffic to a NAT gateway in a public subnet.I need to update the route table to allow Lambda accessing internet! Ports weren't a problem @Anna Geller