Viet Nguyen10/27/2022, 2:50 PM
Viet Nguyen10/27/2022, 3:53 PM
[ERROR] RuntimeError: Cannot create flow run. Failed to reach API at <https://api.prefect.cloud/api/accounts>
Viet Nguyen10/27/2022, 10:55 PM
, needs to use
so that different invocations wont lapse on each other, causing data corruptions to the Zarr store on
Viet Nguyen10/27/2022, 11:41 PM
Path to a directory on a file system that is shared by all processes. N.B., this should be a different path to where you store the array.
on a distributed system like Lambda
Path to a directory on a file system that is shared by all processes
Viet Nguyen10/28/2022, 2:26 AM
EFS which host numpy arrayed serialized with Zarrno for this, EFS only host sync files that multi Lambda invocations access to, this makes sure they write to S3 appropriately, arrays stored on S3. This pipeline to manage data update only (new files added occasionally, or ingested file receiving an update), the actual Zarr store holding aggregated data from hundred thousands of NetCDF files run separately by scheduler.
sync filesProcess Synchronizer specifically
When you connect a function to a VPC in your account, it does not have access to the internet unless your VPC provides access. To give your function access to the internet, route outbound traffic to a NAT gateway in a public subnet.I need to update the route table to allow Lambda accessing internet! Ports weren't a problem @Anna Geller