Thuy Tran
08/08/2022, 8:52 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 646, in _lsdir
async for i in it:
File "/usr/local/lib/python3.8/site-packages/aiobotocore/paginate.py", line 32, in __anext__
response = await self._make_request(current_kwargs)
File "/usr/local/lib/python3.8/site-packages/aiobotocore/client.py", line 265, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 104, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 47, in load_flow_from_flow_run
await storage_block.get_directory(from_path=None, local_path=".")
File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 373, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 251, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 111, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 96, in sync
raise return_result
File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in _runner
result[0] = await coro
File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 536, in _get
rpaths = await self._expand_path(rpath, recursive=recursive)
File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 736, in _expand_path
out = await self._expand_path([path], recursive, maxdepth)
File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 754, in _expand_path
rec = set(await self._find(p, maxdepth=maxdepth, withdirs=True))
File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 718, in _find
out = await self._lsdir(path, delimiter="", prefix=prefix)
File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 669, in _lsdir
raise translate_boto_error(e)
PermissionError: The AWS Access Key Id you provided does not exist in our records.
Mason Menges
08/08/2022, 9:29 PMThuy Tran
08/08/2022, 9:50 PM/root/.aws:/root/.aws
(aws file path containing aws config settings). Then when executing prefect deployment build
I specified the docker block -ib docker-container/my-dock-block-name
and ba da bing ba da boom, it worked. But yes, the custom infrastructure is what we plan to do since we also have to set Env EXTRA_PIP_PACKAGES
for required libraries including s3fs.Mason Menges
08/08/2022, 9:51 PM