https://prefect.io logo
#prefect-community
Title
# prefect-community
t

Thuy Tran

08/08/2022, 8:52 PM
Hi. I'm having trouble running a deployment with docker infrastructure and s3 storage. I think it has to do with the aws region. Our S3 is in a special east region; how do we configure docker to use the aws s3 config on the server that has the region?
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 646, in _lsdir
    async for i in it:
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/paginate.py", line 32, in __anext__
    response = await self._make_request(current_kwargs)
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/client.py", line 265, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 47, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 373, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 251, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 536, in _get
    rpaths = await self._expand_path(rpath, recursive=recursive)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 736, in _expand_path
    out = await self._expand_path([path], recursive, maxdepth)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 754, in _expand_path
    rec = set(await self._find(p, maxdepth=maxdepth, withdirs=True))
  File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 718, in _find
    out = await self._lsdir(path, delimiter="", prefix=prefix)
  File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 669, in _lsdir
    raise translate_boto_error(e)
PermissionError: The AWS Access Key Id you provided does not exist in our records.
👀 1
✅ 1
m

Mason Menges

08/08/2022, 9:29 PM
Hey @Thuy Tran I'm not super familiar with aws but I believe you should be able to create a custom infrastructure block to accommodate this https://docs.prefect.io/concepts/infrastructure/#using-infrastructure
t

Thuy Tran

08/08/2022, 9:50 PM
Thanks @Mason Menges. I figured out I needed to edit docker block and set Volumes to
/root/.aws:/root/.aws
(aws file path containing aws config settings). Then when executing
prefect deployment build
I specified the docker block
-ib docker-container/my-dock-block-name
and ba da bing ba da boom, it worked. But yes, the custom infrastructure is what we plan to do since we also have to set Env
EXTRA_PIP_PACKAGES
for required libraries including s3fs.
🙌 4
m

Mason Menges

08/08/2022, 9:51 PM
That's awesome, Glad you were able to sort it out 😄
10 Views