Hi. I'm having trouble running a deployment with docker infrastructure and s3 storage. I think it has to do with the aws region. Our S3 is in a special east region; how do we configure docker to use the aws s3 config on the server that has the region?

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 646, in _lsdir
    async for i in it:
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/paginate.py", line 32, in __anext__
    response = await self._make_request(current_kwargs)
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/client.py", line 265, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 47, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 373, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 251, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 536, in _get
    rpaths = await self._expand_path(rpath, recursive=recursive)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 736, in _expand_path
    out = await self._expand_path([path], recursive, maxdepth)
  File "/usr/local/lib/python3.8/site-packages/fsspec/asyn.py", line 754, in _expand_path
    rec = set(await self._find(p, maxdepth=maxdepth, withdirs=True))
  File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 718, in _find
    out = await self._lsdir(path, delimiter="", prefix=prefix)
  File "/usr/local/lib/python3.8/site-packages/s3fs/core.py", line 669, in _lsdir
    raise translate_boto_error(e)
PermissionError: The AWS Access Key Id you provided does not exist in our records.

Hey @Thuy Tran I'm not super familiar with aws but I believe you should be able to create a custom infrastructure block to accommodate this https://docs.prefect.io/concepts/infrastructure/#using-infrastructure

Thanks @Mason Menges. I figured out I needed to edit docker block and set Volumes to

/root/.aws:/root/.aws

(aws file path containing aws config settings). Then when executing

prefect deployment build

I specified the docker block

-ib docker-container/my-dock-block-name

and ba da bing ba da boom, it worked. But yes, the custom infrastructure is what we plan to do since we also have to set Env

EXTRA_PIP_PACKAGES

for required libraries including s3fs.

That's awesome, Glad you were able to sort it out 😄