Mark NS
12/21/2022, 9:48 AMmyapp
docker container for execution.myapp
container downloads the flows from the s3 storage bucket, and the entrypoint is executed.
However, instead the container fails with an s3fs Access Denied error.
File "/app/.venv/lib/python3.11/site-packages/s3fs/core.py", line 774, in _find
out = await self._lsdir(path, delimiter="", prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/s3fs/core.py", line 725, in _lsdir
raise translate_boto_error(e)
PermissionError: Access Denied
I would have thought distributing these authentication creds should be handled by the agent and the s3 block somehow.Kelvin DeCosta
12/21/2022, 9:55 AMMark NS
12/21/2022, 10:02 AMKelvin DeCosta
12/21/2022, 10:11 AMS3Bucket
and S3Block
are two separate block types, which confused me in the past. For a storage block, you need the one in prefect.filesystems
• The S3 block takes aws credentials as optional fields. So, maybe it needs to be specified in this caseMark NS
12/21/2022, 10:14 AMs3 = S3(bucket_path=os.environ.get("AWS_S3_BUCKET_NAME", DEFAULT_BLOCK),
aws_access_key_id=os.environ.get("AWS_ACCESS_KEY_ID", DEFAULT_BLOCK),
aws_secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY", DEFAULT_BLOCK)
)
s3.save("default")
I'm going through the tutorial I linked again. Maybe my custom image is not working the same way as the default imageprefect deployment build -sb s3/default ...
works fineKelvin DeCosta
12/21/2022, 10:21 AMMark NS
12/21/2022, 10:28 AMprefecthq/prefect:2.7.3-python3.10
the flows are downloaded correctly from remote storage.
Trying to figure out why my custom image using python:3-slim
with
prefect = "^2.7.3"
prefect-dbt = "^0.2.5"
s3fs = "^2022.11.0"
installed is not behaving the same way...davzucky
12/22/2022, 1:09 AMMark NS
01/06/2023, 1:10 PM