Hi I m trying to understand how remote storage should be use

Question

Hi I m trying to understand how remote storage should be used to distribute flows to a heterogeneous execution environment I ve drawn a couple of diagrams to clarify my understanding The first diagram shows the creation of a deployment in the orion server via the cli Config goes to orion code goes to the s3 remote storage bucket The agent subscribes to the work queue So far so good The second diagram shows the prefect agent receiving an run event from the queue and spawning a `myapp` docker container for execution

Mark NS · Answer

What I would now expect is that the `myapp` container downloads the flows from the s3 storage bucket and the entrypoint is executed However instead the container fails with an s3fs Access Denied error ``` File app venv lib python3 11 site packages s3fs core py line 774 in find out = await self lsdir path delimiter= prefix=prefix ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File app venv lib python3 11 site packages s3fs core py line 725 in lsdir raise translate boto error e PermissionError Access Denied``` I would have thought distributing these authentication creds should be handled by the agent and the s3 block somehow

Kelvin DeCosta · Answer

I think the Docker container must be explicitly supplied with the required credentials

Mark NS · Answer

Hmm I don t see anything about supplying the s3 credentials in the documentation here <https docs prefect io tutorials docker >

Kelvin DeCosta · Answer

Could be 2 cases `S3Bucket` and `S3Block` are two separate block types which confused me in the past For a storage block you need the one in `prefect filesystems` The S3 block takes aws credentials as optional fields So maybe it needs to be specified in this case

Mark NS · Answer

Thanks but yep already doing that ``` s3 = S3 bucket path=os environ get AWS S3 BUCKET NAME DEFAULT BLOCK aws access key id=os environ get AWS ACCESS KEY ID DEFAULT BLOCK aws secret access key=os environ get AWS SECRET ACCESS KEY DEFAULT BLOCK s3 save default ``` I m going through <https docs prefect io tutorials docker |the tutorial> I linked again Maybe my custom image is not working the same way as the default image

Mark NS · Answer

Also I know the s3 block is correctly configured as the upload on `prefect deployment build sb s3 default ` works fine

Kelvin DeCosta · Answer

hmm that is weird indeed

Mark NS · Answer

Ok so using the default image `prefecthq prefect 2 7 3 python3 10` the flows are downloaded correctly from remote storage Trying to figure out why my custom image using `python 3 slim` with ```prefect = ^2 7 3 prefect dbt = ^0 2 5 s3fs = ^2022 11 0 ``` installed is not behaving the same way

davzucky · Answer

We will be using the pattern of creating a docker image with our flow and not use the S3 storage Did you consider this pattern

Mark NS · Answer

Hi < davzucky> I was initially creating a docker image with the flow runs and cloning the application code at runtime However I realised this approach wouldn t work in an environment where you are trying to orchestrate diverse runtimes

Mark NS · Answer

Sorry for the late reply too