I use

prefect_aws

to upload/list/download files to s3 and also for some shared AWS Secrets. For some functionality I rely on

boto3

, e.g.

boto3.client("s3", ...).generate_presigned_url(…)

. Prefect 2.5.0 is running on EKS and the flows are deployed to S3 which requires

s3fs

.

To use it in a deployment: prefect deployment […] -sb s3/dev 
You need to install s3fs to use this block.

https://docs.prefect.io/concepts/filesystems/ My problem is that

prefect_aws

and

s3fs

have dependency conflicts. I am using pip-tools to set my requirements and get the following error:

# simplified <http://requirements.in|requirements.in> (removed the package versions to might easier find matches)
prefect
prefect_aws
s3fs

$ pip-compile <http://requirements.in|requirements.in>
Could not find a version that matches botocore<1.27.60,<1.28.0,>=1.27.53,>=1.27.59,>=1.27.89 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
Tried: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0 ... [lists all versions here] 
1.27.87, 1.27.88, 1.27.88, 1.27.89, 1.27.89
Skipped pre-versions: 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0rc1
There are incompatible versions in the resolved dependencies:
  botocore<1.28.0,>=1.27.89 (from boto3==1.24.89->prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
  botocore>=1.27.53 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
  botocore<1.27.60,>=1.27.59 (from aiobotocore==2.4.0->s3fs==2022.8.2->-r <http://requirements.in|requirements.in> (line 3))

I have found this issue: https://github.com/fsspec/s3fs/issues/615#issuecomment-1094791081 but not a real solution to fix it. How can I use

prefect_aws

and also deploy flows to S3? Does anyone else have the same problem and found a solution?

we plan to introduce S3 block to prefect_aws collection that would only rely on boto3 without s3fs so this should get even easier in the future but no ETA on that for now, can you try switching the order of installing packages? I'm sure it should work to install both prefect_aws and s3fs

That’s great to hear! I would like to contribute in any way if possible 🙂 Thank you for your help! Switching the order did not help unfortunately. But as always after hours of trying, several minutes after posting this issue, I have found the solution 😄 I installed s3fs and prefect_aws with pip in a clean environment (which could somehow resolve it) and then put the exact versions into the requirements.in

prefect==2.5.0
prefect_aws==0.1.4
s3fs==2022.8.2
boto3==1.24.59
botocore==1.27.59

great work! ❤️ and absolutely, feel free to contribute to the collection. The goal here is that we would build the same interface with get_directory and put_directory but using plain boto3 without s3fs so that all blocks and tasks in this collection can rely on a single client and make it easier to e.g. share the AwsCredentials block across all those blocks -- providing those to a boto3 session object and adding this credentials block as a nested block within S3 block if you would want to give it a try, the naming would need to be different, could be something like S3Storage or S3BucketPath

I just had a quick look into the S3 Block and how the storages are used in the methods of the Deployment class. Unfortunately

boto3

is not async compatible. So I see two non-optimal ways: 1. Wrap

boto3

sync function to make them async (probably reduces performance because of the overhead) 2. Use a third party package like

aioboto3

which is an asynchronous version of

boto3

(but not sure how stable it is)

All correct, I'll cc @alex, he knows this best

There is an

S3Bucket

block that currently exists in

prefect-aws

, but in its current form it doesn’t work with deployments. There is a PR here to make that block compatible with deployments: https://github.com/PrefectHQ/prefect-aws/pull/82. Unfortunately, it’s gone a little stale, but I think I should be able to revive it and get it merged this week!

oh that’s great, thank you!