https://prefect.io logo
n

Nico Neumann

10/11/2022, 11:29 PM
I use
prefect_aws
to upload/list/download files to s3 and also for some shared AWS Secrets. For some functionality I rely on
boto3
, e.g.
boto3.client("s3", ...).generate_presigned_url(…)
. Prefect 2.5.0 is running on EKS and the flows are deployed to S3 which requires
s3fs
.
Copy code
To use it in a deployment: prefect deployment […] -sb s3/dev 
You need to install s3fs to use this block.
https://docs.prefect.io/concepts/filesystems/ My problem is that
prefect_aws
and
s3fs
have dependency conflicts. I am using pip-tools to set my requirements and get the following error:
Copy code
# simplified <http://requirements.in|requirements.in> (removed the package versions to might easier find matches)
prefect
prefect_aws
s3fs
Copy code
$ pip-compile <http://requirements.in|requirements.in>
Could not find a version that matches botocore<1.27.60,<1.28.0,>=1.27.53,>=1.27.59,>=1.27.89 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
Tried: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0 ... [lists all versions here] 
1.27.87, 1.27.88, 1.27.88, 1.27.89, 1.27.89
Skipped pre-versions: 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0rc1
There are incompatible versions in the resolved dependencies:
  botocore<1.28.0,>=1.27.89 (from boto3==1.24.89->prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
  botocore>=1.27.53 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
  botocore<1.27.60,>=1.27.59 (from aiobotocore==2.4.0->s3fs==2022.8.2->-r <http://requirements.in|requirements.in> (line 3))
I have found this issue: https://github.com/fsspec/s3fs/issues/615#issuecomment-1094791081 but not a real solution to fix it. How can I use
prefect_aws
and also deploy flows to S3? Does anyone else have the same problem and found a solution?
1
a

Anna Geller

10/11/2022, 11:33 PM
we plan to introduce S3 block to prefect_aws collection that would only rely on boto3 without s3fs so this should get even easier in the future but no ETA on that for now, can you try switching the order of installing packages? I'm sure it should work to install both prefect_aws and s3fs
🙏 1
n

Nico Neumann

10/11/2022, 11:49 PM
That’s great to hear! I would like to contribute in any way if possible 🙂 Thank you for your help! Switching the order did not help unfortunately. But as always after hours of trying, several minutes after posting this issue, I have found the solution 😄 I installed s3fs and prefect_aws with pip in a clean environment (which could somehow resolve it) and then put the exact versions into the requirements.in
Copy code
prefect==2.5.0
prefect_aws==0.1.4
s3fs==2022.8.2
boto3==1.24.59
botocore==1.27.59
💯 1
a

Anna Geller

10/12/2022, 12:09 AM
great work! ❤️ and absolutely, feel free to contribute to the collection. The goal here is that we would build the same interface with get_directory and put_directory but using plain boto3 without s3fs so that all blocks and tasks in this collection can rely on a single client and make it easier to e.g. share the AwsCredentials block across all those blocks -- providing those to a boto3 session object and adding this credentials block as a nested block within S3 block if you would want to give it a try, the naming would need to be different, could be something like S3Storage or S3BucketPath
🙌 1
n

Nico Neumann

10/12/2022, 2:02 AM
I just had a quick look into the S3 Block and how the storages are used in the methods of the Deployment class. Unfortunately
boto3
is not async compatible. So I see two non-optimal ways: 1. Wrap
boto3
sync function to make them async (probably reduces performance because of the overhead) 2. Use a third party package like
aioboto3
which is an asynchronous version of
boto3
(but not sure how stable it is)
I have just found that
prefect_aws
already uses such functionality as in 1. and running the sync function in another thread using
from prefect.utilities.asyncutils import run_sync_in_worker_thread
a

Anna Geller

10/12/2022, 11:19 AM
All correct, I'll cc @alex, he knows this best
👍 1
a

alex

10/12/2022, 11:44 AM
There is an
S3Bucket
block that currently exists in
prefect-aws
, but in its current form it doesn’t work with deployments. There is a PR here to make that block compatible with deployments: https://github.com/PrefectHQ/prefect-aws/pull/82. Unfortunately, it’s gone a little stale, but I think I should be able to revive it and get it merged this week!
🚀 1
🙏 1
n

Nico Neumann

10/12/2022, 12:29 PM
oh that’s great, thank you!