Nico Neumann
10/11/2022, 11:29 PMprefect_aws
to upload/list/download files to s3 and also for some shared AWS Secrets. For some functionality I rely on boto3
, e.g. boto3.client("s3", ...).generate_presigned_url(…)
. Prefect 2.5.0 is running on EKS and the flows are deployed to S3 which requires s3fs
.
To use it in a deployment: prefect deployment […] -sb s3/dev
You need to install s3fs to use this block.
https://docs.prefect.io/concepts/filesystems/
My problem is that prefect_aws
and s3fs
have dependency conflicts. I am using pip-tools to set my requirements and get the following error:
# simplified <http://requirements.in|requirements.in> (removed the package versions to might easier find matches)
prefect
prefect_aws
s3fs
$ pip-compile <http://requirements.in|requirements.in>
Could not find a version that matches botocore<1.27.60,<1.28.0,>=1.27.53,>=1.27.59,>=1.27.89 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
Tried: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0 ... [lists all versions here]
1.27.87, 1.27.88, 1.27.88, 1.27.89, 1.27.89
Skipped pre-versions: 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0rc1
There are incompatible versions in the resolved dependencies:
botocore<1.28.0,>=1.27.89 (from boto3==1.24.89->prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
botocore>=1.27.53 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2))
botocore<1.27.60,>=1.27.59 (from aiobotocore==2.4.0->s3fs==2022.8.2->-r <http://requirements.in|requirements.in> (line 3))
I have found this issue: https://github.com/fsspec/s3fs/issues/615#issuecomment-1094791081 but not a real solution to fix it.
How can I use prefect_aws
and also deploy flows to S3? Does anyone else have the same problem and found a solution?Anna Geller
10/11/2022, 11:33 PMNico Neumann
10/11/2022, 11:49 PMprefect==2.5.0
prefect_aws==0.1.4
s3fs==2022.8.2
boto3==1.24.59
botocore==1.27.59
Anna Geller
10/12/2022, 12:09 AMNico Neumann
10/12/2022, 2:02 AMboto3
is not async compatible. So I see two non-optimal ways:
1. Wrap boto3
sync function to make them async (probably reduces performance because of the overhead)
2. Use a third party package like aioboto3
which is an asynchronous version of boto3
(but not sure how stable it is)prefect_aws
already uses such functionality as in 1. and running the sync function in another thread using from prefect.utilities.asyncutils import run_sync_in_worker_thread
Anna Geller
10/12/2022, 11:19 AMalex
10/12/2022, 11:44 AMS3Bucket
block that currently exists in prefect-aws
, but in its current form it doesn’t work with deployments. There is a PR here to make that block compatible with deployments: https://github.com/PrefectHQ/prefect-aws/pull/82. Unfortunately, it’s gone a little stale, but I think I should be able to revive it and get it merged this week!Nico Neumann
10/12/2022, 12:29 PM