Nico Neumann10/11/2022, 11:29 PM
to upload/list/download files to s3 and also for some shared AWS Secrets. For some functionality I rely on
. Prefect 2.5.0 is running on EKS and the flows are deployed to S3 which requires
https://docs.prefect.io/concepts/filesystems/ My problem is that
To use it in a deployment: prefect deployment […] -sb s3/dev You need to install s3fs to use this block.
have dependency conflicts. I am using pip-tools to set my requirements and get the following error:
# simplified <http://requirements.in|requirements.in> (removed the package versions to might easier find matches) prefect prefect_aws s3fs
I have found this issue: https://github.com/fsspec/s3fs/issues/615#issuecomment-1094791081 but not a real solution to fix it. How can I use
$ pip-compile <http://requirements.in|requirements.in> Could not find a version that matches botocore<1.27.60,<1.28.0,>=1.27.53,>=1.27.59,>=1.27.89 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2)) Tried: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0 ... [lists all versions here] 1.27.87, 1.27.88, 1.27.88, 1.27.89, 1.27.89 Skipped pre-versions: 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0rc1 There are incompatible versions in the resolved dependencies: botocore<1.28.0,>=1.27.89 (from boto3==1.24.89->prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2)) botocore>=1.27.53 (from prefect_aws==0.1.4->-r <http://requirements.in|requirements.in> (line 2)) botocore<1.27.60,>=1.27.59 (from aiobotocore==2.4.0->s3fs==2022.8.2->-r <http://requirements.in|requirements.in> (line 3))
and also deploy flows to S3? Does anyone else have the same problem and found a solution?
Nico Neumann10/11/2022, 11:49 PM
prefect==2.5.0 prefect_aws==0.1.4 s3fs==2022.8.2 boto3==1.24.59 botocore==1.27.59
Nico Neumann10/12/2022, 2:02 AM
is not async compatible. So I see two non-optimal ways: 1. Wrap
sync function to make them async (probably reduces performance because of the overhead) 2. Use a third party package like
which is an asynchronous version of
(but not sure how stable it is)
already uses such functionality as in 1. and running the sync function in another thread using
from prefect.utilities.asyncutils import run_sync_in_worker_thread
alex10/12/2022, 11:44 AM
block that currently exists in
, but in its current form it doesn’t work with deployments. There is a PR here to make that block compatible with deployments: https://github.com/PrefectHQ/prefect-aws/pull/82. Unfortunately, it’s gone a little stale, but I think I should be able to revive it and get it merged this week!
Nico Neumann10/12/2022, 12:29 PM