https://prefect.io logo
Title
y

Yupei Chen

08/01/2022, 6:37 PM
Issue with using an S3 block:
from prefect.filesystems import S3

s3_block = S3.load("dev")
ModuleNotFoundError: No module named ‘s3fs’ ImportError: Install s3fs to access S3 RuntimeError: File system created with scheme ‘s3’ from base path ‘s3://vodori-prefect-orion’ could not be created. You are likely missing a Python module required to use the given storage protocol. An exception occurred. On Prefect 2.0.1
Resolved with pip install s3fs, however got this error during installation
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.24.19 requires botocore<1.28.0,>=1.27.19, but you have botocore 1.24.21 which is incompatible.
r

Rob Freedy

08/02/2022, 1:53 AM
Hey Yupei! Are you using a dependency manager? Would you be able to output the python packages you have installed on your system?
y

Yupei Chen

08/02/2022, 2:00 AM
All I use is pip. Packages attached.
r

redsquare

08/03/2022, 10:58 AM
Just had this same issue
r

Rob Freedy

08/03/2022, 1:02 PM
are you able to install the specific botocore package using
pip install botocore==1.27.19
? Was this dependency previously installed in your environment?
r

redsquare

08/03/2022, 1:13 PM
I did pip install s3fs
r

Rob Freedy

08/03/2022, 9:42 PM
@Marvin open "s3fs dependency issue in filesystems"
r

Rob Freedy

08/03/2022, 9:43 PM
I have opened a git issue here for this dependency issue, you track it's progress using the link above^
c

Chris Pickett

08/03/2022, 11:35 PM
Hi @Yupei Chen! Looking into this issue, prefect doesn’t currently manage boto or related requirements. It did during the 2.0 beta, so it’s possible that you picked up some incompatible requirements during that period. I’m able to get my local environment in the same state as yours and to get out of it I’d suggest uninstalling s3fs and gcsfs (assuming you got into this state because of the beta requirements) and upgrade the boto related packages:
pip uninstall s3fs gcsfs
pip install --upgrade aiobotocore boto3 aiobotocore
pip install s3fs
Alternatively, you could install an older version of s3fs that’s happy with your version of the boto packages which appears to be version `2022.5.0`:
pip install s3fs==2022.5.0
j

James Brady

08/12/2022, 11:42 AM
@Rob Freedy where should these
pip
commands be placed? I ran into the "No module named ‘s3fs’" error when running a flow in a kubernetes deployment – using the default docker image etc.
For right now, I'm going to use @Anna Geller’s suggestion here to create a custom image and pop it in ECR
👍 2