https://prefect.io logo
Title
t

Tibs

11/28/2022, 4:19 PM
Hi everyone, we are having some issues with prefect on AWS. We use an S3 block to store (so using the s3fs), the we use boto3 in some of our tasks. The problem is, s3fs depends on an older botocore then our boto3 version, and we don't want to downgrade it. Does anyone have any recommendation on how to solve this? Will it be a problem in prefect_aws module?
1
r

Ryan Peden

11/28/2022, 4:55 PM
It looks like this is happening because
s3fs
depends on
aiobotocore
, which pins itself to a very specific version of
botocore
. They do this so they can validate and ensure compatibility with whatever version of
botocore
they support, but the downside is that
aiobotocore
doesn't always officially support the latest
botocore
. You mentioned
prefect_aws
, and that might be a good solution here. It contains an
S3Bucket
class that accesses S3 without using
s3fs
. You'd need to run the following commands to make the block and its credentials block show up in your Prefect UI:
prefect block register -m prefect_aws.credentials
prefect block register -m prefect_aws.s3
Then you would use an
S3Bucket
block in your deployments instead of the
S3
block you are using now. Alternatively, you could try spinning up a Dask cluster with worker nodes that have a newer version of
botocore
installed and then use a
DaskTaskRunner
from the
prefect-dask
collection to connect to the cluster and run your tasks. This should work as long as your tasks don't use your
S3
block. Of the two options, I recommend the
prefect_aws
approach.
t

Tibs

11/29/2022, 6:42 AM
Thank you @Ryan Peden, I switched to using S3Bucket from prefect_aws, only I create the block using python, instead of the CLI, still works this way. I want to ask, what is the point of having s3fs then? We used it because it was recommended in the documentation.