Hi everyone, we are having some issues with prefect on AWS. We use an S3 block to store (so using the s3fs), the we use boto3 in some of our tasks. The problem is, s3fs depends on an older botocore then our boto3 version, and we don't want to downgrade it. Does anyone have any recommendation on how to solve this? Will it be a problem in prefect_aws module?

It looks like this is happening because

s3fs

depends on

aiobotocore

, which pins itself to a very specific version of

botocore

. They do this so they can validate and ensure compatibility with whatever version of

botocore

they support, but the downside is that

aiobotocore

doesn't always officially support the latest

botocore

. You mentioned

prefect_aws

, and that might be a good solution here. It contains an

S3Bucket

class that accesses S3 without using

s3fs

. You'd need to run the following commands to make the block and its credentials block show up in your Prefect UI:

prefect block register -m prefect_aws.credentials
prefect block register -m prefect_aws.s3

Then you would use an

S3Bucket

block in your deployments instead of the

S3

block you are using now. Alternatively, you could try spinning up a Dask cluster with worker nodes that have a newer version of

botocore

installed and then use a

DaskTaskRunner

from the

prefect-dask

collection to connect to the cluster and run your tasks. This should work as long as your tasks don't use your

S3

block. Of the two options, I recommend the

prefect_aws

approach.

Thank you @Ryan Peden, I switched to using S3Bucket from prefect_aws, only I create the block using python, instead of the CLI, still works this way. I want to ask, what is the point of having s3fs then? We used it because it was recommended in the documentation.