Hi there, quick question here : what's the difference between `s3` from `prefect.filesystems` and `S...

Romain Vincent

06/13/2023, 1:25 PM

Hi there, quick question here : what's the difference between

s3

from

prefect.filesystems

and

S3Bucket

from

prefect_aws.s3

✅ 1

Jeff Hale

06/13/2023, 1:58 PM

S3Bucket has many more methods available for working with data.

Romain Vincent

06/13/2023, 2:01 PM

Alright thanks! I guess s3 is aimed at s3 storage deployments only whereas s3bucket is meant for data manipulation, right?

Jeff Hale

06/13/2023, 2:04 PM

s3Bucket can handle flow code storage, too.

Romain Vincent

06/13/2023, 2:06 PM

Alright, so

s3

is a bit redundant here. It seems I could get away with only

s3Bucket

blocks in most cases.

Jeff Hale

06/13/2023, 2:10 PM

Yep. That’s true. The S3 is quicker to get going with and fine if just using for flow code storage.

Romain Vincent

06/13/2023, 2:11 PM

Is it possible to have them work in combination with other

s3fs

libs, such as pandas? I did not test yet, but could I pass blocks as parameters for a pandas io function like

<http://df.to|df.to>_csv(s3_bucket_block, aws_credentials=aws_credentials_block)

. Is that a possible use case for blocks?

Romain Vincent

06/13/2023, 2:14 PM

Well, I'm going to try that actually 🙃 Thanks for your time!

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.