https://prefect.io logo
Title
p

pbutler

11/01/2022, 8:57 PM
What's the best way to upload a large file using the S3 block in prefect? It looks like the two options are S3.write_path which expects bytes and means loading the whole thing in memory and put_directory which looks like it uses write_path to do its work. I think maybe I can mmap the file to get adressable bytes object but maybe I am thinking about this all wrong?
1
n

Nate

11/01/2022, 9:26 PM
Hi @pbutler Multipart upload is something we'd like to support in the
s3_upload
task soon (or write_path as you mentioned), but for now you could use the aws credentials block to generate a boto3 session -> client
s3_client = aws_credentials.get_boto3_session().client("s3")
and then run the multipart upload like this
Here’s an issue to track this
p

pbutler

11/02/2022, 12:53 PM
Thanks for the advice! That should work
👍 1