What's the best way to upload a large file using the S3 block in prefect? It looks like the two options are S3.write_path which expects bytes and means loading the whole thing in memory and put_directory which looks like it uses write_path to do its work. I think maybe I can mmap the file to get adressable bytes object but maybe I am thinking about this all wrong?

Hi @pbutler Multipart upload is something we'd like to support in the

s3_upload

task soon (or write_path as you mentioned), but for now you could use the aws credentials block to generate a boto3 session -> client

s3_client = aws_credentials.get_boto3_session().client("s3")

and then run the multipart upload like this

Thanks for the advice! That should work