Josh
01/22/2021, 10:25 PMS3Download
and GCSUpload
tasks.
My suspicion is that the flow is not releasing the memory of the files being transferred. Is there any way to ensure the file contents are being released from memory?nicholas
Spencer
01/22/2021, 10:35 PM@task
that boto3.client('s3').download_file()
to disk and then blob.upload_from_filename()
.
Highly suggest using the built-in tempfile
library to simplify management of disk space.
with tempfile.NamedTemporaryFile('w+') as f:
s3_client.download_fileobj(bucket, key, f)
f.flush()
f.seek(0)
blob.upload_from_filename(f.name)
nicholas