Marc Lipoff12/09/2020, 2:07 PM
s3_client = boto3.client('s3') with requests.request("GET", url, stream=True) as r: r.raise_for_status() for chunk in r: s3_client.put_object(Body=chunk, Bucket='xxx', Key='xxx')
Sébastien12/09/2020, 2:18 PM
for each chunk). The
is a single streaming request. That single stream can't be turned into pieces without limiting yourself to I/O throughput on the initial worker (AFAIK). If you want to keep it simpler yet still speed it up, you should be able to use
to make it async and run on multiple OS threads (which, in turn, would not guarantee chunk ordering so make sure you reconstruct it properly once the whole object is streamed).
Marc Lipoff12/09/2020, 2:27 PM
Sébastien12/09/2020, 2:28 PM