Marc Lipoff
12/09/2020, 2:07 PMs3_client = boto3.client('s3')
with requests.request("GET", url, stream=True) as r:
r.raise_for_status()
for chunk in r:
s3_client.put_object(Body=chunk,
Bucket='xxx',
Key='xxx')
Sébastien
12/09/2020, 2:18 PMurl<>S3
for each chunk).
The requests.request
is a single streaming request. That single stream can't be turned into pieces without limiting yourself to I/O throughput on the initial worker (AFAIK).
If you want to keep it simpler yet still speed it up, you should be able to use requests-futures
to make it async and run on multiple OS threads (which, in turn, would not guarantee chunk ordering so make sure you reconstruct it properly once the whole object is streamed).Marc Lipoff
12/09/2020, 2:27 PMSébastien
12/09/2020, 2:28 PM