Inefficient memory usage when using object.get - ruby

Question

I thought that when downloading s3 object into a file it would write to it by chunks to avoid loading the whole file into memory.
But apparently, this is not the case, this is my code:
puts("Memory (before file loaded): #{((`ps -o rss= -p #{Process.pid}`.to_i) / 1024.0).round(2)} MB")
my_s3_object.get(response_target: file_path)
puts("Memory (after file loaded): #{((`ps -o rss= -p #{Process.pid}`.to_i) / 1024.0).round(2)} MB")
Output:
Memory (before file loaded): 191.08 MB
Memory (after file loaded): 259.41 MB
Where my_s3_object is 130MB zip archive. Ok, so it's not fully loaded into memory but almost half of it.
Is there a way to improve memory usage by passing some params to get method? Or how do I do it?

Answer 1

I think you are looking for ranged requests, which are a "general" HTTP "pattern" and supported by the AWS SDK.
The documentation provides the following examples, which should allow you to download parts of the object, write them, discard the bytes from memory and read the next bytes until the whole file is downloaded. In the end memory usage will depend on the range of bytes you download with every request.
Example: To retrieve a byte range of an object
# The following example retrieves an object for an S3 bucket.
# The request specifies the range header to retrieve a specific
# byte range.
resp = client.get_object({
bucket: "examplebucket",
key: "SampleFile.txt",
range: "bytes=0-9",
})
resp.to_h outputs the following:
{
accept_ranges: "bytes",
content_length: 10,
content_range: "bytes 0-9/43",
content_type: "text/plain",
etag: "\"0d94420ffd0bc68cd3d152506b97a9cc\"",
last_modified: Time.parse("Thu, 09 Oct 2014 22:57:28 GMT"),
metadata: {
},
version_id: "null",
}
Streaming data to a block
# WARNING: yielding data to a block disables retries of networking errors
# However truncation of the body will be retried automatically using a range request
File.open('/path/to/file', 'wb') do |file|
s3.get_object(bucket: 'bucket-name', key: 'object-key') do |chunk, headers|
# headers['content-length']
file.write(chunk)
end
end