How to download a delayed file via HTTP in Ruby? - ruby

Question

I use the following Ruby function to download various files via HTTP:
def http_download(uri, filename)
bytes_total = nil
begin
uri.open(
read_timeout: 500,
content_length_proc: lambda { |content_length|
bytes_total = content_length
},
progress_proc: lambda { |bytes_transferred|
if bytes_total
print("\r#{bytes_transferred} of #{bytes_total} bytes")
else
print("\r#{bytes_transferred} bytes (total size unknown)")
end
}
) do |file|
open filename, 'w' do |io|
file.each_line do |line|
io.write line
end
end
end
rescue => e
puts e
end
end
I also want to download files (csv, kml, zip, geojson) from this website. However, there is some kind of delay set up. When I click the download link in the browser it takes a bit until the download window appears. I suppose the file needs to be processed on the server before it can be served.
How can I modify my script to take the delay into account?
I am running Ruby 2.2.2.

Answer 1

Here is the modification according to the post and comment:
require 'open-uri'
def http_download(uri, filename)
bytes_total = nil
index = 1
begin
open(
uri,
read_timeout: 500,
content_length_proc: lambda { |content_length|
bytes_total = content_length
},
progress_proc: lambda { |bytes_transferred|
if bytes_total
print("\r#{bytes_transferred} of #{bytes_total} bytes")
else
print("\r#{bytes_transferred} bytes (total size unknown)")
end
}
) do |io|
# if "application/json" == io.content_type
if io.is_a? StringIO
raise " --> Failed, server is processing. Retry the request ##{index}"
else # Tempfile
puts "\n--> Succeed, writing to #{filename}"
File.open(filename, 'w'){|wf| wf.write io.read}
end
end
rescue => e
puts e
return if e.is_a? OpenURI::HTTPError # Processing error
index += 1
return if index > 10
sleep index and retry
end
end