How to make persistent HTTP requests using multiple threads in Ruby/Faraday? - ruby

Question

I'm using faraday with net-http-persistent adapter to make HTTP requests.
I want to optimise my requests by making them execute asynchronously but as I need a persistent connection I keep getting errors such as too many connections reset which I assume is due to the fact that I have multiple threads creating new connections.
I tried changing my adapter to typhoeus but as the connection is not persistent the final result of executing all request is not as expected.
My goal is to add items to a basket by making this HTTP requests. Without the persistent connection items are not added to the basket.
So, my question is:
Is it possible to make persistent HTTP requests reusing the connection between threads? If so, how can this be achieved?
Here is a piece of my code:
Create the connection:
Faraday.new do |c|
c.use :cookie_jar, jar: cookie_jar
c.options.open_timeout = 5
c.options.timeout = 10
c.request :url_encoded
c.response :logger, logger
c.adapter :net_http_persistent do |http| # yields Net::HTTP::Persistent
http.idle_timeout = 2
end
end
Creating threads and getting the result of each one of them
result = []
threads = []
total_items = items.size
items.each_slice(5) do |sliced_items|
# Create a thread for a batch of 5 items and store its result
threads << Thread.new do
Thread.current[:output] = browser.add_all_items(sliced_items)
end
end
# Wait for all threads to finish their work and store their output into result
threads.each do |t|
t.join
result << t[:output]
end
add_all_items and add_to_cart methods:
# Add a batch of items by the key passed (id, gtin, url)
def add_all_items(items_info, key)
results = []
items_info.each do |item|
begin
add_to_cart(item[key], item[:quantity])
item[:message] = nil
rescue => e
item[:message] = e.message
puts "---------- BACKTRACE -------------- \n #{e.backtrace}"
end
puts "\n--------- MESSAGE = #{item[:message]} --------- \n"
results << item
puts "-------- RESULTS #{results}"
end
results
end
def add_to_cart(url, count = 1)
response = connection.get(url) do |req|
req.headers["User-Agent"] = #user_agent
end
doc = Nokogiri::HTML(response.body)
stoken = doc.search('form.js-oxProductForm input[name=stoken]').attr('value').value
empty_json = '""'
product_id = get_item_id(url)
data = { #removed payload for security reasons }
# Using example.com for question purposes
response = connection.post('https://www.example.com/index.php?') do |req|
req.headers["Origin"] = "https://www.example.com"
req.headers["Content-Type"] = "application/x-www-form-urlencoded; charset=UTF-8"
req.headers["Accept"] = "application/json, text/javascript, */*; q=0.01"
req.headers["Referer"] = url
req.headers["Pragma"] = "no-cache"
req.headers["Accept-Language"] = "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
req.headers["User-Agent"] = #user_agent
req.headers["Cache-Control"] = "no-cache"
req.headers["Connection"] = "keep-alive"
req.headers["DNT"] ="1"
req.headers["Content-Length"] = data.size.to_s
req.headers["Accept"] = "*/*"
req.headers["X-Requested-With"] = "XMLHttpRequest"
req.headers["Connection"] = "keep-alive"
req.body = data
end
begin
json = JSON.parse(response.body)
raise "Could not add item: #{json['message']}" if json['success'] != 1 || json['item'] != product_id
rescue JSON::ParserError => e
puts "JSON Error"
end
end
def get_item_id(url)
response = connection.get(url) do |req|
req.headers["User-Agent"] = #user_agent
end
doc = Nokogiri::HTML(response.body)
doc.search('.js-oxProductForm input[name=aid]').attr('value').value
end
Thanks in advance.