Send http requests in Ruby and stay logged in? - ruby

Question

I am trying to crawl websites with ruby. The way I am implementing it is that I send request for a page, and get all the links(href tags) in the page, and then generate another GET request. The problem is that I want to stay logged in during the whole process. I wrote some code as followed.
def start_crawling
uri = URI(#host + "/login")
#visited.push #host + "/login"
req = Net::HTTP::Post.new(uri)
req.set_form_data({
'email' => 'test',
'password' => 'test'
})
Net::HTTP.start(uri.hostname, uri.port) do |http|
res = http.request req
puts uri
puts res.code
content = res.body
puts content
puts res.response
cookie = res.response['Set-Cookie'] # this gives nothing
puts cookie
puts res["Set-Cookie"] # prints nothing here
hrefs = get_href_tag_array_from_html(content)
send_get_requests(hrefs, cookie)
end
end
def send_get_requests(hrefs, cookie)
while not hrefs.empty?
href = hrefs.pop
href = #host + href if not href.start_with?"http"
next if #visited.include?(href)
puts "href: " + href
uri = URI(href)
Net::HTTP.start(uri.host, uri.port) do |http|
req = Net::HTTP::Get.new uri
res = http.request req
puts "------------------href: #{href}---------------------------"
puts res.code
puts res.message
puts res.class.name
puts "Cookie: "
puts res['Set-Cookie'] # this works and prints cookies
puts res.body
puts "------------------end of: #{href}---------------------------"
new_hrefs = get_href_tag_array_from_html(res.body)
hrefs += new_hrefs
end
#visited.push href
end
end
I want to start crawling from the login page. Ideally I want to stay logged in during the whole crawling procedure. I don't know much about session/cookie stuff, but I guess if I can get the cookie from the previous response and send it with the next request, I should be able to stay logged in. However, I cannot get any cookie from the login response. The response body is a 302 redirection, as I would expect. I checked it on the browser and the 302 response header does contain a cookie field, and this cookie is used for the next get request to redirect to the home page, but I cannot get the cookie field.
When I send GET request and get the response, I can get the cookie field out of it, but when I send POST request for the login page, I cannot get any cookie. Is there any fundamental difference between GET and POST requests in such cases?
Any idea how can I get this cookie field? Or do I have some basic misunderstanding in solving this crawling problem? Thanks.