Reading from a TCPSocket in Ruby - ruby

Question

This code sends two HTTP requests to the www.example.com website:
require 'socket'
#host = 'www.example.com'
#port = 80
#path = "/"
# Build HTTP request
def request(close=false)
"GET #{#path} HTTP/1.1\r\nHost: #{#host}#{"\r\nConnection: close" if close}\r\n\r\n"
end
# Build socket
socket = TCPSocket.open(#host,#port) # Connect to server
# Send request twice via socket
2.times {socket.print request}
Here are various methods I have found for reading the response:
# Method 1: close_write and read
socket.close_write # Without this line, the next line hangs
response = socket.read
puts response.length
# Method 2: send another http request with 'Connection: close' header, then use 'read'
socket.print request(true) # Without this line, the next line hangs
response = socket.read
puts response.length
# Method 3: recv
# puts socket.eof? # Method fails when I add this line
r1, r2 = socket.recv(1000000), socket.recv(1000000)
puts r1.length, r2.length
# Method 4: IO.select and read_nonblock
puts socket.eof?
# IO.select([socket]) # The code still works without this IO.select...
r1 = socket.read_nonblock(9999999)
IO.select([socket]) # ...but not without this one
r2 = socket.read_nonblock(9999999)
puts r1.length, r2.length
puts socket.eof? # Hangs for ages before returning 'true'
Questions:
What exactly is 'socket.close_write' line doing in Method 1 and why is it necessary for the method to work?
In Method 2, is the 'Connection: close' header somehow achieving the same result as the 'socket.close_write' line in Method 1? If not, what is it doing, and why is it necessary for the rest of the method to work?
Why, in Method 3, does the addition of the commented line, 'puts socket.eof?', cause the rest of the code to hang?
In Method 3, how and why does the recv call stop at the end of an HTTP response (as opposed to picking up the next response as well)?
Why is it that the second IO.select in Method 4 is necessary, but the first isn't?
What does IO.select actually do?
Why does the last line, 'puts socket.eof?', in Method 4 hang for ages before returning true?
Is there a general way of checking how many responses a socket is currently expecting, and reading that number of responses from the socket, without closing the socket for writing?
Finally, if not possible in an answer here, is there a good resource somewhere where I can get some clarity on all of the above, and general clarity on reading from TCP sockets (or network sockets in general)?
Thanks.

Answer 1

socket.close_write is not necessary. I mean you can get what socket returns by socket.read, but you are gonna to wait some time. The reason why is because you are trying to read the whole stream in one with socket.read. It takes time. You can find out what socket returns by doing:
socket.each_line do |line|
puts line
end
BTW, Nagle's algorithm makes it slow too.
Whatclose_write does is let the client side half-close the socket.When the server side notice this, it will also close its side. Then you can finish reading very quickly.
Or you can use IO::select.As the documentation said:
It monitors given arrays of IO objects, waits one or more of IO objects ready for reading, are ready for writing, and have pending exceptions respectively, and returns an array that contains arrays of those IO objects. It will return nil if optional timeout value is given and no IO object is ready in timeout seconds.
ready = IO.select([socket], nil, nil, 10)
if ready
# do something
else
# raise timeout
end
Here we pass the first parameter, which is the object ready for reading and the last is the timeout you want set. This means if the reading is not ready in 10 seconds it will return nil then raise timeout error.