I'm investigating man-in-the-middle attacks and trying to pipe raw HTTPS data (that is, before decryption) to and from a pair of sockets. For now, I just want to listen to the encrypted traffic, so I want any data going out to go from my web browser, through my script, and out to the intended recipient, and any data coming in to do the reverse. Ideally I'd just like to connect the incoming and outgoing sockets together and have them transfer data between each other automatically, but I haven't seen a way to do it in Ruby so I have been using the following, which I took from How can I create a two-way SSL socket in Ruby .
Here is my code:
def socketLoop(incoming, outgoing)
loop do
puts "selecting"
ready = IO.select([outgoing, incoming])
if ready[0].include?(incoming)
data_to_send = incoming.read_nonblock(32768)
outgoing.write(data_to_send)
puts "sent out"
puts data_to_send
end
if ready[0].include?(outgoing)
data_received = outgoing.read_nonblock(32768)
incoming.write(data_received)
puts "read in"
puts data_received
break if outgoing.nil? || outgoing.closed? || outgoing.eof?
end
end
end
server = TCPServer.open(LISTENING_PORT)
loop {
Thread.start(server.accept){ |incoming|
outgoing = TCPSocket.new(TARGET_IP, TARGET_PORT)
socketLoop(incoming, outgoing)
outgoing.close # Disconnect from target
incoming.close # Disconnect from the client
}
}
It works beautifully for HTTP but for HTTPS, my browser keeps spinning, and the output seems to indicate that at least part of a certificate has been sent over, but not much more. I presume I was being naïve to think that it would work for SSL, but as far as I know it uses TCP as the transport layer so I'm not sure why it doesn't work. Is it possible to get the raw data in this way? Is it an issue with my Ruby or have I made some wrong assumptions? I'd prefer not to use a system-wide packet sniffer if possible. If it would not be easy in Ruby, I'd be very grateful for any pointers in another language too.
Thanks a lot for your help!
EDIT: It seems that I can do this easily with netcat -
sudo nc -l 443 0<backpipe | nc $TARGET_IP 443 >backpipe
so I am rather embarassed that I didn't think of something so simple in the first place, however I would still be interested to see what I was not doing right in Ruby.
Related
I am working on an assignment where I have to develop a web server in Ruby using the socket library. I was able to get a simple web server up and running as seen in this thread here .
I am currently working on getting and storing the body of an HTTP request into a variable in my web server. The problem I am running into is trying to define a while loop that gets the entire body of a HTTP request.
I am attempting to get the body of a HTTP request by using the gets method. I could not find any documentation on this method (I saw it being used here)
and was wondering if there were more documentation online.
In my first post here, someone suggested that I use the Content-Length header to determine the size of the body and how much data to read from the socket. I don't really understand how I would go about implementing this because I am unsure how the gets method functions.
Since this is for an assignment, I don't think posting code would be a good idea. I am looking for more information on the gets method and any tips to point me towards the right direction.
You shouldn't be using gets. gets tries to read complete lines (ie it reads up to a line separator), but there is no guarantee that an http request body ends with a line separator.
Instead you should be using read - this allows you to read an arbitrary amount of data (as you mentioned you can use the content length header to know how much to read)
Your ultimate problem isn't related to gets, or even really anything in your code. But before we get to that, let's answer this question & explore sockets a little bit.
If you follow the chain up, you find that Ruby's TCPSocket class inherits from its IO class. It's IO that provides gets. gets will read, line-by-line, until there's nothing more to read. Let's create a simple client that connects to a port, spits out 4 lines of poetry, and then quits:
# poetry_sender.rb
require 'socket'
poem = ["'God save thee, ancient Mariner!",
"From the fiends, that plague thee thus!—",
"Why look'st thou so?'—With my cross-bow",
"I shot the ALBATROSS."]
puts "Client establishing connection..."
s = TCPSocket.new 'localhost', 2000
puts "Client sending poetry..."
poem.each { |line| s.puts line } # Print each line out on the socket
s.close # Close our socket
puts "All done."
And a simple server, that displays what the client sends us:
# poetry_receiver.rb
require 'socket'
server = TCPServer.new 2000 # Server bind to port 2000
loop do
puts "Server now awaiting some poetry..."
socket = server.accept # Wait for a client to connect
while line = socket.gets
puts "A client sent us this beautiful line: #{line}"
end
puts "They had nothing more to say; let's disconnect them."
socket.close
end
If you run the server (poetry_receiver.rb) first, and then the client, you'll see some output like this:
Server now awaiting a connection...
A client sent us this beautiful line: 'God save thee, ancient Mariner!
A client sent us this beautiful line: From the fiends, that plague thee thus!—
A client sent us this beautiful line: Why look'st thou so?'—With my cross-bow
A client sent us this beautiful line: I shot the ALBATROSS.
They had nothing more to say; let's disconnect them.
Server now awaiting a connection...
The last two lines are the important ones; they indicate that socket.gets returned nil and we exited the while loop.
So, how can we modify our poetry_sender.rb so the server doesn't detect the end of the poem? You might think it's got something to do with blank lines, but if you set poem = [] or poem = ["", "", ""] then you'll find that it still gets disconnected OK. But what if we added a delay before closing the socket in poetry_sender.rb?
sleep 60
s.close # Close our socket
puts "All done."
Now you'll see a big delay in the server output. The TCP server doesn't break out of its while loop until the TCP client closes its socket.
Now we can turn to your broader problem: you're trying to implement a simple HTTP server, but your server is getting hung up in a while loop when you try to connect via your web browser. It's because your web browser is keeping that socket open; but it has to, otherwise it has no way to send you back a response. So, how do we know when a client has finished sending us a response? The HTTP 1.1 spec says:
A client sends an HTTP request to a server in the form of a request message... followed by header fields... an empty line to indicate the end of the header section, and finally a message body containing the payload body (if any).
Let's not worry about the message body; how could we write a while loop that terminates if it has no more impact, or if it receives a blank line? Here's one way, in a simple HTTP server that just sends back "Hello world" no matter what request it receives:
require 'socket'
server = TCPServer.new('localhost', 2345)
http_request = [] # We'll store the lines of our incoming request here.
loop do
socket = server.accept
while (line = socket.gets) && line.chomp != '' # While the client is connected, and hasn't sent us a blank line yet...
http_request << line
end
# Send response headers
socket.print "HTTP/1.1 200 OK\r\n" +
"Content-Type: text/plain\r\n" +
"Connection: close\r\n" +
"\r\n"
# Send response body
socket.print "Hello world!"
socket.close
end
Quite late to the party, but I'm currently implementing my own rack app server (for fun).
Here you can see how I do it: https://github.com/tak1n/reifier/blob/master/lib/reifier/request.rb
The first line of a HTTP request is always the request line, which is basically something like GET /test HTTP/1.1
After the request line until \r\n you get the headers.
After that you are able to read the body (if PUT / POST request) with just using the CONTENT_LENGTH you parsed from the headers.
How do I send and receive data multiple times using TCPSocket?
require 'socket'
TCPSocket.open("example.com", 80) {|s|
s.send "GET / HTTP/1.0\r\n\r\n", 0
puts s.read
s.send "GET / HTTP/1.0\r\n\r\n", 0
# Here is no data. PS: Without reconnect.
puts s.read
}
I searched but did not find the answer.
the short answer is that you cannot easily do that using a TCPSocket (it can probably be done but not worth your trouble).
In case of HTTP the server will close the connection after sending the response.
What you want to do is use something that does HTTP and also understands how to do keep-alive's (i.e. persistent) on the connection so that you can reuse the connection. For some background: https://en.wikipedia.org/wiki/HTTP_persistent_connection
Look at:
http://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison
slide 30. may be outdated but will give you a good starting point.
So, I'm trying to simulate some basic HTTP persistent connections using sockets and Ruby - for a college class.
The point is to build a server - able to handle multiple clients - that receives a file path and gives back the file content - just like an HTTP GET.
The current server implementation loops listening for clients, fires a new thread when there's an incoming connection and reads the file paths from this socket. It's very dumb, but it works fine when working with non-presistent connections - one request per connection.
But they should be persistent.
Which means the client shouldn't worry about closing the connection. In the non-persistent version the servers echoes the response and close the connection - goodbye client, farewell.
But being persistent means the server thread should loop and wait for more incoming requests until... well until there's no more requests. How does the server knows that? It doesn't! Some sort of timeout is needed. I tried to do that with Ruby's Timeout, but it didn't work.
Googling for some solutions - besides being thoroughly advised to avoid using Timeout module - I've seen a lot of posts about the IO.select method, that should handle this socket waiting issue way better than using threads and stuff (which really sounds cool, considering how Ruby threads (don't) work). I'm trying to understand here how IO.select works, but still wasn't able to make it work in the current scenario.
So I aske basically two things:
how can I efficiently work this timeout issue on the server-side, either using some thread based solution, low-level socket options or some IO.select magic?
how can the client side know that the server has closed its side of the connection?
Here's the current code for the server:
require 'date'
module Sockettp
class Server
def initialize(dir, port = Sockettp::DEFAULT_PORT)
#dir = dir
#port = port
end
def start
puts "Starting Sockettp server..."
puts "Serving #{#dir.yellow} on port #{#port.to_s.green}"
Socket.tcp_server_loop(#port) do |socket, client_addrinfo|
handle socket, client_addrinfo
end
end
private
def handle(socket, addrinfo)
Thread.new(socket) do |client|
log "New client connected"
begin
loop do
if client.eof?
puts "#{'-' * 100} end connection"
break
end
input = client.gets.chomp
body = content_for(input)
response = {}
if body
response.merge!({
status: 200,
body: body
})
else
response.merge!({
status: 404,
body: Sockettp::STATUSES[404]
})
end
log "#{addrinfo.ip_address} #{input} -- #{response[:status]} #{Sockettp::STATUSES[response[:status]]}".send(response[:status] == 200 ? :green : :red)
client.puts(response.to_json)
end
ensure
socket.close
end
end
end
def content_for(path)
path = File.join(#dir, path)
return File.read(path) if File.file?(path)
return Dir["#{path}/*"] if File.directory?(path)
end
def log(msg)
puts "#{Thread.current} -- #{DateTime.now.to_s} -- #{msg}"
end
end
end
Update
I was able to simulate the timeout behaviour using the IO.select method, but the implementation doesn't feel good when combining with a couple of threads for accepting new connections and another couple for handling requests. The concurrency makes the situation mad and unstable, and I'm probably not sticking with it unless I can figure out a better way of using this solution.
Update 2
Seems like Timeout is still the best way to handle this. I'm sticking with it till find a better option.
I still don't know how to deal with zombie client connections.
Solution
I endend up using IO.select (got inspired when looking at the webrick code). You cha check the final version here (lib/http/server/client_handler.rb)
You should implement something like heartbeat packets.Client side should send special packets to after few secs/mins to ensure that server doesn't time out the connection on the client end.You just avoid doing anything in this call.
I am writing a Ruby script which automatically crawls websites for data analysis, and now I have a requirement which is fairly complicated: I have to be able to simulate access from a variety of countries, about 20 different ones. The website will contain different information depending on the IP location, so the only way to get it done is to request it from a server which is actually in that country.
Since I don't want to buy servers in each of those 20 countries, I chose to give Tor a try - as many of you will know, by editing the torrc configuration file it is possible to specify the exit node and hence the country from which the actual request will originate.
When I do this manually, e.g. by editing the torrc file to use an Argentinian server, then disconnecting Tor using Vidalia, reconnecting Vidalia, and then rerunning the request, it works fine. However, I want to automate this process entirely, and do it as efficiently as possible. Tor is written in C, and I'd like to avoid taking apart its entire source code for this. Any idea of what's the easiest way to automate the whole process using only Ruby?
Also, if I'm missing something and there's a simpler alternative to this whole ordeal, let me know.
Thanks!
Please take a look at Tor control protocol. You can control circuits using telnet.
http://thesprawl.org/memdump/?entry=8
To switch to a new circuit wich switches to a new endpoint:
require 'net/telnet'
def switch_endpoint
localhost = Net::Telnet::new("Host" => "localhost", "Port" => "9051", "Timeout" => 10, "Prompt" => /250 OK\n/)
localhost.cmd('AUTHENTICATE ""') { |c| print c; throw "Cannot authenticate to Tor" if c != "250 OK\n" }
localhost.cmd('signal NEWNYM') { |c| print c; throw "Cannot switch Tor to new route" if c != "250 OK\n" }
localhost.close
end
Be aware of the delay to make a new circuit, may take couple seconds, so you'd better add a delay in the code, or check if your address has changed by calling some remote IP detection site.
I'm working on a Ruby TCP client/server app using GServer and TCPSocket. I've run into a problem that I don't understand. My TCPSocket client successfully connects to my GServer, but I can only send data using puts. Calls to TCPSocket.send or TCPSocket.write do nothing. Is there some magic that I'm missing?
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.puts( 'Z' ) # -> GServer receives "Z\n"
But if I use write or send...
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.write( 'Z' ) # -> nothing is received
tcp_client.send( 'Z' ) # -> nothing is received
Thanks for the help
Additional information:
The behavior is the same on Linux & Windows.
Flushing the socket after write doesn't change the behavior.
Are you sure the problem isn't on the server side? Are you using some method to read that expects a string or something ending in "\n"?
With buffering taken care of in previous posts to address the question of whether the data is being sent consider capturing the data on the line using something like wireshark. If the data you are sending is seen on the line then the server isn't receiving it.
Otherwise, if the data isn't going onto the line, TCP may hold onto data to avoid sending a single segment with only a few bytes in it (see Nagle's Algorithm). Depending on your OS or TCP vendor you may have different behaviour, but most TCP stacks support the TCP_NODELAY option which may help get the data out in a more timely manner.
tcp_client.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)
This can help debugging, but typically shouldn't be left in production code if throughput is higher priority than responsiveness.
Try explicitly flushing:
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.write( 'Z' )
tcp_client.send( 'Z' )
tcp_client.flush
This way, the output is buffered at most only until the point at which you decide it should be sent out.
Hi there the reason should be related to the fact puts add automatic LF and CRL to your string.
If you want to use send or write you need to add them yourself so for instance that would be:
tcp_client.send( "Z\r\n",0)
I had the same problem, so after reading the socket, I had to explicitly delete the last instance of "\n" by doing the following:
client_socket.gets.gsub(/\n$/, '')