I'm working on a Ruby TCP client/server app using GServer and TCPSocket. I've run into a problem that I don't understand. My TCPSocket client successfully connects to my GServer, but I can only send data using puts. Calls to TCPSocket.send or TCPSocket.write do nothing. Is there some magic that I'm missing?
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.puts( 'Z' ) # -> GServer receives "Z\n"
But if I use write or send...
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.write( 'Z' ) # -> nothing is received
tcp_client.send( 'Z' ) # -> nothing is received
Thanks for the help
Additional information:
The behavior is the same on Linux & Windows.
Flushing the socket after write doesn't change the behavior.
Are you sure the problem isn't on the server side? Are you using some method to read that expects a string or something ending in "\n"?
With buffering taken care of in previous posts to address the question of whether the data is being sent consider capturing the data on the line using something like wireshark. If the data you are sending is seen on the line then the server isn't receiving it.
Otherwise, if the data isn't going onto the line, TCP may hold onto data to avoid sending a single segment with only a few bytes in it (see Nagle's Algorithm). Depending on your OS or TCP vendor you may have different behaviour, but most TCP stacks support the TCP_NODELAY option which may help get the data out in a more timely manner.
tcp_client.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)
This can help debugging, but typically shouldn't be left in production code if throughput is higher priority than responsiveness.
Try explicitly flushing:
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.write( 'Z' )
tcp_client.send( 'Z' )
tcp_client.flush
This way, the output is buffered at most only until the point at which you decide it should be sent out.
Hi there the reason should be related to the fact puts add automatic LF and CRL to your string.
If you want to use send or write you need to add them yourself so for instance that would be:
tcp_client.send( "Z\r\n",0)
I had the same problem, so after reading the socket, I had to explicitly delete the last instance of "\n" by doing the following:
client_socket.gets.gsub(/\n$/, '')
Related
Trying to see if I can get a response from ctrader server.
Getting no response and seems to hang at "s.recv(1024)". So not sure what could be going wrong here. I have limited experience with sockets and network coding.
I have checked my login credentials and all seems ok.
Note: I am aware of many FIX engines that are available for this purpose but wanted to
try this on my own.
ctrader FIX guides
require 'socket'
hostname = "h51.p.ctrader.com"
port = 5201
#constructing a fix message to see what ctrader server returns
#8=FIX.4.4|9=123|35=A|49=demo.ctrader.*******|56=cServer|57=QUOTE|50=QUOTE|34=1|52=20220127-16:49:31|98=0|108=30|553=********|554=*******|10=155|
fix_message = "8=FIX.4.4|9=#{bodylengthsum}|" + bodylength + "10=#{checksumcalc}|"
s = TCPSocket.new(hostname, port)
s.send(fix_message.force_encoding("ASCII"),0)
print fix_message
puts s.recv(1024)
s.close
Sockets are by default blocking on read. When you call recv that call will block if no data is available.
The fact that your recv call is not returning anything, would be an indication that the server did not send you any reply at all; the call is blocking waiting for incoming data.
If you would use read instead, then the call will block until all the requested data has been received.
So calling recv(1024) will block until 1 or more bytes are available.
Calling read(1024) will block until all 1024 bytes have been received.
Note that you cannot rely on a single recv call to return a full message, even if the sender sent you everything you need. Multiple recv calls may be required to construct the full message.
Also note that the FIX protocol gives the msg length at the start of each message. So after you get enough data to see the msg length, you could call read to ensure you get the rest.
If you do not want your recv or read calls to block when no data (or incomplete data) is available, then you need to use non-blocking IO instead for your reads. This is complex topic, which you need to research, but often used when you don't want to block and need to read arbitary length messages. You can look here for some tips.
Another option would be to use something like EventMachine instead, which makes it easier to deal with sockets in situations like this, without having to worry about blocking in your code.
I'm putting together a TCPServer in Ruby 3.0.2 and I'm finding that I can't seem to read the entire packet without blocking (until the socket is closed).
Edit: There was some confusion on what I was trying to do - my bad - so just to help clarify: I wanted to read everything that had been sent over the TCP connection so far. (end edit)
My first try was:
#!/snap/bin/ruby
require 'socket'
server = TCPServer.new('localhost', 4200)
loop {
Thread.start(server.accept) do |connection|
puts connection.gets # The important line
end
}
But that hangs until the client closes the connection. Okay, so I take a look at connection.methods, and the ruby docs and try a bunch of options that seem promising. Basically, there is two types of read methods: blocking and nonblocking.
The blocking methods that I tried are .read, .gets, .readlines, .readline, .recv, and .recvmsg. Now .read, .readlines, and .gets all hang (until the socket is closed) - so that's not helpful. The other ones (eg. .readline, the recv methods) don't read the entire message. Now, I could read each line until I see an empty line and parse the HTTP header from there. But there's got to be a better way; I don't want to have to worry about getting a corrupted message and hanging because I didn't read an empty line at the end of the header.
So I went looking at the non-blocking options. Specifically .recv_nonblock and .recvmsg_nonblock. Both of these throw errors (Resource temporarily unavailable - recvfrom(2) would block and Resource temporarily unavailable - recvmsg(2) respectively).
Any ideas on what could be going on? I think it has something to with me using Ruby 3, because trying out the code on Ruby 2.5, client.gets returns a line (doesn't hang), although .readlines does hang - so not sure what's going on.
Ideally, I could just call something along the lines of client.get_message and I would get the entire message that has been sent, but I'd also be okay with working at the TCP level and getting the packet size, reading that size, and reconstructing the message from there.
TCP just transmits the bytes that you write to the socket, and guarantees that the are received in the order they were sent. If you have the concept of a 'message' then you'll need to add that into your server and client.
.gets specifically will block until it reads a new 'line', or whatever you define as the separator for the string - see the docs IO#gets. This means that until your server receives that byte from the client, it will block.
In your client have a look at how you're writing your data - if you're using ruby then puts would work, as it will terminate the string with a new line. If you're using write then it will only write the string without a new line
Ie.
# client.rb
c = TCPSocket.new 'localhost', 5000
c.puts "foo"
c.write "bar"
c.write "baz\n"
# server.rb
s = TCPServer.new 5000
loop do
client = s.accept
puts client.gets
puts client.gets
end
will output
foo
barbaz
Thanks to everyone who commented/answered, but I found the solution that I think was intended by the creators of the Socket class!
The recv_nonblock method takes some optional arguments - one of which is a buffer that the Socket will store what it has read to. So a call like client.recv_nonblock(1000, 0, buffer) stores up to 1000 characters from the Socket into buffer and then exits instead of blocking.
Just to make life easy, I put together a monkey patch to the TCPSocket class:
class TCPSocket
def eat_buffer
contents = ''
buffer = ''
begin
loop {
recv_nonblock(256, 0, buffer)
contents += buffer
}
rescue IO::EAGAINWaitReadable
contents
end
end
end
The point that Steffen makes in the comments is well taken - TCP isn't designed to be used this way. This is a hacky (in the bad sense) method, and should be avoided.
I'm investigating man-in-the-middle attacks and trying to pipe raw HTTPS data (that is, before decryption) to and from a pair of sockets. For now, I just want to listen to the encrypted traffic, so I want any data going out to go from my web browser, through my script, and out to the intended recipient, and any data coming in to do the reverse. Ideally I'd just like to connect the incoming and outgoing sockets together and have them transfer data between each other automatically, but I haven't seen a way to do it in Ruby so I have been using the following, which I took from How can I create a two-way SSL socket in Ruby .
Here is my code:
def socketLoop(incoming, outgoing)
loop do
puts "selecting"
ready = IO.select([outgoing, incoming])
if ready[0].include?(incoming)
data_to_send = incoming.read_nonblock(32768)
outgoing.write(data_to_send)
puts "sent out"
puts data_to_send
end
if ready[0].include?(outgoing)
data_received = outgoing.read_nonblock(32768)
incoming.write(data_received)
puts "read in"
puts data_received
break if outgoing.nil? || outgoing.closed? || outgoing.eof?
end
end
end
server = TCPServer.open(LISTENING_PORT)
loop {
Thread.start(server.accept){ |incoming|
outgoing = TCPSocket.new(TARGET_IP, TARGET_PORT)
socketLoop(incoming, outgoing)
outgoing.close # Disconnect from target
incoming.close # Disconnect from the client
}
}
It works beautifully for HTTP but for HTTPS, my browser keeps spinning, and the output seems to indicate that at least part of a certificate has been sent over, but not much more. I presume I was being naïve to think that it would work for SSL, but as far as I know it uses TCP as the transport layer so I'm not sure why it doesn't work. Is it possible to get the raw data in this way? Is it an issue with my Ruby or have I made some wrong assumptions? I'd prefer not to use a system-wide packet sniffer if possible. If it would not be easy in Ruby, I'd be very grateful for any pointers in another language too.
Thanks a lot for your help!
EDIT: It seems that I can do this easily with netcat -
sudo nc -l 443 0<backpipe | nc $TARGET_IP 443 >backpipe
so I am rather embarassed that I didn't think of something so simple in the first place, however I would still be interested to see what I was not doing right in Ruby.
According to the spec for websockets protocol 13 (RFC 6455), the payload length for any given frame can be 0.
frame-payload-data ; n*8 bits in
; length, where
; n >= 0
I am building a websocket client to this spec, but when I sent echo.websocket.org a frame with an empty payload, I get nothing back. I experience the same using their GUI:
This is troublesome for me, since the way I'm building my client somewhat requires me to send empty frames when I FIN a multi-frame message.
Is this merely a bug in the Echo Test server? Do a substantial number of server implementations drop frames with empty payloads?
And if this is a bug in Echo Test, does anyone know how I might get in touch with them? The KAAZING site only has tech support contact info for their own products.
If you send a data-frame with no payload, there is nothing to echo back. This behaviour is fully correct. However, it might be standard-conform, too, to send back a dataframe with 0 payload, too. The main question is, if the application layer is informed at all, when a dataframe with no payload is received. This is probably not the case in most implementations.
With TCP this is similar: A TCP-Keepalive is a datagram with 0 payload. It is ack'd by the remote TCP-stack, but the application layer is not informed about it (i.e. select() does not return or a read()-syscall remains blocking), which is the expected behaviour.
An application-layer protocol should not rely on the datagrams to structure the data, but should merely expect a stream of bytes without taking regard on how these are transported.
I just tried the echo test on websocket.org with empty payloads and it seems to work fine using Chrome, Safari and Firefox (latest versions of each). Which browser are you using?
Btw, that demo program doesn't abide by any "echo protocol" (afaik), so there's no formal specification that dictates what to do on empty data in a WebSocket set of frames.
If you need help using WebSocket, there are Kaazing forums: http://developer.kaazing.com/forums.
I'm creating a ruby server which is connecting to a TCP client. My server is using a TCPServer and I'm attempting to use TCPServer::recv(), but it doesn't wait for data, so just continues in a tight loop until data is received.
What is the most efficient way to process intermittant data? I'm unable to change the data being sent in since I'm attempting to emulate another server. Which read like statement from TCPServer/TCPSocket would wait for data being sent?
require "socket"
dts = TCPServer.new('localhost', 20000)
s = dts.accept
print(s, " is accepted\n")
loopCount = 0;
loop do
Thread.start(s) do
loopCount = loopCount + 1
lineRcvd = s.recv(1024)
if ( !lineRcvd.empty? )
puts("#{loopCount} Received: #{lineRcvd}")
s.write(Time.now)
end
end
end
s.close
print(s, " is gone\n")
Thanks for your time.
are you sure recv isn't returning "" -- meaning the socket is closed?
If not then perhaps your sockets are set to non blocking somehow?
EventMachine is indeed far faster than using threads for socket programming :)
GL.
-r
Based on the questions you've been asking, I think you should try a framework like EventMachine and write a server that implements what you want instead of trying to fuss around with writing a server wrapper.
That being said, the most efficient way to read from a socket is to use a proper select call and poll all the open connections. While this is a fairly academic thing for anyone who's developed client-server applications before, it is a nuisance because there are a lot of things you can easily get wrong. For example. handling multiple connections can lead to all kinds of troublesome situations if you're not especially careful to avoid blocking calls.
The EventMachine framework makes it easy to develop query/response-type servers because you can always start with a template and work from there, for example, the built-in EventMachine::Protocols::LineAndTextProtocol one works as a great basis for most.