I'm trying to create a custom game server protocol in Ruby but i'm failing to understand how i can/should do a couple things:
Q->1#
Server sends an array to client using TCPServer/TCPSocket. but i can't use JSON as the transfer needs to be binary.. How can i convert a ruby array into binary which can then be transformed back into an array in client side?
This is my server:
#!/usr/bin/env ruby
require 'socket'
server = TCPServer.new 5020
loop do
Thread.start(server.accept) { |c|
print "new Client connected"
a = ["Hello, there"]
payload_with_header = CustomProtocolModule.constructMethod(a) # what method do i use to convert a (array -> binary representation? don't really understand .unpack, .pack, what template would i use if i did use String#Unpack? will String#Unpack work for what i need?
client.write payload_with_header
client.close
}
end
(Above is my first question)
Q->2#
Next since i know i need to use some sort of termination to check the end of message or a way to determine if the client/server has received the full message could i set the first two bytes of the payload to contain the size of the message? How would i do this server-side & client-side? The receiving party would have to loop TCPSocket.recv ? until it receives the first two bytes (the header), then read up the size the first two bytes contains? Something like this is what i had in mind:
| Payload Header (2 bytes) | array (contains cmd, message etc) |
If someone could help guide me in the right direction and/or provide pseudo code that can help. It would be greatly appreciated. Specifically i'm interested in how to loop for header, then read the payload back into an array and constructing the payload header.
Thanks!
payload_with_header = CustomProtocolModule.constructMethod(a)
You could serialize with
payload = Marshal.dump(a)
Then you can use
[payload.length].pack('l>') # to create a big endian signed long integer size for the packet
On the other side you can read an integer from the socket, then read the binary and deserialize with
array= Marshal.load(read_binary)
This will work for almost every data structure. Proc instances are an exception.
Ruby also supports remote objects and procedure calls so you can have interprocess communication, even between different ruby implementations (JRuby and MRI for instance)
Related
I've been tasked with creating a custom SNMP agent, and one of the things I need to do is manually encrypt the response I'm sending back.
To do so, I'd ideally like to send back a custom set of varBinds based on the get command, like so:
response = [(varBinds[0][0], v2c.OctetString(b'127.0.0.4')),
(varBinds[0][0], v2c.OctetString((sal+signature+iv+encryptor.tag+cipher_text)))]
Where I'd send the salt, signature, IV, tag and ciphered text (all in bytes) and could manage to
decrypt on the other side (a symmetric key is used), but I can't seem to obtain the same result when on the other side I use the following:
msg = bytes(vars[1][1])
How can I get back exactly the bytes I sent in the OctetString from the agent? Something does misformat in the way I'm decoding, because the bytes I get back are not the same I sent.
Edit: the bytestring does work flawlessly, the bytes on the second variable don't.
Thank you very much for any help :)
I don't know how can I write a smart contract in Solana that after executing the logic, returns an array of integers, strings, ... to the client, and how can I fetch it using Web3?
There's a syscall available to on-chain programs called set_return_data, which puts data into a buffer that can be read by the higher-level programs using get_return_data. This all mediated through opaque byte buffers, so you'll need to know how to decode the response.
If you want to fetch the data from the client side, you can simulate the transaction and read the data back from the return_data field in the response: https://edge.docs.solana.com/developing/clients/jsonrpc-api#results-50
The RPC support in simulated transactions is very new in version 1.11, but the return data is available in earlier versions.
Source code for set_return_data at https://github.com/solana-labs/solana/blob/658752cda710cb358d7ccbbc2cee06bf8009c2d4/sdk/program/src/program.rs#L102
Source code for get_return_data at https://github.com/solana-labs/solana/blob/658752cda710cb358d7ccbbc2cee06bf8009c2d4/sdk/program/src/program.rs#L117
So, programs do not return data (other than success or failure).
However; most programs write data to a program owned account's data field and this could be read from client apps (Rust, Python, TS/JS, etc.).
If using the Solana web3 library, you can call getAccountInfo on the Connection object. This will return the byte array of the account. You will then need to deserialize that data. You have to know how the program serializes the data to reverse it successfully.
Check the Solana Cookbook for overview using borsh https://solanacookbook.com/guides/serialization.html#how-to-deserialize-account-data-on-the-client
I'm putting together a TCPServer in Ruby 3.0.2 and I'm finding that I can't seem to read the entire packet without blocking (until the socket is closed).
Edit: There was some confusion on what I was trying to do - my bad - so just to help clarify: I wanted to read everything that had been sent over the TCP connection so far. (end edit)
My first try was:
#!/snap/bin/ruby
require 'socket'
server = TCPServer.new('localhost', 4200)
loop {
Thread.start(server.accept) do |connection|
puts connection.gets # The important line
end
}
But that hangs until the client closes the connection. Okay, so I take a look at connection.methods, and the ruby docs and try a bunch of options that seem promising. Basically, there is two types of read methods: blocking and nonblocking.
The blocking methods that I tried are .read, .gets, .readlines, .readline, .recv, and .recvmsg. Now .read, .readlines, and .gets all hang (until the socket is closed) - so that's not helpful. The other ones (eg. .readline, the recv methods) don't read the entire message. Now, I could read each line until I see an empty line and parse the HTTP header from there. But there's got to be a better way; I don't want to have to worry about getting a corrupted message and hanging because I didn't read an empty line at the end of the header.
So I went looking at the non-blocking options. Specifically .recv_nonblock and .recvmsg_nonblock. Both of these throw errors (Resource temporarily unavailable - recvfrom(2) would block and Resource temporarily unavailable - recvmsg(2) respectively).
Any ideas on what could be going on? I think it has something to with me using Ruby 3, because trying out the code on Ruby 2.5, client.gets returns a line (doesn't hang), although .readlines does hang - so not sure what's going on.
Ideally, I could just call something along the lines of client.get_message and I would get the entire message that has been sent, but I'd also be okay with working at the TCP level and getting the packet size, reading that size, and reconstructing the message from there.
TCP just transmits the bytes that you write to the socket, and guarantees that the are received in the order they were sent. If you have the concept of a 'message' then you'll need to add that into your server and client.
.gets specifically will block until it reads a new 'line', or whatever you define as the separator for the string - see the docs IO#gets. This means that until your server receives that byte from the client, it will block.
In your client have a look at how you're writing your data - if you're using ruby then puts would work, as it will terminate the string with a new line. If you're using write then it will only write the string without a new line
Ie.
# client.rb
c = TCPSocket.new 'localhost', 5000
c.puts "foo"
c.write "bar"
c.write "baz\n"
# server.rb
s = TCPServer.new 5000
loop do
client = s.accept
puts client.gets
puts client.gets
end
will output
foo
barbaz
Thanks to everyone who commented/answered, but I found the solution that I think was intended by the creators of the Socket class!
The recv_nonblock method takes some optional arguments - one of which is a buffer that the Socket will store what it has read to. So a call like client.recv_nonblock(1000, 0, buffer) stores up to 1000 characters from the Socket into buffer and then exits instead of blocking.
Just to make life easy, I put together a monkey patch to the TCPSocket class:
class TCPSocket
def eat_buffer
contents = ''
buffer = ''
begin
loop {
recv_nonblock(256, 0, buffer)
contents += buffer
}
rescue IO::EAGAINWaitReadable
contents
end
end
end
The point that Steffen makes in the comments is well taken - TCP isn't designed to be used this way. This is a hacky (in the bad sense) method, and should be avoided.
I am implementing a file downloader by using the down gem.
I need to add a progress bar to my program for fancy outputs. I found a gem called ruby-progressbar. However, I couldn't integrate it to my code base even though I followed the instructions documented on the official site. Here's what I have done so far:
First, I thought of using progress_proc. It was a bad idea because progress_proc returns chunked partial of the data.
Second, I streamed the data and built an idea on calculating chunked data. It worked well actually, but it smells bad to me.
Plus, here is the small part of my code base. I hope it helps you understand the concept.
progressbar = ProgressBar.create(title: 'File 1')
Down.download(url, progress_proc: ->(progress) { progressbar.progress = progress }) # It doesn't work
progressbar = ProgressBar.create(title: 'File 1')
file = Down.open(url, progress_proc: ->(progress) { progressbar.progress = progress })
chunked = 0
loop do
break if file.eof?
file.read(1024)
chunked += 1024
progressbar.progress = (chunked / file.size) * 100
end
# This worked well as I remember. It can be faulty because I wrote it down without testing.
In the HTTP protocol, there are two different ways on how a client can determine the full length of a response:
In the most common case, the entire response is sent by the server in one go. Here, the length of the response body in bytes is set in the Content-Length header of the response. Thus, if the response is not chunked, you can get the value of this header and read the response in one go as it is sent by the server.
The second option is for the server to send a chunked response. Here, the server sends chunks of the entire response, one after another. Each chunk is prefixed with the length of the chunk. However, the client has no way to know how many chunks there are in total, nor how large the total response may be. Often, this is even unknown to the server as the first chunks are already sent before the entire response is available to the server.
The down gem follows these two approaches by offering two interfaces:
In the first case (i.e. if the content length of the entire response is known), the gem will call the given content_length_proc once.
In the second case, as the entire length of the response is unknown before it was received in total, the down gem calls the progress_proc once for each chunk received. In this case, it is up to you to show something useful. In general, you can NOT show a progress bar as a percentage of completion here.
I'm working on a Ruby TCP client/server app using GServer and TCPSocket. I've run into a problem that I don't understand. My TCPSocket client successfully connects to my GServer, but I can only send data using puts. Calls to TCPSocket.send or TCPSocket.write do nothing. Is there some magic that I'm missing?
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.puts( 'Z' ) # -> GServer receives "Z\n"
But if I use write or send...
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.write( 'Z' ) # -> nothing is received
tcp_client.send( 'Z' ) # -> nothing is received
Thanks for the help
Additional information:
The behavior is the same on Linux & Windows.
Flushing the socket after write doesn't change the behavior.
Are you sure the problem isn't on the server side? Are you using some method to read that expects a string or something ending in "\n"?
With buffering taken care of in previous posts to address the question of whether the data is being sent consider capturing the data on the line using something like wireshark. If the data you are sending is seen on the line then the server isn't receiving it.
Otherwise, if the data isn't going onto the line, TCP may hold onto data to avoid sending a single segment with only a few bytes in it (see Nagle's Algorithm). Depending on your OS or TCP vendor you may have different behaviour, but most TCP stacks support the TCP_NODELAY option which may help get the data out in a more timely manner.
tcp_client.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)
This can help debugging, but typically shouldn't be left in production code if throughput is higher priority than responsiveness.
Try explicitly flushing:
tcp_client = TCPSocket.new( ipaddr, port )
tcp_client.write( 'Z' )
tcp_client.send( 'Z' )
tcp_client.flush
This way, the output is buffered at most only until the point at which you decide it should be sent out.
Hi there the reason should be related to the fact puts add automatic LF and CRL to your string.
If you want to use send or write you need to add them yourself so for instance that would be:
tcp_client.send( "Z\r\n",0)
I had the same problem, so after reading the socket, I had to explicitly delete the last instance of "\n" by doing the following:
client_socket.gets.gsub(/\n$/, '')