It seems I don't understand something simple about Protocol Buffers, but this is very important question for me and for my real use-case.
While reading documentation about Protocol Buffers I don't understand how one know which message you should decode from the stream? All examples about some defined Message, but if you have defined several completelly different messages and you want to send those messages between 2 processes -- how do you know which message you have just received?
Or maybe Protocol Buffers do not try to address this problem and leave this question for another abstraction level?
Or maybe I should pack the message into structure like that:
message wrapper {
required string message_name = 1;
string packed_message = 2;
}
And then I should decode message in 2 stages: get the message_name at first, and then decode real packed message at second stage, should not I?
Look at self describing messages section
Related
Trying to see if I can get a response from ctrader server.
Getting no response and seems to hang at "s.recv(1024)". So not sure what could be going wrong here. I have limited experience with sockets and network coding.
I have checked my login credentials and all seems ok.
Note: I am aware of many FIX engines that are available for this purpose but wanted to
try this on my own.
ctrader FIX guides
require 'socket'
hostname = "h51.p.ctrader.com"
port = 5201
#constructing a fix message to see what ctrader server returns
#8=FIX.4.4|9=123|35=A|49=demo.ctrader.*******|56=cServer|57=QUOTE|50=QUOTE|34=1|52=20220127-16:49:31|98=0|108=30|553=********|554=*******|10=155|
fix_message = "8=FIX.4.4|9=#{bodylengthsum}|" + bodylength + "10=#{checksumcalc}|"
s = TCPSocket.new(hostname, port)
s.send(fix_message.force_encoding("ASCII"),0)
print fix_message
puts s.recv(1024)
s.close
Sockets are by default blocking on read. When you call recv that call will block if no data is available.
The fact that your recv call is not returning anything, would be an indication that the server did not send you any reply at all; the call is blocking waiting for incoming data.
If you would use read instead, then the call will block until all the requested data has been received.
So calling recv(1024) will block until 1 or more bytes are available.
Calling read(1024) will block until all 1024 bytes have been received.
Note that you cannot rely on a single recv call to return a full message, even if the sender sent you everything you need. Multiple recv calls may be required to construct the full message.
Also note that the FIX protocol gives the msg length at the start of each message. So after you get enough data to see the msg length, you could call read to ensure you get the rest.
If you do not want your recv or read calls to block when no data (or incomplete data) is available, then you need to use non-blocking IO instead for your reads. This is complex topic, which you need to research, but often used when you don't want to block and need to read arbitary length messages. You can look here for some tips.
Another option would be to use something like EventMachine instead, which makes it easier to deal with sockets in situations like this, without having to worry about blocking in your code.
Suppose I want to serialize and transmit protobuf binaries with ZMQ using a protocol defined in cake.proto:
syntax = "proto3";
message Cake {
int32 radius = 1;
}
I can find plenty of examples for the PUB/SUB pattern where a subscriber filters a topic with a string:
socket.setsockopt_string(zmq.SUBSCRIBE, "abc")
But how does subscribing to topics work when it comes to protobuf binaries? Do I use the bytes themselves or does ZMQ provide an wrapper for a message with a header I can use for cases like this?
There is no wrapper for this, the subject is just the first frame of the zeromq message.
If you are confident your protobuf messages will always start with the specific sequence of bytes (that make your subject) then yes you can just subscribe to that byte prefix pattern.
The other option is to copy the subject pattern into an initial frame then add the protobuf frame(s) via ZMQ_SNDMORE. If you can pack many protobuf frames into that same zmq message then the efficiency is good. If each protobuf message has its own "subject" then you will have the overhead of an extra subject frame per protobuf.
The question is regarding the deserialization process. I thought the first bytes describe the tag and the corresponding on-wiretyp. But what if two services using in their messages the same tag/identifier and datatype?
Protobuf actually doesn't distinguish between the two! Consider the following two messages:
message Foo {
int32 foo = 5;
}
message Bar {
int32 bar = 5;
}
Both of these messages will appear to be exactly the same. The reason is that the message doesn't carry the schema along with it. This makes the messages more compact and faster to process, with the minor downside of possibly being misinterpreted.
If you are using gRPC, the messages may appear the same, but they can be distinguished by which service they are sent to. For example:
service MyService {
rpc EatTheFoo(Foo) returns (Bar);
}
service YourService {
rpc GoToTheBar(Bar) returns (foo);
}
Even though both services take and receive messages that appear to be the same, gRPC will include the name of the service and message when sending the message. Under the hood, it will turn into an HTTP/2 request that looks like:
POST /MyService/EatTheFoo HTTP/2
which is then followed by the Foo message. If someone accidentally tried to send a Bar message, the server would see that the method name was wrong and reject the RPC. Thus, the chance of being misinterpreted is pretty small.
I'm using the github.com/hypebeast/go-osc/osc package to send OSC messages to an OSC server. For this I'm using OSCulator so that I can route the data as MIDI to Abelton Live.
The problem I'm having is I can not find any information on message formatting for things like note on, note off, duration etc. I found a guide on the OSCulator website that's a little helpful, but it doesn't go into much detail on messaging: http://s3.amazonaws.com/osculator/doc/OSCulator+2.12+Manual.pdf
For example, the following function works just fine, but I have no idea what the message is really doing:
func note(pitch float32 , velocity float32) {
// TODO: Pass client into function. Find out it's type.
client := osc.NewClient("localhost", 8765)
noteMsg := osc.NewMessage("/4/toggle2")
client.Send(noteMsg)
msg := osc.NewMessage("/4/xy")
msg.Append(pitch)
msg.Append(velocity)
client.Send(msg)
}
I mean, what purpose does the 4 play in this, and what is xy? Also, what other messages are available apart from toggle2? I thought there would be some sort of documentation online that has all the different types of messages available for MIDI type applications.
Your question seems to be more related to OSC itself.
OSC works like this:
You send a message to a server. A message is composed by an adress and some values.
In this case, /4/xy is the address. The 4 and the slashes you define what will be. When you receive it on the other side you will know what you want to receive, which means the address you're sending. So you will configure the server or the receiver to do something when it receives a message from a specific adress.
In the same way, you are appending values to the message. The quantity of values you already know, so you just have to do what you want with them when you receive.
Basically, if you decide to have a keyboard sending notes, you would use something like /keyboard/note as adress and send one value at a time, so you would read this value and do something with it.
I am designing an app that has multiple components, mainly written in java and python.
I am thinking of using "JMS-Active MQ" as Message Oriented Middleware for components and "protocol buffers".
1) Is it good way to go ahead? In our case "message size" can go well over 10MB, will protocol buffer still have advantages for cross component communication? Are there any better communication "protocols" for cross platform applications that can handle "huge amounts of data"?
2) I created a Proof of concept my sending a "protocol buff" as message over the "ActiveMQ", I am using the sample proto file, in google's java tutorial.
AddressBook.Builder book = AddressBook.newBuilder();
Person.Builder person = Person.newBuilder();
person.setName("mayank");
person.setId(2);
book.addPerson(person);
TextMessage message = session.createTextMessage();
message.setText(book.build().toString());
In another java app, I listen to this message and try to deserialize it back into AddressBook object:
public void onMessage(Message message) {
TextMessage msg = (TextMessage) message;
try {
System.out.println(msg.getText());
CodedInputStream stream =CodedInputStream.newInstance(msg.getText().getBytes());
AddressBook book = AddressBook.parseFrom(stream);
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
This causes an exception:
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol
message, the input ended unexpectedly in the middle of a field. This could
mean either than the input has been truncated or that an embedded message
misreported its own length.
at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
I do not know what's wrong ..?
Regarding 1), protocol buffers' documentation talks about transferring large messages here.
About 2), the problem seems to be the way you are transferring book.
Take a look at how the tutorial you mention writes the message to an OutputStream. Instead of a TextMessage, you should use a binary one, for example by writing the bytes first to a ByteArrayOutputStream and then to the message.
Main issue is that for some reason you are using TextMessage instead of correct choice of BytesMessage: protobuf is binary encoding and not textual one.
And if you absolutely want to misuse TextMessage for non-textual message, you MUST specify encoding to use for getBytes(); and encoding to use MUST match whatever was used to convert binary payload to bytes. And if you use UTF-8, you have probably already corrupted message... (ISO-8859-1, aka Latin-1 actually would work because it is single-byte encoding).
Beyond this, you can use protobuf as well as any number of other formats over JMS. I prefer JSON for readability and extensibility, but if you are already using protobuf, and this is internal system (not something exposed to external parties), protobuf works too.