I am a beginner learning about gRPC and protocol buffers. I am using grpc in golang.
How can I define a struct in protocol buffer and access that in mongodb?
Is it necessary to create a bson struct as well to get the data from mongodb?
Related
I understand that we need to marshall object to serialise it, and unMarshall to deserialise it. However my question is, when do we invoke marshal and unmarshal and why do we serialise the objects if we are going to deserialise it again soon?
PS I just started to learn Go and Proto so would really appreciate your help. Thank you.
Good question!
Marshaling (also called serializing) converts a struct to raw bytes. Usually, you do this when you're sending data to something outside your program: you might be writing to a file or sending the struct in an HTTP request body.
Unmarshaling (also called deserializing) converts raw bytes to a struct. You do this when you're accepting data from outside your program: you might be reading from a file or an HTTP response body.
In both situations, the program sending the data has it in memory as a struct. We have to marshal and unmarshal because the recipient program can't just reach into the sender's memory and read the struct. Why?
Often the two programs are on different computers, so they can't access each other's memory directly.
Even for two processes on the same computer, shared memory is complex and usually dangerous. What happens if you're halfway through overwriting the data when the other program reads it? What if your struct includes sensitive data (like a decrypted password)?
Even if you share memory, the two programs need to interpret the data in the same way. Different languages, and even different versions of Go, represent the same struct differently in memory. For example, do you represent integers with the most-significant bit first or last? Depending on their answers, one program might interpret 001 as the integer 1, while the other might interpret the same memory as the integer 4.
Nowadays, we're usually marshaling to and unmarshaling from some standardized format, like JSON, CSV, or YAML. (Back in the day, this wasn't always true - you'd often convert your struct to and from bytes by hand, using whatever logic made sense to you at the time.) Protobuf schemas make it easier to generate marshaling and unmarshaling code that's CPU- and space-efficient (compared to CSV or JSON), strongly typed, and works consistently across programming languages.
I am working on an ECG module, which is giving out data in bytes. There's a protocol document about it explaining like how to structure the packets, that are coming out of the module.
I want to decode that data.
I am confused whether protocol buffers will help in this or not. Any other methods that would be helpful in this decoding and writing that protocol in Python ?
Protocol buffers only works with its own encoding format.
For decoding a manufacturer-specific custom binary format, I would suggest Python's built-in struct module. Here is a basic tutorial which is easy get started with.
If the manufacturer-specific format is text based instead, you can either use basic string manipulation to split it into tokens, or one of the parsing libraries for Python.
I am working on a project where we are using protocol buffers to create and parse some of our messages (protobuf-net). This is so elegant, that I would like to use this same deserialization method to parse other messages emanating from external non-protobuf generated sources. Is this possible?
I would imagine that it could be possible to specify all of the .proto fields to be fixed size (i.e. not like variable ints). The question is then whether you could replace the protobuf headers with other magic numbers or whichever header the 3rd party protocol uses.
If this is a bit confusing, an example may shed some light:
Let's say you buy a fancy toaster that exposes an ethernet port. It supports a proprietary but well documented protocol. Can you burn heart shaped patterns on your toast using protobuf?
At the moment, no: the library is tied to the protobuf wire specification; it does not have support for non-protobuf data.
In a way, it is a bit like asking: "can XmlSerializer read/write json?". It isn't something that is on my list of things to look at, to be honest.
What are the pros and cons of protocol buffer (protobuf) over GSON?
In what situation protobuf is more appropriate than GSON?
I am sorry for a very generic question.
Both json (via the gson library) and protobuf are portable between platorms; but
protobuf is smaller (bandwidth) and cheaper (CPU) to read/write
json is human readable / editable (protobuf is binary; hard to parse without library support)
protobuf is trivial to merge fragments - just concatenate
json is easily passed to web page clients
the main java version of protobuf needs contract-definition (.proto) and code-generation; gson seems to allow arbitrary pojo usage (there are protobuf implementations that work on such objects, but not for java afaik)
If performance is key : protubuf
For use with a web page (JavaScript), or human readable: json (perhaps via gson)
If you want efficiency and cross-platform you should send raw messages between applications containing the information that is necessary and nothing more or less.
Serialising classes via Java's own mechanisms, gson, protobufs or whatever, creates data that contains not only the information you wish to send, but also information about the logical structures/hierarchies of the data structures that has been used to represent the data inside your application.
This makes those classes and data mapping dual purpose, one to represent the data internally in the application, and two to be transmitted to another application. Those two roles can be conflicting there is an onus on the developer to remember that the classes, collections and data layout he is working with at any time will also be serialised.
Does google protocol buffers support stl vectors, maps and boost shared pointers? I have some objects that make heavy use of stl containers like maps, vectors and also boost::shared_ptr. I want to use google protocol buffers to serialize these objects across the network to different machines.
I want to know does google protobuf support these containers? Also if I use apache thrift instead, will it be better? I only need to serialize/de-serialize data and don't need the network transport that apache thrift offers. Also apache thrift not having proper documentation puts me off.
Protocol buffers directly handles an intentionally small number of constructs; vectors map nicely to the "repeated" element type, but how this is presented in C++ is via "add" methods - you don't (AFAIK) just hand it a vector. See "Repeated Embedded Message Fields" here for more info.
Re maps; there is no inbuilt mechanism for that, but a key/value pair is easily represented in .proto (typically key = 1, value = 2) and then handled via "repeated".
A shared_ptr itself would seem to have little meaning in a serialized file. But the object may be handled (presumably) as a message.
Note that in the google C++ version the DTO layer is generated, so you may need to map between them and any existing object model. This is usually pretty trivial.
For some languages/platforms there are protobuf variants that work against existing object models.
(sorry, I can't comment on thrift - I'm not familiar with it)