I am working on a gRPC service. And we are using protocol buffer. We removed some fields from an existing protocol buffer file. We added them as reserved fields so no one would use them in an incompatible way.
The usage of those fields we deleted doesn't break the old clients' behavior.
But when I contacted them, I told them that if they want, they can update the protocol buffers because we changed them.
They asked me not to change the protocol buffers (My team's protocol buffers) before contacting them and asking them for a review.
I'm not so experienced in the protocol buffer world, but doesn't that defeats the purpose of using protocol buffer in the first place?
I mean the API should stay backward and forward compitable. Unless we change the behavior of course. But the way it is used stays backward/forward compatible.
Is his request valid, and we must communicate with all of our clients before changing the protocol buffer?
thanks.
Related
I am evaluating using gRPC. On the subject of compatibility with 'schema evolution', i did find the information that protocol buffers, which are exchanged by gRPC for data serialization, have a format such that different evolutions of a data in protobuf format can stay compatible, as long as the schemas evolution allows it.
But that does not tell me wether two iterations of a gRPC client/server will be able to exchange a command that did not change in the schema, regardless of the changes in the rest of the schema?
Does gRPC quarantee that an older generated version of a client or server code will always be able to launch/answer a command that has not changed in the schema file, with any more recent schema generated code on the interlocutor side, reglardless of the rest of the schema? (Assuming no other breaking changes like a non backward compatible gRPC version change)
gRPC has compatibility for a method as long as:
the proto package, service name, and method name is unchanged
the proto request and response messages are still compatible
the cardinality (unary vs streaming) of the request and response message remains unchanged
New services and methods can be added at will and not impact compatibility of preexisting methods. This doesn't get discussed much because those restrictions are mostly what people would expect.
There is actually some wiggle room in changing cardinality as unary is encoded the same as streaming on-the-wire, but it's generally just better to assume you can't change cardinality and add a new method instead.
This topic is discussed in the Modifying gRPC Services over Time talk (PDF slides and Youtube links available). I'll note that the slides are not intended to stand alone.
I am building a set of programs that consist of multiple clients and a single server.
The clients are frequently pushing small packets of data to the server, which will validate the information (returning an error if the data is invalid), and process the received information. The information may then incur the firing of events, which clients will be subscribed to, allowing for clients to be instantly (or as close as possible) notified (along with a small amount of data).
I have some ideas about how to do this, but I am trying to avoid creating a protocol of my own, mainly as I'm sure it would take forever and I would probably make a few errors. So I was wondering if there are any existing protocols that I could implement into my system that would provide such functionality.
The number of clients will initially be quite small, but will be growing over time to potentially include 1000's of clients (with their own subscriptions), and several front end servers (each one handling a subset of subscriptions) parsing the information back and forth with back end servers for improved capability.
So, if anyone knows of any existing protocols that implement these requirements and functionality, that would be fantastic.
EDIT
I am currently looking at the XMPP protocol, and the JXTA protocol suite (for reference, and implement with another language). Both seem quite good and provide the necessary connectivity, but I have not had the opportunity to test each of them out in my environment, or if they are even suitable for what I am attempting.
Additionally, some of the network clients will be outside of the local network and operating over WAN. Security is not so much of an issue, but I need to take into account the increase latency of this, and firewall rules (local to the connection that is hosting the application and ISP firewalls) that could be blocking certain ports or transport protocols (I have read some text that said that some ISPs where blocking UDP packets, but not sure of how wide this goes. I can do it at home, the office, mobile, friends houses, etc and have yet to experience it myself).
I'm sorry if the following is not exactly what you're after but I am slightly confused by your use of the word 'protocol'. I understand a protocol to be a 'communication specification' only, where the implementation is left entirely to you. If that is the case I always find the the following graphic usefull, link.
If on the other hand you are looking for a solution which allows you to easily implement the networking side of your application, helping save time, then checkout the following network libraries, which implement their own custom protocol:
NetworkComms.Net
Lidgren
ZeroMQ
Disclaimer: I'm a developer for NetworkComms.Net
At my company, we've standardized on using Protocol Buffers over a message bus as a way to allow services to communicate.
This is fine, however I'm running into a problem in trying to figure out how to structure common definition files that I'd like to share amongst different teams. Is there a commonly accepted way to make collections of protocol buffer definitions available across teams?
Also, is it just a fact of life that all import headers have to refer to the directory in which the protocol buffer compiler executes? Frankly this seems a little silly since the protocol buffers allow for namespace definitions. Or is this just an artifact of the Java centric origins of Protocol Buffers?
I can only answer part of the question:
Also, is it just a fact of life that all import headers have to refer to the directory in which the protocol buffer compiler executes?
you can use --proto_path= option to specify where *.proto's exist + all the imbedded proto's
An opinion question: do you think it's safe already to use WebSockets what with the changing protocols? If not, when do you reckon the protocols will be finished?
Thanks!
The protocol isn't really changing much any more. Most of the discussion is around optional extensions and phrasing in the specification. There was no wire protocol change between HyBi-08, 09 and 10 (which is why the handshake version has stayed at '8') and very little change between 08 and the previous several versions. The protocol has also completed last call and been referred to the IESG/IETF so radical changes are not likely unless some serious issue is discovered that throws the protocol back into the HyBi work group for rework.
One of the bigger changes coming soon is for in the HTML API related to binary data support and close events. However, these changes are basically just additive and still backwards compatible with the current API.
If you want to open a two-way connection between the browser and server, the only choice is to poll (hammer the server), or use comet (crufty and prone to disconnects).
Why don't browsers just let you open up a plain TCP connection? Is there any practical benefit to not having this ability?
The underlying protocol HTTP is basically a half duplex communication protocol which is stateless as well and does not supports full duplex communication. However, with HTML 5 websockets things are going to change. Websockets is a new standard which is being considered in HTML 5 specification. Once the specifications have been finalized and all the browser vendors have adapted the standards you can possibly use websockets to establish a dedicated TCP connection through browsers itself.
We must also keep in mind that HTTP was basically designed to deliver documents & share information between the geographically distributed teams and it was not intended to be a communication protocol as such.
Having said that, there are already companies which have built some messaging gateways to enable you to implement full duplex communication.
Given that this functionality is effectively available through flash, there's no real security rationale - but these days no browser wants to be the first to implement a non-standard extension like that. Moreover, there's no easy way to do threads, which could make using a socket rather awkward.
Over the years so many aspects or elements of the web have been hijacked in order to deliver richer experiences. Comet is but one example, where long lived connections were exploited in order to allow server side push. Originally web pages were just meant to be hyperlinked documnts of text and not the rich applications we often see today. Hacks and abuses of what the original thought intended will continue, until one day these things become more standardised.
The answer to your question is essentially no, there is no tangible advantage to not being able to open a two-way connection between client and server in a browser. The reason it can't be done is simply that this was not the intention of web browsers, which were developed to poll/retrieve documents. With the advent of Rich Internet Applications, it has become desirable to have such functionality, but previously this had never been the goal of a browser. Currently there is a void to be filled by an eventual protocol or implementation of an existing protocol which will govern two-way communication between a browser and the server. There are existing techniques used to simulate this behavior to different degrees (AJAX, Comet, etc.) or it can be accomplished with embedded objects (Java, Flash, ActiveX Controls in IE) but these are simply paths around the void, not bridges over it.
We will simply have to wait (or act) for the standard to be written and the implementation to follow. More than likely, the implementation will actually come first, and we will have a fistfull of new cross-browser compatibility issues to enjoy :) Oh, the bleeding edge!
Firewalls. Non-HTTP traffic is often blocked by firewalls, so opening up a random TCP port for communication will often fail.