I searched on the internet but couldn't find anything useful. First, I was thinking to use Protocol Buffers but it doesn't provide built in feature to track multiple messages (where one message finish and second starts) or message self delimiting, but I read about this feature in Thrift white paper and it seems good to me. Now I am thinking to use Thrift instead of Protocol Buffers.
I am working on custom protocol for that I don't require RPC, could someone suggest if I can use Thrift without RPC (as its in the Protocol Buffers, one simply use the streams function) and some starting point as thrift documentation is a bit cumbersome.
Thanks!
Yes, It is possible. A similar answer is given Here. Apache thrift can be used without RPC you can simply use transport and protocol layers related libraries as they are defined in the documentation.
Apache Thrift is indeed a RPC- and serialization framework. The serialization part is used as part of the RPC mechanism, but can be used standalone. For the various languages there are samples and/or supporting helper classes available. If this is not the case for your particular language, the necessary code pretty much boils down to this (pseudo code):
var data = InitializeMyDataStructure(...);
var trans = new TStreamTransport(...);
var prot = new TJSONProtocol(trans);
data.write(prot);
Both transport(s) and protocol are pluggable, so instead JSON and a stream you are free to use your own protocol, and (for example) a file transport. Or whatever else combination makes sense for your use case and is supported for your target language.
as thrift documentation is a bit cumbersome.
You are free to ask any question, be it here or in the mailing list. Furthermore, we have a nice tutorial and the Test server/client pairs are also good examples for typical use cases.
Related
I currently have a primitive RPC setup relying on JSON transferred over secured sockets, but I would like to switch to gRPC. Unfortunately I also need access to AF_UNIX on windows (Which Microsoft recently started supporting, but gRPC has not implemented).
Since I have an existing working connection (managed with a different library), my preference would be to just use that in conjunction with GRPC to send/receive commands in place of my JSON parsing, but I am struggling to identify the best way to do that.
I have seen Plugging custom transport into gRPC but this question differs in the following ways (As well as my hope for a more recent answer)
I am wanting to avoid making changes to the core of gRPC. I'd prefer to extend it if possible from within my library, but the answer here implies adding a new transport to gRPC.If I did need to do this at the transport level, is there a mechanism to register it with gRPC after the core has been built?
I am unsure if I need to define this as a full custom transport, since I do already have an existing connection established and ready. I have seen some things that imply I could simply extend Channel, but I might be wrong.
I need to be able to support Windows, or at least modern versions of it (Which means that the from_fd options gRPC provides are not available since they are currently only implemented for POSIX)
Has anyone solved similar problems with gRPC?
I may have figured out my own answer. I seem to have been overly focused on gRPC, when the service definition component of Protobuf is not dependent on that.
How can i write my own RPC Implementation for Protocol Buffers utilizing ZeroMQ is very similar to my use case, with https://developers.google.com/protocol-buffers/docs/proto#services seeming to resolve my issue (And this also explains why I seem to have been mixing up the different kinds of "Channels" involved
I welcome any improvements/suggestions, and hope that maybe this can be found in future searches by people that had the same confusion.
I'm trying to have a messaging service(over TCPIP) between windows on PC(running C# app) and linux on an SoC board(running C++ app). I went through Google's protocol buffers and thought that I can serialize the structure (data struct) into a buffer and write this buffer over sockets(saw an example too).
My question, is there another way to do this? What does protocol buffer-RPC do? I went through their documentation but they don't specify on how to do it with TCPIP. Maybe someone has examples for this or detailed documentation?
Thank you,
Karthik.
My question, is there another way to do this?
There are many ways to do this. Google Protocol Buffers is library for cross-platform object serialization. You can share this serialized data in many different ways. My recommendation for you is using ZMQ (ZMQ GUIDE) which is perfectly suited for applications like yours.
I have a similar use case to you. I wrote Linux C++ ZMQ server on Raspberry Pi and Python ZMQ client dedicated for telemetry and remote management purposes. This library is efficient, lightweight, works perfectly with Google Protocol Buffers and has binding to all major programming languages. If you decide to use this library do not hesitate to ask me about working code example.
See if the Google Developers page on how the protocol buffers wire format is encoded helps with your question: https://developers.google.com/protocol-buffers/docs/encoding
Thrift sounds awesome but can't find some basic stuff I'm used to in RPC frameworks (as HttpServlet). Example of the things I can't find: session management, filtering, upload/download progress.
I understand that the missing stuff might be a management layer on top of Thrift. If so, any example of such a layer? Perhaps AOP (Aspect Oriented)?
I can't imagine such a layer that compiles to all languages and that's I'm missing. Taking session management as an example, there might be several clients that all need to do some authentication and pass the session_id upon each RPC. I would expect a similar API for all languages doing so.
Anyone knows of a a management layer for Thrift?
So thrift itself is not going to help you out a lot here.
I have had similar desires, and have a few suggestions:
1. Put your management objects into the IDL
Simply add an api token or common transfer data struct as a parameter to all of your service methods. Set it as parameter id 15 so that it will always be the last parameter, even if you add others in the middle.
As the first step in your handler you can validate/store/do whatever with the extra data.
This has the advantage that it is valid in any platform that thrift supports.
2. Use thrift over http
If you use http as your transport, you can include whatever data as you want as http headers, and the thrift content as the body.
This will often require a custom http client for every platform you use to inject the data, and a custom handler on the server to use the data, but neither of those are prohibitively difficult.
3. Hack the protocol
It is possible to create your own custom protocol that wraps another protocol and injects custom data. Take a look at how the multiplexed protocol works in the thrift library for most languages:
c# here. It sends the method name across the wire as service:method. The multiplexed processor unwraps this encoding and passes it on to the appropriate processor.
I have used a similar method to encode arbitrary key/value pairs (like http headers) inside the method name.
The downside to this is that you need to write a more complicated extension for each platform you will be using. Once. It varies a bit from language to language how this works, but it is generally simple enough once you figure it out once.
These are just a few ideas I have had, and I am sure there are others. The nice thing about thrift is how the individual components are decoupled from each other. If you have special needs you can swap any of them out as you need to to add specific functionality.
EDIT: I forgot to include the prime candidate for web applications: JSON over HTTP/REST + Comet. It combines the best features of the others (below)
Persevere basically bundles everything I need in a server
The focus for Java and such is definitely on Comet servers, but it can't be too hard to use/write a client.
I'm embarking on an application with a server holding data, and clients executing operations which would affect this data, and thus require some sort of notification across all interested/subscribed clients.
The first client will probably be written in WPF, but we'll probably need to add clients written in other languages, e.g. a Java (Swing?) client, and possibly, a web client.
The actual question(s): What protocol should I use to implement this? How easy would it be to integrate with JS, Java and .NET (precisely, C#) clients?
I could use several interfaces/protocols, but it'd be easier overall to use one that is interoperable. Given that interoperability is important, I have researched a few options:
JSON-RPC
lightweight
supports notifications
The only .NET lib I could find, Jayrock doesn't support notifications
works well with JS
also true of XML-based stuff (and possibly, even binary protocols) BUT this would probably be more efficient, thanks to native support
Protobuf/Thrift
IDL makes it easy to spit out model classes in each language
doesn't seem to support notifications
Thrift comes with RPC out of the box, but protobufs don't
not sure about JS
XML-RPC
simple enough, but doesn't support notifications
SOAP: I'm not even sure about this one; I haven't grokked this yet.
seems rather complex
Message Queues/PubSub approach: Not strictly a protocol, but might be fitting
I hardly know anything about them, and got lost amongst the buzzwords`-- JMS? **MQ?
Perhaps combined with some RPC mechanism above, although that might not be strictly necessary, and possibly, overkill.
Other options are, of course, welcome.
I am partial to the pub/sub design you've suggested. I'd take a look at ZeroMQ. It has bindings to C#, Java, and many other platforms.
Bindings list: http://www.zeromq.org/bindings:clr
I also found this conversation on the ZeroMQ dev listing that may answer some questions you have about multiple clients and ZeroMQ: http://lists.zeromq.org/pipermail/zeromq-dev/2010-February/002146.html
As XMPP was mentioned, SIP has a similar functionality. This might be more accessible for you.
We use Servoy for this. It does automatic data broadcasting to web-clients and java-clients. I'm not sure if broadcasts can be sent to other platforms, you might be able to find an answer to that on their forum.
If you want to easily publish events to clients across networks, you may wish to look at a the XMPP standard. (Used by, amongst other things, Jabber and Google Talk.)
See the extension for publish-subscribe functionality.
There are a number of libraries in different languages including C#, Java and Javascript.
You can use SOAP over HTTP to modify the data on the server and SOAP over SMTP to notify the subscribed clients.
OR
The server doesn't know anything about the subscription and the clients call the server by timeout to track updates they are interested in, using XML-RPC, SOAP (generated using WSDL), or simply HTTP GET if there is no need to pass back complex data on tracking.
I'm trying to create a raw socket using Ruby.
The problem is, there isn't anything called "raw socket" there and, on the other hand, the Socket class itself is not fully documented.
Does anybody have some code samples for that kind of socket in Ruby, or maybe some kind of a documentation for that?
By the way, I already know how to work with TCPSocket and TCPServer classes, and what I need is particularly a raw socket.
Google brings up the following result: http://www.ruby-forum.com/topic/90408
Short version:
require 'socket'
rsock = Socket.open(Socket::PF_INET, Socket::SOCK_RAW, Socket::IPPROTO_RAW)
rsock.send(string, flags)
rsock.recv(1024)
More documentation on the various Socket classes: http://www.rubycentral.com/pickaxe/lib_network.html
(The whole raw sockets thing is rather nasty on unices since it usually requires root access. I did not test this code. You may need to construct the whole packet yourself if you're not using IPSocket)
Have a look at the racket gem (https://rubygems.org/gems/racket). It seems to be a bit outdated since the last version was released in 2009 but its also used in the metasploit framework.
Have a look at PacketFu. It is very well maintained and used by the Metasploit Project.