If I want to write a node for a P2P application (like Bitcoin, Bitorrent, etc.) there are a lot of parts that are the same:
I need to bootstrap to the network (discover other peers)
I need to manage a list of peers, and monitor their states
I need to retrieve lists of more peers from my neighbour peers
Etc, etc.
Since I don't want to re-invent the wheel, is their a framework that I could as a sort of base library to build on?
You mention both bitcoin and bittorrent, which are quite different, so I'm assuming you don't want to be bound to any specific protocol or even serialization format.
And yet you mention peer-discovery and stats-management which are high-level concerns, be built on top of some network protocol.
But the protocol dictates how such a mechanism would work.
It sortof sounds like you're asking if there are pre-built roofs that would fit on skyscrapers just as well as on a wood cabin.
So if you actually want to design your own protocol you probably should look more at the foundation first.
which language do you want to use
what IO / event processing libraries are available
what protocol parsers and serializers are available
do you aim for throughput? low memory footprint? low latency? minimal amount of programmer-hours spent?
what kind of security is needed? heavy crypto use at the protocol level will need a trustworthy crypto library (don't roll your own!)
what kind of auxiliary things do you need (where does the data go? filesystem? databases? do you need a UI?)
Alternatively, depending how one interprets your question, if you want to write a client for a specific network then you should simply look for a library implementing the core concepts of that specific network while freeing you up to implement the rest of the application.
In bittorrent's case such an example would be libtorrent
Related
I need a embeddable messaging component for Golang application.I looked into distributed messaging system like NSQ etc, but prefer one which is much simpler, stable and embeddable (no separate daemon). Systems like NSQ would be overkill for my requirement and I don't think it is easily embeddable.
I need such a component to distribute the payload/data reliably (delivered once atleast) among my processing distributed engines and embeddable would make installation and configurations easy.
Any information would be appreciated.
I use RabbitMQ in production with this library https://github.com/streadway/amqp. This is not an official library but the implementation is quite good.
I use it to improve fault tolerance and scalability for IOT.
Pro:
Easy to use
Great administration tools
Many languages implementations
Many patterns availables https://www.rabbitmq.com/getstarted.html
Cons:
Not really efficient for big payloads
Probably not the fastest (but fast enough to my projects)
I need to do something relatively simple, and I don't really want to install a MOM like RabittMQ etc.
There are several programs that "register" with a central
"service" server through TCP. The only function of the server is to
call back all the registered clients when they all in turn say
"DONE". So it is a kind of "join" (edit: Barrier) for distributed client processes.
When all clients say "DONE" (they can be done at totally different times), the central server messages
them all saying "ALL-COMPLETE". The clients "block" until asynchronously called back.
So this is a kind of distributed asynchronous Observer Pattern. The server has to keep track of where the clients are somehow. It is ok for the client to pass its IP address to the server etc. It is constructable with things like Boost::Signal, BOOST::Asio, BOOST::Dataflow etc, but I don't want to reinvent the wheel if something simple already exists. I got very close with ZeroMQ, but non of their patterns support this use-case very well, AFAIK.
Is there a very simple system that does this? Notice that the server can be written in any language. I just need C++ bindings for the clients.
After much searching, I used this library
https://github.com/actor-framework
It turns out that doing this with this framework is relatively straightforward. The only real "impediment" to using it is that the library seems to have gotten an API transition recently and the documentation .pdf file has not completely caught up with the source. No biggie since the example programs and the source (.hpp) files get you over this hump. However, they need to bring the docs in sync with the source. In addition, IMO they need to provide more interesting examples on how to use c++ Actors for extreme performance. For my case it is not needed, but the idea of actors (shared nothing) in this use-case is one of the reasons people use it instead shared memory communication when using threads.
Also, getting used to the syntax that the library enforces (get used to lambdas!) if one is not used to state of the art c++11 programs it can be a bit of a mind-twister at first. Then, the triviality of remembering all the clients that registered with the server was the only other caveat.
STRONGLY RECOMMENDED.
It is unbelievable that ZeroMQ uses select() on Windows, I didn't know that until I have completes my code and started performance test. They should present this information on their web site with big red font.
Is there anyway to replace ZeroMQ's select()?
IOCP is proactor model and can't be easily integrated into it, how about WSAEventSelect, this is also a reactor model and have a near performance like poll.
Another choice for me is http://nanomsg.org/, but it is still alpha.
One of the main objectives in Zeromq is to provide a consistent API for communication between threads, processes, nodes, and clusters. Protocol specific optimization is outside of this scope because of the ways that it can effect other areas of communication. For example, shared memory would be a better form of IPC, but UNIX domain sockets make a consistent API easier. It would also be nice to know when an endpoint disconnects, but how would you implement such behavior between threads?
Their main goal is to allow every pattern to work the same way regardless of topology, protocol, system, or language, to the point that any mixture can be used regardless of how odd it may seem (node.js Websockets communicating with C# brokers passing messages to Ruby and PHP workers which share work with java threads, etc.)
Each of it's features would be enhanced greatly if optimised for each specific protocol and system, but that would also make uniform patterns close to impossible.
BTW, they might accept a pactch if you could find a way to implement iocp while still maintaining this versatility and neutrality.
PPS, nanomsg is made by one of the main original developers of Zeromq. Crossroads.IO is a direct fork of Zeromq, by original Zeromq developers as well and including some developers of nanomsg. if I'm not mistaken, Nano will likely become the core of crossroads when complete.
I am looking into using ZeroMQ as the messaging/transport layer for a fairly large distributed system, mainly targeting monitoring and data collection (many producers, a few consumers).
As far as I can see there are currently two different implementations of the same concept; ZeroMQ and Crossroads I/O, the latter being a fork of ZeroMQ (in 2012?).
I am trying to figure out which one to use and wonder about the differences between them, but have so far not found much information regarding this.
For example:
Are they compatible on the wire?
Are they API compatible, i.e. some kind of common base API, possibly with different add-ons?
Do they both implement support for ZMTP (ZeroMQ Message Transport Protocol)?
Do they share some kind of common understanding of future development or will they continue in two separate and possible different directions?
What are the pros/cons in relation to the other?
Basically, how do one choose one over the other?
Crossroads.io is pretty dead since Martin Sustrik has started on a new stack, in C, called nano: https://github.com/250bpm/nanomsg
Crossroads.io does not, afaik, implement ZMTP/1.0 nor ZMTP/2.0 but its own version of the protocol.
Nano has pluggable transports and we'll probably make a ZMTP transport for that. Nano is really nice, a rethinking of the original libzmq library, and if it's successful would make a good new kernel.
Ideally, Nano would interoperate both at the API and the protocol level, so be a pluggable replacement for libzmq. It does have quite a long way to go, though.
Note that there are now several rewrites of libzmq emerging, including JeroMQ (Java) and NetMQ (C#). These two do implement ZMTP/1.0 and ZMTP/2.0 properly. There are also other libraries like Axon (https://github.com/visionmedia/axon) which are heavily inspired by 0MQ but not compatible.
Based on experience, users value interoperability more than almost anything else, so it's quite likely that different 0MQ-like stacks will end up speaking the same protocols.
I am looking for a library which I can plug into a distributed application which implements any gossip-based membership protocol.
Such a library would allow me to send/receive membership lists, merge received membership lists, etc... Even better would be if the library implemented a protocol with performance O(logn) performance guarantees.
Does anyone know of any open source library like this? It doesn't need to meet all of the aforementioned requirements; even something partially implemented would be helpful.
Take a look at this on google code:
http://code.google.com/p/gossip-protocol-java/
I happen to stumble upon it yesterday whilst looking for java based gossip implementation. It's more a reference implementation for someone to build upon, but it gives the general idea, and after reading through the code you'll definitely be able to build your own or branch what's there to add any features you need.
HTH
Have you looked at Apache Zookeeper? I'm not sure if it's what you're looking for.
ZooKeeper is a high-performance
coordination service for distributed
applications. It exposes common
services - such as naming,
configuration management,
synchronization, and group services -
in a simple interface so you don't
have to write them from scratch. You
can use it off-the-shelf to implement
consensus, group management, leader
election, and presence protocols.
C# bindings are also available.