I am looking for a library which I can plug into a distributed application which implements any gossip-based membership protocol.
Such a library would allow me to send/receive membership lists, merge received membership lists, etc... Even better would be if the library implemented a protocol with performance O(logn) performance guarantees.
Does anyone know of any open source library like this? It doesn't need to meet all of the aforementioned requirements; even something partially implemented would be helpful.
Take a look at this on google code:
http://code.google.com/p/gossip-protocol-java/
I happen to stumble upon it yesterday whilst looking for java based gossip implementation. It's more a reference implementation for someone to build upon, but it gives the general idea, and after reading through the code you'll definitely be able to build your own or branch what's there to add any features you need.
HTH
Have you looked at Apache Zookeeper? I'm not sure if it's what you're looking for.
ZooKeeper is a high-performance
coordination service for distributed
applications. It exposes common
services - such as naming,
configuration management,
synchronization, and group services -
in a simple interface so you don't
have to write them from scratch. You
can use it off-the-shelf to implement
consensus, group management, leader
election, and presence protocols.
C# bindings are also available.
Related
I need a embeddable messaging component for Golang application.I looked into distributed messaging system like NSQ etc, but prefer one which is much simpler, stable and embeddable (no separate daemon). Systems like NSQ would be overkill for my requirement and I don't think it is easily embeddable.
I need such a component to distribute the payload/data reliably (delivered once atleast) among my processing distributed engines and embeddable would make installation and configurations easy.
Any information would be appreciated.
I use RabbitMQ in production with this library https://github.com/streadway/amqp. This is not an official library but the implementation is quite good.
I use it to improve fault tolerance and scalability for IOT.
Pro:
Easy to use
Great administration tools
Many languages implementations
Many patterns availables https://www.rabbitmq.com/getstarted.html
Cons:
Not really efficient for big payloads
Probably not the fastest (but fast enough to my projects)
I am looking into using ZeroMQ as the messaging/transport layer for a fairly large distributed system, mainly targeting monitoring and data collection (many producers, a few consumers).
As far as I can see there are currently two different implementations of the same concept; ZeroMQ and Crossroads I/O, the latter being a fork of ZeroMQ (in 2012?).
I am trying to figure out which one to use and wonder about the differences between them, but have so far not found much information regarding this.
For example:
Are they compatible on the wire?
Are they API compatible, i.e. some kind of common base API, possibly with different add-ons?
Do they both implement support for ZMTP (ZeroMQ Message Transport Protocol)?
Do they share some kind of common understanding of future development or will they continue in two separate and possible different directions?
What are the pros/cons in relation to the other?
Basically, how do one choose one over the other?
Crossroads.io is pretty dead since Martin Sustrik has started on a new stack, in C, called nano: https://github.com/250bpm/nanomsg
Crossroads.io does not, afaik, implement ZMTP/1.0 nor ZMTP/2.0 but its own version of the protocol.
Nano has pluggable transports and we'll probably make a ZMTP transport for that. Nano is really nice, a rethinking of the original libzmq library, and if it's successful would make a good new kernel.
Ideally, Nano would interoperate both at the API and the protocol level, so be a pluggable replacement for libzmq. It does have quite a long way to go, though.
Note that there are now several rewrites of libzmq emerging, including JeroMQ (Java) and NetMQ (C#). These two do implement ZMTP/1.0 and ZMTP/2.0 properly. There are also other libraries like Axon (https://github.com/visionmedia/axon) which are heavily inspired by 0MQ but not compatible.
Based on experience, users value interoperability more than almost anything else, so it's quite likely that different 0MQ-like stacks will end up speaking the same protocols.
I am trying to develop a publish/subscribe system.
To this end, I have read some papers and articles regarding it.
And they all talk about Messaging service as an integral part of publish/subscribe system.
My question is, can I develop a publish subscribe system without using MOM like JMS?
Or am I missing or oversimplifying things?
I do not think you are oversimplifying things. There are stand-alone products available that provide advanced functionality based on publish/subscribe, without being part of a larger MOM system.
One of them is a group of products implementing the Data Distribution Service (DDS) specification, as standardized by the Object Management Group (OMG). Check out this Wikipedia entry for a very brief introduction and list of references.
DDS supports many advanced data management features like a strong-typed and content aware databus, distributed state management and historical data access. Its rich set of Quality of Service settings allows to off-load a lot of the complexity from your applications to the middleware. This is all based on the publish/subscribe paradigm.
If you would tell more about your application, then I might be able to point you to similar use cases using this technology -- if you are interested.
It depends what you mean by "MOM". If you think MOM = JMS then yes, there are plenty of pub/sub applications which are not JMS servers (off the top of my head): 0MQ, TIBCO Rendezvous and the many AMQP implementations around.
I guess my definition of MOM is an infrastructure for reliably getting a message from one system to another in an asynchronous manner. Pub/sub is a feature on top of the message transport which allows a message to be distributed to multiple other systems. Once you get beyond the point of opening a socket and stuffing a bunch of bytes down it, I would argue you are in the realm of MOM.
So, no you don't need JMS to do pub/sub....there are plenty of open-source and closed-source alternatives out there. Which one depends on your requirements and skills.
You can look at multicast that provides one to many communication. Multicast does not require MOM, instead it requires multicast enabled IP network. Usually the network routers take care of creating copies of message and delivering messages to destinations.
I'm in the middle of a project where we will migrate a major software system based on a larger set of custom made technologies to be based on OSGi services. For this we will likely need a some sort of message bus that plays nice with OSGi services.
Sync and ASync delivery
Point-to-point only
Guaranteed delivery - preferable with persistence via files
Strict message ordered from the same client (Async mode), but necessarily from different clients
Support for process-to-process and node-to-node nice but not strictly required
Open source solutions will be preferred, but not required.
I have looked at eventbus (as recommended in https://stackoverflow.com/a/1953453/796559), but that does not seem to work well.
So the question is, which technologies match the above?
Tonny,
Having just come from a very similar, and successful project, please let me share my experience with you to save you some time and your company some money. First and foremost, ESB's were a really good idea 8 years ago when they were proposed. And, they solved an important problem: how do you define a business problem in a way that those pesky coders will understand? The goal was to develop a system that would allow a business person to create a software solution with little or no pesky developer interaction needed that would suck up money better spent on management bonuses.
To answer this the good folks at many organizations came up with JBI, BPMN and a host of other solutions that let business folks model the business processes they wanted to "digitize". But really, they were all flawed at a very critical level: they addressed business issues, but not integration issues. As such, many of these implementations were unsuccessful unless done by some high-priced consultant, and even then your prospects were sketchy.
At the same time, some really smart folks in the very late 90's published a book called "Enterprise Integration Patterns" which identified over 60 design patterns used to solve common integration problems. Many of the folks performing ESB stuff realized that their problem wasn't one of business modelling. Rather the problem was how to integrate thier existing applications. To help solve this Michael Strachan and some really smart guys started the Apache Software Foundation Project "Camel". Camel is a strict implementation of Enterprise Integration Patterns in addition to a huge number of components designed to allow folks like you and I to hook stuff together.
So, if you think of your business process as simply a need to send data from one application to another to another, with the appropriate data transformations between, then Camel is your answer. Now, what if you want to base the "route" (a specified series of application endpoints you want to send data thorugh) off of a set of configurable rules in a database? Well, Camel can do that too! There's an endpoint for that! Anyhow, dont' do the traditional ESB, its old and busted. And Absolutely do the camel thing.
Please let me know if this helps.
The OSGi specification defines a component "Event Admin" which is a lightweight pub-sub event subsystem.
From the RFC0157:
Event Admin specifies a means for an event source to send events to
event listeners. Event sources can create events with a topic and
properties and request Event Admin to deliver the events to event
listeners which have declared interest in specific event topics and/or
property values. The event source can request synchronous (and
unordered) delivery or asynchronous (and ordered) delivery.
Compared to your requirements, it would score as follows:
Sync and ASync delivery: Check
Point-to-point only: No. Pub-Sub
Guaranteed delivery - preferable with persistence via files: NO
Strict message ordered from the same client (Async mode): YES
Support for process-to-process: if (process == OSGi service) -> Yes
Support for node-to-node: Not yet. The guys of
Distributed OSGi have been working on this, but I've not seen
anything concrete.
I like the concept of Camel, but recently decided to go for the (lighter) Event Admin as my requirements are limited. +1 to Mike on the Camel motivation. I'd look into it and then compare options before deciding.
Aren't you looking for an ESB? ServiceMix is a:
flexible, open-source integration container that unifies the features and functionality of Apache ActiveMQ, Camel, CXF, ODE, Karaf into a powerful runtime platform you can use to build your own integrations solutions. It provides a complete, enterprise ready ESB exclusively powered by OSGi.
iPOJO Event Admin Handlers are a nicer-to-use way to access the Event Admin service mentioned by #maasg.
looks like you are talking about an ESB here. If its the case, then you might have look at wso2 ESB. It is powered by apache synapse. it uses OSGi as the modular framework, so that you can add/remove features according to your requirement. There is a whole product stack from wso2 like message brokers, Business process servers (ODE), etc based on the same OSGi core platform.
disclaimer : I work for wso2.
I was browsing for an open source messaging software and after some good bit of research I came across these three products. I've taken these out for a preliminary test drive, having had them handle messages for queues and topics, and from what I've read all three of these products are good picks for an Open Source messaging solution for most companies. What I was wondering was what are the advantages that these products may have over one another? What I'm particularly interested in is messaging throughput, including persistent messaging throughput, security, scalability, reliability, support, routing capabilities, administrative options such as metrics and monitoring, and generally just how well each program runs in a large business environment.
Check out http://queues.io/
From their site:
The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.
'messaging' covers a lot of options - and there must be at least a dozen different types of technologies that could be the right answer - having built many production messaging environments, using a variety of technologies/approaches, having a better understanding your requirements would help.
are you needing subject-based subscriptions? do you need multicast delivery? do you need dynamic subscribers/listeners? would your listeners be requerying for best sources even after finding an acceptable publisher/feed?
do you need guaranteed delivery? delivery confirmation? is you publisher storing any undelivered messages, or do you need the messaging system to do that for you automagically? how often does your feed data go stale - e.g. email-ish alerts can be store-and-forward but real-time pricing data is only valid for a short interval (and then probably needs to go away rather than cause confusion)
how volatile is your network topology? are your subscribers (or publishers) expecting to live at a fixed address? or are they mobile devices? could they appear to you over more complex internetwork topologies requiring registration and possibly imposing routing restrictions? if so any idea the frequency of these topology changes?
do you only need a java interface? are any of your subscribers to be integrated into windows components (like feeds into excel)?
if you're only interested in experience comparing the similar products you named then perhaps you have already thought through these topics.
as to products, in my experience Tibco is still the leader in throughput and scalability, especially in a real-time environment. ibm MQ would be next, especially in a store-and-forward architecture. with both of those products you get a level of support on which you can justify betting a fundamental part of your business systems. there's a reason both of those have been around for a couple of decades.
another often overlooked option is Tuxedo - it provides not only messaging but a proven transactional capability that remains unparalleled. Oracle continue to be committed to this product and, again, the level of support available is second to none.
i love open sourced solutions and am always glad to find production quality software for free - but if you are creating a fundamental part of your business infrastructure then an active community still might not indicate whether a particular voluntary project is the best bet.
my 2c worth. hope it helps.
First, I am no expert in this, but maybe I can give you some thought hints.
ActiveMQ and Qpid are both under the Apache umbrella and are message queues. But Qpid is an implementation of the AMQP specification.
AMQP is a protocol specification, on the wire level, so messages can be exchanged with other AMQP message queues (e.g RabbitMQ).
ActiveMQ and HornetQ are queues that you can use with a JMS API. The Java Message Service is a specification on an API level.
But you have the option to access Qpid via a JMS API, too.
I think performance is a secondary thought. To have an active community is more important.
http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
Benchmark includes some performance numbers for you to decide, with both persistent and transient results.