I'm in the middle of a project where we will migrate a major software system based on a larger set of custom made technologies to be based on OSGi services. For this we will likely need a some sort of message bus that plays nice with OSGi services.
Sync and ASync delivery
Point-to-point only
Guaranteed delivery - preferable with persistence via files
Strict message ordered from the same client (Async mode), but necessarily from different clients
Support for process-to-process and node-to-node nice but not strictly required
Open source solutions will be preferred, but not required.
I have looked at eventbus (as recommended in https://stackoverflow.com/a/1953453/796559), but that does not seem to work well.
So the question is, which technologies match the above?
Tonny,
Having just come from a very similar, and successful project, please let me share my experience with you to save you some time and your company some money. First and foremost, ESB's were a really good idea 8 years ago when they were proposed. And, they solved an important problem: how do you define a business problem in a way that those pesky coders will understand? The goal was to develop a system that would allow a business person to create a software solution with little or no pesky developer interaction needed that would suck up money better spent on management bonuses.
To answer this the good folks at many organizations came up with JBI, BPMN and a host of other solutions that let business folks model the business processes they wanted to "digitize". But really, they were all flawed at a very critical level: they addressed business issues, but not integration issues. As such, many of these implementations were unsuccessful unless done by some high-priced consultant, and even then your prospects were sketchy.
At the same time, some really smart folks in the very late 90's published a book called "Enterprise Integration Patterns" which identified over 60 design patterns used to solve common integration problems. Many of the folks performing ESB stuff realized that their problem wasn't one of business modelling. Rather the problem was how to integrate thier existing applications. To help solve this Michael Strachan and some really smart guys started the Apache Software Foundation Project "Camel". Camel is a strict implementation of Enterprise Integration Patterns in addition to a huge number of components designed to allow folks like you and I to hook stuff together.
So, if you think of your business process as simply a need to send data from one application to another to another, with the appropriate data transformations between, then Camel is your answer. Now, what if you want to base the "route" (a specified series of application endpoints you want to send data thorugh) off of a set of configurable rules in a database? Well, Camel can do that too! There's an endpoint for that! Anyhow, dont' do the traditional ESB, its old and busted. And Absolutely do the camel thing.
Please let me know if this helps.
The OSGi specification defines a component "Event Admin" which is a lightweight pub-sub event subsystem.
From the RFC0157:
Event Admin specifies a means for an event source to send events to
event listeners. Event sources can create events with a topic and
properties and request Event Admin to deliver the events to event
listeners which have declared interest in specific event topics and/or
property values. The event source can request synchronous (and
unordered) delivery or asynchronous (and ordered) delivery.
Compared to your requirements, it would score as follows:
Sync and ASync delivery: Check
Point-to-point only: No. Pub-Sub
Guaranteed delivery - preferable with persistence via files: NO
Strict message ordered from the same client (Async mode): YES
Support for process-to-process: if (process == OSGi service) -> Yes
Support for node-to-node: Not yet. The guys of
Distributed OSGi have been working on this, but I've not seen
anything concrete.
I like the concept of Camel, but recently decided to go for the (lighter) Event Admin as my requirements are limited. +1 to Mike on the Camel motivation. I'd look into it and then compare options before deciding.
Aren't you looking for an ESB? ServiceMix is a:
flexible, open-source integration container that unifies the features and functionality of Apache ActiveMQ, Camel, CXF, ODE, Karaf into a powerful runtime platform you can use to build your own integrations solutions. It provides a complete, enterprise ready ESB exclusively powered by OSGi.
iPOJO Event Admin Handlers are a nicer-to-use way to access the Event Admin service mentioned by #maasg.
looks like you are talking about an ESB here. If its the case, then you might have look at wso2 ESB. It is powered by apache synapse. it uses OSGi as the modular framework, so that you can add/remove features according to your requirement. There is a whole product stack from wso2 like message brokers, Business process servers (ODE), etc based on the same OSGi core platform.
disclaimer : I work for wso2.
Related
I am a programmer working in the domain of information system integration (SOA, ESB, messages broker etc...).
At the moment I am working on a proprietary ESB which unfortunately does not implement the Scatter-Gather pattern.
Actually, I would be interested in a solution to implement this pattern asynchronously. Meaning in this example, the Best Quote is not managed in the same synchronous transaction than Quote Request (e.g. not a synchronous request/reply service).
Because we are talking about asynchronous processing, I am looking for a reliable solution. As an example, if the Vendor B fails (due to a technical issue), I do not want to send back a error Quote Request. I have to consider the whole transaction as a guaranteed one and being able to reprocess the call to Vendor B at one point in time. Then the transaction would magically continue and I would be able to send back a successful Quote Request.
I have already been able to implement this pattern in the past using a proprietary Complex Event Processing (CEP) tool. Indeed the CEP tool was able to persist the global transaction state and to correlate events coming back from Vendor A, B and C.
So I was wondering if there was an existing lightweight solution to implement this pattern in a reliable way.
Last but not least I do not search another ESB tool. I know that Apache Camel, Spring Integration, Mule or WSO2 are implementing this pattern but I was rather interested in a dedicated solution.
Thanks
Your requirements fall in two broad spaces. Event correlation/filtering is usually done with a CEP engine while the integration patterns like Scatter-Gather are to be done with an ESB. It's true that a CEP engine will facilitate some level of message transformation and integration capabilities while an ESB will also support basic filtering/correlations of events (mostly event processing rather than complex event processing), but these are not the purposes they are originally designed for.
So if it can't be implemented using one of ESB or CEP servers, your solution can include both CEP and ESB servers, each performing some specific set of tasks they are best suited for. (it's unlikely that a vendor building a combined server to perform all these stuff)
Having said that, if you are going to implement this with WSO2 products, and if you really need a single server instance, you can consider installing CEP features on top of the ESB as explained in this doc. Otherwise, you can connect the two servers using a high-performant protocol like Thrift.
I am trying to develop a publish/subscribe system.
To this end, I have read some papers and articles regarding it.
And they all talk about Messaging service as an integral part of publish/subscribe system.
My question is, can I develop a publish subscribe system without using MOM like JMS?
Or am I missing or oversimplifying things?
I do not think you are oversimplifying things. There are stand-alone products available that provide advanced functionality based on publish/subscribe, without being part of a larger MOM system.
One of them is a group of products implementing the Data Distribution Service (DDS) specification, as standardized by the Object Management Group (OMG). Check out this Wikipedia entry for a very brief introduction and list of references.
DDS supports many advanced data management features like a strong-typed and content aware databus, distributed state management and historical data access. Its rich set of Quality of Service settings allows to off-load a lot of the complexity from your applications to the middleware. This is all based on the publish/subscribe paradigm.
If you would tell more about your application, then I might be able to point you to similar use cases using this technology -- if you are interested.
It depends what you mean by "MOM". If you think MOM = JMS then yes, there are plenty of pub/sub applications which are not JMS servers (off the top of my head): 0MQ, TIBCO Rendezvous and the many AMQP implementations around.
I guess my definition of MOM is an infrastructure for reliably getting a message from one system to another in an asynchronous manner. Pub/sub is a feature on top of the message transport which allows a message to be distributed to multiple other systems. Once you get beyond the point of opening a socket and stuffing a bunch of bytes down it, I would argue you are in the realm of MOM.
So, no you don't need JMS to do pub/sub....there are plenty of open-source and closed-source alternatives out there. Which one depends on your requirements and skills.
You can look at multicast that provides one to many communication. Multicast does not require MOM, instead it requires multicast enabled IP network. Usually the network routers take care of creating copies of message and delivering messages to destinations.
As per title. I don't know if this is the right place or way to ask this, admins feel free to edit/move/close the question if appropriate.
I'd like to get pointers to recent material clarifying the market trends, as well as real life examples. Even pseudo-pundit, Gartner-like stuff is OK. Thanks.
I am curious about the second part of the question. What is the basis of your statement that 'the ESB thing' appears to be fading? I don't believe it is.
The problem with ESBs however is that some vendors call their product an ESB, but it actually is much much more than that. In some companies this happened with their integration product just because Gartner or some other analysts company says that ESB is hot. Marketing strategy is changed: The product is called ESB and maybe somethings are added that are expected in an ESB.
Paul Fremantle of WSO2 wrote a very good article about what an ESB really is [1].
As for OSGi: The first company I saw using it in their middleware was WSO2. I have heard, that TIBCO, another middleware vendor, is also moving or has moved towards using it in their Active Matrix platform.
OSGi may help in various ways. The most important is that it decreases the effort of the installation of the platform. Install a minimum on each system used to deploy the application, and during deployment the components required to run the application will be added. You do not have to worry about having installed the right plug-ins, add-ons and what not. This is what both WSO2 and TIBCO are doing.
With some vendors, you see that you need to install an awful amount of software, of which you in the end may be using just a small part (e.g. IBM WebSphere). Because of this, you may have to use over-dimensioned systems, which adds extra costs.
OSGi may prevent this.
Have a look at the presentation of WSO2 about the WSO2 Carbon platform [2].
The statement at the end of the presentation says it all:
Adapt the middleware to your architecture, not the architecture to the middleware
So yes, I think OSGi has a future in enterprise apps.
[1] http://wso2.org/library/2913
[2] http://www.slideshare.net/wso2.org/the-carbon-story-presentation-855666
Disclaimer:
I am in no way affiliated with WSO2, TIBCO or IBM. I am a certified TIBCO BusinessWorks Developer and have been developing applications for the IBM WebSphere Process Server platform. Above all, I am a WSO2 Enthusiast.
I would say yes..WSO2 has proof for that..Check the following links
http://osgi.dzone.com/articles/carbon-osgi-and-soa
http://www.infoworld.com/d/developer-world/wso2-upgrades-osgi-middleware-695
I was browsing for an open source messaging software and after some good bit of research I came across these three products. I've taken these out for a preliminary test drive, having had them handle messages for queues and topics, and from what I've read all three of these products are good picks for an Open Source messaging solution for most companies. What I was wondering was what are the advantages that these products may have over one another? What I'm particularly interested in is messaging throughput, including persistent messaging throughput, security, scalability, reliability, support, routing capabilities, administrative options such as metrics and monitoring, and generally just how well each program runs in a large business environment.
Check out http://queues.io/
From their site:
The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.
'messaging' covers a lot of options - and there must be at least a dozen different types of technologies that could be the right answer - having built many production messaging environments, using a variety of technologies/approaches, having a better understanding your requirements would help.
are you needing subject-based subscriptions? do you need multicast delivery? do you need dynamic subscribers/listeners? would your listeners be requerying for best sources even after finding an acceptable publisher/feed?
do you need guaranteed delivery? delivery confirmation? is you publisher storing any undelivered messages, or do you need the messaging system to do that for you automagically? how often does your feed data go stale - e.g. email-ish alerts can be store-and-forward but real-time pricing data is only valid for a short interval (and then probably needs to go away rather than cause confusion)
how volatile is your network topology? are your subscribers (or publishers) expecting to live at a fixed address? or are they mobile devices? could they appear to you over more complex internetwork topologies requiring registration and possibly imposing routing restrictions? if so any idea the frequency of these topology changes?
do you only need a java interface? are any of your subscribers to be integrated into windows components (like feeds into excel)?
if you're only interested in experience comparing the similar products you named then perhaps you have already thought through these topics.
as to products, in my experience Tibco is still the leader in throughput and scalability, especially in a real-time environment. ibm MQ would be next, especially in a store-and-forward architecture. with both of those products you get a level of support on which you can justify betting a fundamental part of your business systems. there's a reason both of those have been around for a couple of decades.
another often overlooked option is Tuxedo - it provides not only messaging but a proven transactional capability that remains unparalleled. Oracle continue to be committed to this product and, again, the level of support available is second to none.
i love open sourced solutions and am always glad to find production quality software for free - but if you are creating a fundamental part of your business infrastructure then an active community still might not indicate whether a particular voluntary project is the best bet.
my 2c worth. hope it helps.
First, I am no expert in this, but maybe I can give you some thought hints.
ActiveMQ and Qpid are both under the Apache umbrella and are message queues. But Qpid is an implementation of the AMQP specification.
AMQP is a protocol specification, on the wire level, so messages can be exchanged with other AMQP message queues (e.g RabbitMQ).
ActiveMQ and HornetQ are queues that you can use with a JMS API. The Java Message Service is a specification on an API level.
But you have the option to access Qpid via a JMS API, too.
I think performance is a secondary thought. To have an active community is more important.
http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
Benchmark includes some performance numbers for you to decide, with both persistent and transient results.
So far I have seen non-Java client support only for open source message brokers like Apache ActiveMQ, JBoss HornetQ and Open Message Queue (OpenMQ).
Are there also closed-source products like WebSphere, WebLogic or Tibco which offer non-Java access to their MOM brokers, using a documented wire protocol (opposite to a closed-source binary client library) which allows to write clients in other languages?
This is getting more interesting as products (like WebLogic) are available in the (EC2) cloud so that developers can use the cloud instance to develop and test a client application without the need to purchase and install the full version.
I don't have a definitive answer because I specialize in WMQ exclusively. However, I believe the answer is "no" for the most part. (More on that in a minute.)
Regarding WMQ IBM makes available exit points to tailor the behavior of the channels, API calls and authorizations. Exits are very well documented and perform narrow functions within the scope of a particular action - i.e. receive a message, initiate a connection, etc. These are written in C and, more recently, Java. For the most part these are unused and customers I talk to generally cite complexity. They want something customizable through configuration and not through low-level code. I suspect other MOM vendors experience similar requirements from customers.
What does this have to do with your question? My take on this is that if customers are reluctant to code up exits with limited function, it seems far fetched that they would code up a full-featured and robust client that supports reliable message delivery, one- and two-phase commit, client-side exits, diagnostics, and all the other functionality that WMQ channels provide.
Assuming that this task was undertaken by an open-source team capable of that level of code, who would support it? the MOM vendors currently provide end-to-end support when using their proprietary clients. The notion of how a trouble ticket might be resolved when using a third-party client that is community-supported is a bit scary to many customer. For example, IBM supplies add-ons for WMQ called SupportPacs. Although there are SupportPacs that are fully supported and are considered product extensions, some of the SupportPacs are provided as-is. Many of my customers won't run as-is code even when it is supplied by the vendor.
Finally, there is the notion of the interface contract. WMQ supports a few verbs with a lot of options. The underlying channel protocol is MUCH more complex. When WMQ v7 came out, the channels had considerable new functionality and tuning. this was possible at this scale because the internals are not exposed to clients and so IBM was able to make massive changes without fear of negative impact to 3rd party clients. Exposing all of that would create dependencies on an order or two higher magnitude than exist with just the API's exposed.
So, according to my theory (I don't pretend to speak for the MQ development team here) the big MOM vendors have a vested interest in not exposing their channel protocols to independent developers. The new wrinkle here is AMQP which I alluded to above. It defines the wire protocol and allows each vendor to code a compliant product. Although this provides the opportunity you describe for open-source solutions, the ability of any one implementation to improve the product is limited by the fact that they don't own the protocol. For the time being though I don't expect you'll find any of the big MOM vendors exposing their wire protocols for 3rd party development. That said, this is just a guess and if I'm wrong, I'm sure someone here will jump in and provide the counter-example.