Related
I am exploring ActiveMQ for advanced messaging between heterogeneous applications based on different technologies - C, Java, Ruby and Python. While looking at supported protocols I am stumbled to understand the use case of mixing protocols while performing message exchange. I had searched ActiveMQ documentation but unable to find any such reference talking about this.
My question is, say -
Producer (NewsPublisher) is publishing news(Sports, Finance, World) to a topic (NewsTopic) using AMQP. After publishing, this topic is storing these news under respective queues (Sports, Finance and World Queues) In this situation, a client subscribed to Sports queue is JMS based, another client subscribed to Finance queue is Stomp based; can these clients will be able to receive message available on the queue which was published using AMQP by NewPublisher?
I see a somewhat related question posted earlier however found answers unrelated to original question so thought to double check.
Mixing protocols in ActiveMQ is not all that hard, the broker takes care of all the internal routing and converting of messages from the incoming protocol to the outgoing bit so you don't have to worry to much about that.
What you do have to focus on is the common denominator of message types that your client mix allows you to use. In you question you've listed three different protocols AMQP, OpenWire, and STOMP (I'm guessing the JMS you are referring to is the ActiveMQ JMS client which uses OpenWire).
In this mix STOMP is the one to start with as it offers the least amount of options for message payload (Text and Binary) so you need to start there. Can you messages be restrained to those two domains? To send to STOMP from a JMS client then you can restrict yourself to a JMS TextMessage and or a JMS BytesMessage. In AMQP you will also need to restrict the message payload then to either a Text based payload for binary (think Data section containing a Binary wrapping an array of bytes).
ActiveMQ's STOMP protocol handler does offer some options to do Message transformations from other type such as converting a MapMessage into a JSON based string payload but it best to start simple and work your way up.
Given the breadth of the subject matter there a lot more that could be said, but this should help get you started. In short, yes you can mix clients on different protocols just fine in ActiveMQ but you do need to have some understanding of the limitations in doing so.
I'm having a play around with websockets and I'm having a bit of trouble wrapping my head around some stuff. Specifically, being able to send a whole bunch of subscribers different data without using a stupid amount of resources.
For example, if you had some sort of twitter like service, how would you send all followers of a person a newly posted tweet that they have made (and do the same for the other hundreds of people doing the same). It just seems that handling that many separate people is a bit absurd.
Can someone talk me through how you would go about treating each client individually? Please tell me if I have the whole idea of websockets wrong.
Thanks in advance!
P.S. for reference, I'm probably going to play around using either node or clojure (with aleph)
Use an established messaging protocol and broker on top of websockets.
It seems you are looking at websockets at the application layer when it is more of a network protocol. A variety of messaging APIs exists (such as JMS) with open source message brokers that are designed to do the complex and scalable message routing.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
or do Server-Sent Events and WebSocket replace Comet techniques?
I will approach this answer from both a terminology and historical perspective.
As I wrote in this other answer, we can use one of the several umbrella terms to refer to the set of technologies available to asynchronously send events from a web server to a web client (and vice versa). The "Push Technology" term has been used for fifteen years (for a short history of Push Technology you can see this old white paper I wrote many years ago—full disclosure: I am the creator of Lightstreamer). Now, the "Web Streaming" term is gaining consensus among the IT analysts (see Gartner, "Cool Vendors in Application and Integration Platforms, 2012", by Massimo Pezzini and Jess Thompson, 11 April 2012).
The important aspect is that we are talking about Web-based communication, that is, leveraging Web protocols. There are tons of messaging protocols and technologies that are not Web based (most of MOMs, for example) and we do not consider them as part of Push Technology (or Web Streaming).
That being said, you can distinguish between two sub-categories of Push Technology (or Web Streaming):
HTTP based
WebSockets based
Both HTTP and WebSockets are Web protocols.
If you explode the HTTP-based push mechanisms, you can identify:
HTTP Streaming
HTTP Long Polling
HTTP Polling
Traditionally, the "Comet" term (coined in 2006 by Alex Russell) has been referring to both HTTP Streaming and HTTP Polling. But consider that the first implementations of HTTP Streaming go back to 2000, well before the Comet term was coined (examples are Pushlets and Lightstreamer).
Now WebSockets make it simpler to implement Web Streaming, especially for the "backward" channel (messages sent from the browser to the server). For a more detailed explanation on the peculiarities of the backward channel over HTTP, see the final part of this article I wrote for CometDaily: http://cometdaily.com/2011/07/06/push-technology-comet-and-websockets-10-years-of-history-from-lightstreamers-perspective/
As pointed out by Phil, Comet is still necessary and will probably be for some more years, as there are not only old browsers around (including IE9, which does not support WebSockets…) but also infinite pieces of network intermediaries that do not speak WS. For example, we have seen that some mobile carriers in some countries (for example Vodafone Italy) support WSS but block WS. So a world without the Comet "hacks" is still far away… And let me add, on a personal note, that I've never loved the term "hack" applied to Comet (or, from a more correct historical point of view, applied to HTTP Streaming and HTTP Long Polling). Having worked on these techniques for 12 years now, I can say we have been able to refine them so much that they have become a full-blown technology themselves, completely reliable and used every day in many critical production scenarios (in finance, aerospace, and military, to name a few industries).
Now, let's imagine a world where WebSockets are universally supported and Comet is no more necessary. What do you get exactly? Well, just a bi-directional transport, nothing more... On the top of it you need to build everything: a messaging protocol (perhaps based on pub/sub), a server-side interface to talk to your server code, and a good set of optimization techniques and algorithms to manage the data flow, including bandwidth management, data conflation, automatic throttling, delta delivery, etc.
The good thing is that both the messaging protocols and the optimization mechanisms have already been implemented by good Comet solutions. So, extending former Comet servers to support WebSocket is the natural evolution that all of us vendors have implemented.
So, in a nutshell, in a not-so-near future WebSockets might make Comet transports obsolete, but will need to suck in all the higher layers already implemented and well tested on traditional Comet servers.
Comet is a set of technology principles/communication patterns that are typically implemented using HTTP long-poll. It enables a server to send data to the browser on demand (i.e. server push). Current comet implementations require some complex Javascript on the client side and support from the server-side (for long-held requests).
Server-Sent Events is a standard (HTML5) browser API for enabling this sort of on demand server push. You can think of Server-Sent Events as taking what has been done with complex Javascript and pushing it down into the browser itself.
WebSockets allows a browser to establish a persistent full-duplex/bi-directional connection to a server with WebSocket support. It does not require the client to keep making periodic HTTP requests to the server in order to maintain the connection as with AJAX/long-poll. Once the connection is established the overhead per message is very low (a few bytes) compared to the overhead with normal HTTP/HTTP long-poll. You can use WebSockets for efficient server push, but this is just one application.
There are also libraries that build on the AJAX/comet/WebSockets transport layer to provide things like session management, channels, broadcast, pubsub, etc. CometD is an example of this. Another popular example is Socket.IO. Both support WebSockets if it is available for the underlying transport but also support standard AJAX/long-poll if WebSockets is not available.
I initially thought that WebSockets realise Comet. They’re not an alternative. However, after some discussion I was later corrected and convinced by Alex Russell, the creator of "Comet", that this was not the case.
Comet, as #kanaka says, is a set of principles for simulating bi-directional communication between client and server (server push is half of the solution and is now provided by Server-Sent Events and the Event Source API).
However, Comet solutions are hacks because they work inconsistently across web browsers. For that reason Alex Russell says:
Next, are Web Sockets a form of Comet? Or is Comet just the HTTP hacks? I'm gonna go for the latter definition. The phrase and the hacks should probably ride off into the sunset together. I, for one, welcome our non-HTTP realtime overlords. To the extent that we can forget about old browsers (and god know I'm doing my bit: http://google.com/chromeframe), we can all get on board with "Web Sockets" and the need for any particular umbrella goes away.
So lets focus on that: how do we get users into a shiny new browser car? What sort of offer can we make to them about the richness and real-time experiences that an app based on WebSockets can deliver? Comet is about the past. Lets make the future real.
I'm now with Alex on this one. However, Comet - HTTP solutions - aren't going to become obsolete until:
Browser support is 100% and we don't need fallbacks for < IE10. I don't believe that Firefox, Safari, and Opera users are going to be a problem. There may be a small percentage of users that ignore the auto-update prompts for browsers like Firefox but not many.
Anti-virus manufacturers (such as Avast!) start support HTML5 web technologies and stop their software interfering with connectivity.
Some Internet infrastructure is updated to ensure the support of WebSockets. In my experiences connected over WSS over port 443 (a secure WebSocket connection) normally means connections make their way through Firewalls and Proxies but we want WS over port 80 to always be supported too.
While WebSocket does provide at the most fundamental level a way to communicate bi-directionally between a client and a server within the context of the Web and HTTP, it is a simple form of communications.
Comet provides more functionality on top of WebSocket (in fact, cometd even supports websocket), for features:
like publish/subscribe and channels of communications
fallback to older client<->server communications techniques when websocket is unvailable
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
We'd be interested to hear any experiences with the pros and cons of ActiveMQ vs RabbitMQ vs ZeroMQ. Information about any other interesting message queues is also welcome.
Edit: My initial answer had a strong focus on AMQP. I decided to rewrite it to offer a wider view on the topic.
These 3 messaging technologies have different approaches on building distributed systems :
RabbitMQ is one of the leading implementation of the AMQP protocol (along with Apache Qpid). Therefore, it implements a broker architecture, meaning that messages are queued on a central node before being sent to clients. This approach makes RabbitMQ very easy to use and deploy, because advanced scenarios like routing, load balancing or persistent message queuing are supported in just a few lines of code. However, it also makes it less scalable and “slower” because the central node adds latency and message envelopes are quite big.
ZeroMq is a very lightweight messaging system specially designed for high throughput/low latency scenarios like the one you can find in the financial world. Zmq supports many advanced messaging scenarios but contrary to RabbitMQ, you’ll have to implement most of them yourself by combining various pieces of the framework (e.g : sockets and devices). Zmq is very flexible but you’ll have to study the 80 pages or so of the guide (which I recommend reading for anybody writing distributed system, even if you don’t use Zmq) before being able to do anything more complicated than sending messages between 2 peers.
ActiveMQ is in the middle ground. Like Zmq, it can be deployed with both broker and P2P topologies. Like RabbitMQ, it’s easier to implement advanced scenarios but usually at the cost of raw performance. It’s the Swiss army knife of messaging :-).
Finally, all 3 products:
have client apis for the most common languages (C++, Java, .Net, Python, Php, Ruby, …)
have strong documentation
are actively supported
Why did you miss Sparrow, Starling, Kestrel, Amazon SQS, Beanstalkd, Kafka, IronMQ ?
Message Queue Servers
Message queue servers are available in various languages, Erlang (RabbitMQ), C (beanstalkd), Ruby (Starling or Sparrow), Scala (Kestrel, Kafka) or Java (ActiveMQ). A short overview can be found here
Sparrow
written by Alex MacCaw
Sparrow is a lightweight queue written in Ruby that “speaks memcache”
Starling
written by Blaine Cook at Twitter
Starling is a Message Queue Server based on MemCached
written in Ruby
stores jobs in memory (message queue)
documentation: some good tutorials, for example the railscast about starling and workling or this blog post about starling
Kestrel
written by Robey Pointer
Starling clone written in Scala (a port of Starling from Ruby to Scala)
Queues are stored in memory, but logged on disk
RabbitMQ
RabbitMQ is a Message Queue Server in Erlang
stores jobs in memory (message queue)
Apache ActiveMQ
ActiveMQ is an open source message broker in Java
Beanstalkd
written by Philotic, Inc. to improve the response time of a Facebook application
in-memory workqueue service mostly written in C
Docu: http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue
Amazon SQS
Amazon Simple Queue Service
Kafka
Written at LinkedIn in Scala
Used by LinkedIn to offload processing of all page and other views
Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)
Supports both on-line as off-line processing
ZMQ
The socket library that acts as a concurrency framework
Faster than TCP, for clustered products and supercomputing
Carries messages across inproc, IPC, TCP, and multicast
Connect N-to-N via fanout, pubsub, pipeline, request-reply
Asynch I/O for scalable multicore message-passing apps
EagleMQ
EagleMQ is an open source, high-performance and lightweight queue manager.
Written in C
Stores all data in memory and support persistence.
It has its own protocol. Supports work with queues, routes and channels.
IronMQ
IronMQ
Written in Go
Fully managed queue service
Available both as cloud version and on-premise
I hope that this will be helpful for us.
source
More information than you would want to know:
http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
UPDATE
Just elaborating what Paul added in comment. The page mentioned above is dead after 2010, so read with a pinch of salt. Lot of stuff has been been changed in 3 years.
It really depends on your use-case.
Comparing 0MQ with ActiveMQ or RabbitMQ is not fair.
ActiveMQ and RabbitMQ are Messaging Systems wich require installation and administration.
They offer featurewise a lot more than ZeroMQ. They have real persistent Queues, Support for transactions etc.
ZeroMQ is a lightweight message orientated socket implementation. It is also suitable for in-process asynchronous programming. It is possible to run a "Enterprise Messaging System" over ZeroMQ, but you would have to implement a lot on your own.
So:
ActiveMQ, RabbitMQ, Websphere MQ & MSMQ are "Enterprise Message Queues"
ZeroMQ is a message orientated IPC Library.
There's a comparison between RabbitMQ and ActiveMQ here. Out of the box, ActiveMQ is configured to guarantee message delivery - which can give the impression its slow compared to less reliable messaging systems. You can always change the configuration for performance if you wish and get at least as good performance as any other messaging system. At least you have that option. There's a lot of information on the forums and the ActiveMQ FAQ for configuration for scaling, performance and high availability. Also, ActiveMQ will support AMQP 1.0 when the spec is finalized, together with other wire formats, like STOMP.
Another plus for ActiveMQ is its an Apache project, so there is diversity in the developer community - and its not tied to one company.
I have not used ActiveMQ or RabbitMQ but have used ZeroMQ. The big difference as I see it between ZeroMQ and ActiveMQ etc. is that 0MQ is brokerless and does not have built in reliabilty for message delivery. If you are looking for an easy to use messaging API supporting many messaging patterns,transports, platforms and language bindings then 0MQ is definitely worth a look. If you are looking for a full blown messaging platform then 0MQ may not fit the bill.
See www.zeromq.org/docs:cookbook for plenty examples of how 0MQ can be used.
I an successfully using 0MQ for message passing in an electricity usage monitoring application (see http://rwscott.co.uk/2010/06/14/currentcost-envi-cc128-part-1/)
I'm using zeroMQ. I wanted a simple message passing system and I don't need the complication of a broker. I also don't want a huge Java oriented enterprise system.
If you want a fast, simple system and you need to support multiple languages (I use C and .net) then I'd recommend looking at 0MQ.
I can only add my 2 cents about ActiveMQ but since this is one of the most popular:
The language you want to write in might be important. Although ActiveMQ does have a client for most, their C# implementation is far from complete compared to the Java Library.
This means some basic functionality is flaky (fail-over protocol that ... well ... fails in some cases, no redelivery support) and other simply isn't there. Since .NET doesn't seem to be all that important to the project, development is rather slowish and there doesn't seem to be any release plan. Trunk is often broken so if you do consider this, you might want to consider contributing to the project if you want things to go ahead.
Then there is ActiveMQ itself which has a lot of nice features but some very strange issues aswell. We use the Fuse (Progress) version of activemq for stability reasons but even then there are a couple of weird "bugs" that you want to keep in mind:
Brokers that stop sending messages in some occasions
Journal Errors making the queue show messages that are not there anymore (they don't get delivered to the consumer but still)
Priority is still not implemented (is on the Issues list since the start of human kind)
etc. etc.
All and all, it is a pretty nice product IF you can live with its issues:
A) are not afraid to actively get involved when using .NET
B) develop in java ;-)
ZeroMQ is really with zero queues! It is a really mistake! It does not hav queues, topics, persistence, nothing! It is only a middleware for sockets API. If it is what you are looking cool! otherwise forget it! it is not like activeMQ or rabbitmq.
There is a comparison of the features and performance of RabbitMQ ActiveMQ and QPID given at
http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/
Personally I have tried all the above three. RabbitMQ is the best performance wise according to me, but it does not have failover and recovery options. ActiveMQ has the most features, but is slower.
Update :
HornetQ is also an option you can look into, it is JMS Complaint, a better option than ActiveMQ if you are looking for a JMS based solution.
I wrote about my initial experience regarding AMQP, Qpid and ZeroMQ here: http://ron.shoutboot.com/2010/09/25/is-ampq-for-you/
My subjective opinion is that AMQP is fine if you really need the persistent messaging facilities and is not too concerned that the broker may be a bottleneck. Also, C++ client is currently missing for AMQP (Qpid didn't win my support; not sure about the ActiveMQ client however), but maybe work in progress. ZeroMQ may be the way otherwise.
I've used ActiveMQ in a production environment for about 3 years now. While it gets the job done, lining up versions of the client libraries that work properly and are bug free can be an issue. Were currently looking to transition to RabbitMQ.
There is some discussion in the comments of this blog post, about Twitter writing their own message queue, which may be interesting.
Steve did extensive load and stress
testing of ActiveMQ, RabbitMQ, etc.
ActiveMQ is actually quite slow (much
slower than Kestrel), RabbitMQ
consistently crashes with too many
producers and too few consumers.
You probably won't have Twitter-like load initially however :)
Few applications have as many tuning configurations as ActiveMQ. Some features that make ActiveMQ stand out are:
Configurable Prefetch size.
Configurable threading.
Configurable failover.
Configurable administrative notification to producers.
...
details at:
http://activemq.net/blog
http://activemq.apache.org
If you are also interested in commercial implementations, you should take a look at Nirvana from my-channels.
Nirvana is used heavily within the Financial Services industry for large scale low-latency trading and price distribution platforms.
There is support for a wide range of client programming languages across the enterprise, web and mobile domains.
The clustering capabilities are extremely advanced and worth a look if transparent HA or load balancing is important for you.
Nirvana is free to download for development purposes.
Abie, it all comes down to your use case. Rather than relying on someone else's account of their use case, feel free to post your use case to the rabbitmq-discuss list. Asking on twitter will get you some responses too. Best wishes, alexis
About ZeroMQ aka 0MQ, as you might already know, it's the one that will get you the most messages per seconds (they were about 4 millions per sec on their ref server last time I checked), but as you might also already know, the documentation is non existent. You will have a hard time finding how to start the server(s), let alone how to use them. I guess that's partly why no one contributed about 0MQ yet.
Have fun!
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
Can anyone recommend a tool for quickly posting test messages onto a JMS queue?
Description:
The tool should allow the user to enter some data, perhaps an XML
payload, and then submit it to a queue.
I should be able to test consumer without producer.
This answer doesn't apply to all JMS brokers, but if you happen to be using Apache ActiveMQ, the web-based admin console (by default at http://localhost:8161/admin) allows you to manually send text messages to topics or queues. It's handy for debugging.
HermesJMS seems to be a rather powerful client for interacting with JMS providers. In my opinion, it is pretty unintuitive and hard to set up, though. (At least I'm mostly failing at it...)
Other, more user-friendly clients are often vendor-specific. Sonic Message Manager is a very nice and simple-to-use open-source JMS client for SonicMQ. It would be great to have a client like that working with different providers.
The ActiveMQ's web-based admin console has a big deficiency - one cannot specify any headers / custom properties when posting a message.
I came across a neat FOSS tool that can post a message and also specify headers/properties:
http://sourceforge.net/projects/activemqbrowser/
HTH
Apache JMeter is a tool (written for the Java platform) which allows:
sending messages to a queue ( point to point)
publishing/subscribing to a topic
sending both persistent and non persistent messages
sending text , map and object messages
Apache ActiveMQ includes a ProducerTool and a ConsumerTool example sources (Java) with many command-line configuration options. As it is based on the JMS API, using it with other message brokers should be easy with minor modifications.
IBM provide a free, powerful command line tool called perfharness.
Although aimed at benchmarking JMS providers, it's really good at generating (and consuming) test messages. You can use data either generated randomly or taken from a file.
The power features include sending and consuming messages at a fixed rate, using a specific number of threads, using either JMS or native MQ, etc. It generates statistics telling you exactly how fast your queue is performing (hence the name).
The only down side is that it's not super intuitive, given the number of operations it supports.
I recommend the approach of #Will and using the Web Console of ActiveMQ which lets you post messages and browse queues or delete messages easily.
Another approach I often use is to use a directory of files as sample data and use a Camel route to move the messages from the directory to a JMS queue - or to take them from a queue and save them to disk etc
e.g.
from("file://someDirectory").
to("activemq:MyQueue");
This would move all the files from someDirectory and send them to an ActiveMQ queue called MyQueue. If you'd rather leave the files in place you can use the URI "file://someDirectory?noop=true".
For more details see
the file endpoint in Camel
a sample Camel example routing from files to JMS
the various enterprise integration patterns Camel supports
Also if the JMS broker supports JMX like ActiveMQ does you can use JConsole to post message and do a lot more.
ActiveMQ has a web console for sending test messages (like mentioned above), but if your provider doesn't have this, it might be easiest to just write a console app/web page to post test messages. Sending a message in JMS isn't too hard, you might get the most benefit just writing your own test client.
If you can use Spring in Java, it has some really powerful utilities, check out the JmsTemplate.
I'm not aware of a simple client. I remember looking for one a long time ago when I researched different queue systems and trying JMS I couldn't find one then, and I couldn't find one now. One thing though - there are a ton of tutorials that get you started and you could do a simple form to achieve that.
Sorry to be not more helpful.
I have built a GUI tool for administering Open Source JMS Servers (Currently Activemq and Hornetq). It can send and receive messages and most of the usual stuff, as well as aggregate queues and topics into logical "groups".
Its a commercial product but the BETA is free and is fully functional.
try it out at http://www.rockeyesoftware.com/
For ActiveMQ the examples directory holds scripts. For Rubyists, look at example/ruby/stompcat.rb and catstomp.rb for subscribing and publishing.
I'm a brazilian developer and I made a Java program for Post HTTP and JMS Messages his available for download at: https://sites.google.com/site/felipeglino/softwares/posttool
In thath page you can found english instructions.