Is it possible to use JMeter to push messages to Apache Kafka.
How to implement producer (in JAVA) to push messages to Kafka.
Regards,
Anand
I thought there was an answer earlier, maybe not. Have you taken a look at these? I'm using the original kafkameter myself.
https://github.com/BrightTag/kafkameter
https://github.com/EugeneYushin/new-api-kafkameter
and tutorials on kafkameter:
http://www.technix.in/load-testing-apache-kafka-using-kafkameter
http://codyaray.com/2014/07/custom-jmeter-samplers-and-config-elements
For use outside of JMeter, I've found it easier to write a producer load tool in say ruby, python, or node.js than in Java. But it's personal preference. Load scalability is another matter but other languages are easier to prototype out a producer tool.
Update:
Since the original post, there's now another solution/option for JMeter:
https://github.com/GSLabDev/pepper-box
and rather than post specific tutorials about it, you're better off googling for some mix of terms like "Pepper-Box kafka jmeter" and go over the tutorial results for those as there are quite a few. The ones from BlazeMeter should be good.
Yes you can go with the Jmeter by using the above external libraries given by #David, above. Just to add, I'd recommend you to have two different programs for both consumer and producer so that you'll be in more control on what's going on. Such as optimizing and changing the property files within the config according to your requirements. Even though JMeter sounds easy on loadtesting, I'm not sure whether you'll be able to identify the efficiency of message consumption or production, such as identifying the number of messages published or consumed within a certain amount of period (ie: if you're dealing with large number of messages).
Kafka Produce Sample given in the doc and this
I was browsing for an open source messaging software and after some good bit of research I came across these three products. I've taken these out for a preliminary test drive, having had them handle messages for queues and topics, and from what I've read all three of these products are good picks for an Open Source messaging solution for most companies. What I was wondering was what are the advantages that these products may have over one another? What I'm particularly interested in is messaging throughput, including persistent messaging throughput, security, scalability, reliability, support, routing capabilities, administrative options such as metrics and monitoring, and generally just how well each program runs in a large business environment.
Check out http://queues.io/
From their site:
The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.
'messaging' covers a lot of options - and there must be at least a dozen different types of technologies that could be the right answer - having built many production messaging environments, using a variety of technologies/approaches, having a better understanding your requirements would help.
are you needing subject-based subscriptions? do you need multicast delivery? do you need dynamic subscribers/listeners? would your listeners be requerying for best sources even after finding an acceptable publisher/feed?
do you need guaranteed delivery? delivery confirmation? is you publisher storing any undelivered messages, or do you need the messaging system to do that for you automagically? how often does your feed data go stale - e.g. email-ish alerts can be store-and-forward but real-time pricing data is only valid for a short interval (and then probably needs to go away rather than cause confusion)
how volatile is your network topology? are your subscribers (or publishers) expecting to live at a fixed address? or are they mobile devices? could they appear to you over more complex internetwork topologies requiring registration and possibly imposing routing restrictions? if so any idea the frequency of these topology changes?
do you only need a java interface? are any of your subscribers to be integrated into windows components (like feeds into excel)?
if you're only interested in experience comparing the similar products you named then perhaps you have already thought through these topics.
as to products, in my experience Tibco is still the leader in throughput and scalability, especially in a real-time environment. ibm MQ would be next, especially in a store-and-forward architecture. with both of those products you get a level of support on which you can justify betting a fundamental part of your business systems. there's a reason both of those have been around for a couple of decades.
another often overlooked option is Tuxedo - it provides not only messaging but a proven transactional capability that remains unparalleled. Oracle continue to be committed to this product and, again, the level of support available is second to none.
i love open sourced solutions and am always glad to find production quality software for free - but if you are creating a fundamental part of your business infrastructure then an active community still might not indicate whether a particular voluntary project is the best bet.
my 2c worth. hope it helps.
First, I am no expert in this, but maybe I can give you some thought hints.
ActiveMQ and Qpid are both under the Apache umbrella and are message queues. But Qpid is an implementation of the AMQP specification.
AMQP is a protocol specification, on the wire level, so messages can be exchanged with other AMQP message queues (e.g RabbitMQ).
ActiveMQ and HornetQ are queues that you can use with a JMS API. The Java Message Service is a specification on an API level.
But you have the option to access Qpid via a JMS API, too.
I think performance is a secondary thought. To have an active community is more important.
http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
Benchmark includes some performance numbers for you to decide, with both persistent and transient results.
I have a chance to learn how to administer and develop for WebSphere. But after trying to find a book for it I am at a loss where to begin. I was hoping for some advice. I may have access to the IBM RAD IDE (I think that's what it is), but I am not sure.
I have a project I'd like to put on it but I am lost. Thanks.
edit: I do not care which language I end up using.
Ok. There are two parts to your question (and I'm sticking to the base application server here):
a) writing applications: WebSphere Application Server is a J2EE application server. So any web resource or forum or blog describing how to write servlets or JSPs or applications that use EJBs will work. There are lots of those. If you want to stick with IBM resources, you could look for redbooks or developerworks articles. For more general information (patterns, practices, etc) you could try http://theserverside.com.
b) administering the application server: Redbooks like this can give a pretty good overview (as the other poster mentioned).
There is also a WebSphere education channel on youtube which addresses both questions.
Download a trial MQ server from IBM (to some hosts or VMs? that won't matter if you break them...), and (optionally) add a WMB in front of it to transform some messages and fire some messages at it from a GUI client via another broker?
so you 'could':
ia92 java gui to fire messages->microbroker(or rsmb or mosquitto)->WMB(optional)->MQ
have a look at the messages arriving on MQ via Websphere explorer (or its command line client, try googling for 'websphere MQ quick reference card') have a fiddle about with its subscriptions etc
and then start consuming those messages from its queues on the 'other' side of the chain via a WAS and then a java client
MQ->MQ(bridge maybe)->WAS->some Java EE client you've written
don't forget the red books, keep it simple and then keep on adding complexity e.g. encryption etc.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
We are facing a choice to use IBM MQ over SFTP for file transfer. I've heard advantages of such approach, but I've never see anyone actually using it for a large files.
So main question: how well IMB MQ can handle transfer of large files (up to 100 MB)? Is it stable? It's from mainframe to UNIX server, if it does matter.
Thanks a lot.
I've used MQ with files up to 8GB in size, without incident. You have to allocate enough space for MQ to manage them, but it works.
MQ itself offers message-based communication between programs. You can't use it directly to transfer files. People talking about "sending files via MQ" often mean that they will write the programs themselves to read from a file, send over MQ, receive from MQ and write to another file. This sounds temptingly trivial at first but quickly becomes very complex if you want it to be robust. Not recommended.
What you should be doing if you want to involve MQ in your file transfers is use one of the file-transfer products that sit upon MQ. There are a number of mature offerings, about which I know next to nothing, together with a new product from IBM itself ("Websphere MQ File Transfer Edition") which is being actively developed and recently announced an updated version (7.0.1).
To answer your specific question, I've personally used WMQFTE to transfer a file of 1000GB from an AIX machine to a Windows one, so I don't think your "large" 100MB files will be a problem :-). I don't get involved with the mainframe as much but it is very much a core platform for FTE.
Vendors have been doing File transfer over a messaging system such as WebSphere MQ for over 12 years. I have not used the IBM product. Yet, I know until recently IBM resold Metastorm’s product (under the brand PM4Data) and only recently entered the market with a first generation product.
I’ve seen Metastorm Integration Manager handle files of terabyte sizes all of the time; breaking up the messages across WebSphere MQ and performing buffers acknowledgements so not to overflow message queues or having to do a lot of MQ Administration.
Other worthwhile considerations of using messaging systems for file transfer vs. SFTP include:
Pre-and Post-Data Movement Handling - As a network transport, SFTP moves data (as can file transfer products over WebSphere MQ). To do pre- or post-movement processing, you’d have to write your own monitoring, triggering, error handling, etc. Mature vendors in the file transfer over messaging space provide infrastructure out-of-the-box for you. Depending on your application, they can also help
Audit Trail - SFTP provides often provide system-level logging. Often time, vendors providing file integration over MQ have greater audit capabilities, including the Pre- and Post-Data Movement Handling. Some vendors also offer secure web-based systems to expose certain integrations to authorized users.
Exception Handling – SFTP will give synchronous error messages from which you then code your error logic. Vendors providing file integration over MQ often provide exception capabilities ranging from "here’s a message on queue signifying a problem happened... good luck" to built-in integration to Web-based consoles or even BPM system.
Treating File Transfer as a Service – Often times, File Transfer is part of a larger business process. Some vendors leverage this to chain file transfers and even have them collaborate with human-centric processes to provide end-to-end visibility around the context of a point-to-point file transfer. Additionally, using messaging systems for file transfer helps move an organization to a more real-time, message-driven approach.
I hope the additional details give some details in your assessment. Good luck!
In one of the previous answers, there is the followng statement:
People talking about "sending files via MQ" often mean that they will write the programs themselves to read from a file, send over MQ, receive from MQ and write to another file. This sounds temptingly trivial at first but quickly becomes very complex if you want it to be robust. Not recommended.
In theory it would be nice to "sending files via MQ" because you would receive a Confirmation on Delivery (CoD) message when the recipient got (read) the file from the queue. Another benefit would be that the file would be automatically removed from the queue when the recipient got (read) it from the queue.
Do you perhaps know whether the above (or a similar) is offered by MQFTE?
Capitalwares Universal File Mover is an option here.
Also WebSphere Message Broker for Remote Adapter Deployment and WebSphere MQ File Transfer Edition are options to consider.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
We'd be interested to hear any experiences with the pros and cons of ActiveMQ vs RabbitMQ vs ZeroMQ. Information about any other interesting message queues is also welcome.
Edit: My initial answer had a strong focus on AMQP. I decided to rewrite it to offer a wider view on the topic.
These 3 messaging technologies have different approaches on building distributed systems :
RabbitMQ is one of the leading implementation of the AMQP protocol (along with Apache Qpid). Therefore, it implements a broker architecture, meaning that messages are queued on a central node before being sent to clients. This approach makes RabbitMQ very easy to use and deploy, because advanced scenarios like routing, load balancing or persistent message queuing are supported in just a few lines of code. However, it also makes it less scalable and “slower” because the central node adds latency and message envelopes are quite big.
ZeroMq is a very lightweight messaging system specially designed for high throughput/low latency scenarios like the one you can find in the financial world. Zmq supports many advanced messaging scenarios but contrary to RabbitMQ, you’ll have to implement most of them yourself by combining various pieces of the framework (e.g : sockets and devices). Zmq is very flexible but you’ll have to study the 80 pages or so of the guide (which I recommend reading for anybody writing distributed system, even if you don’t use Zmq) before being able to do anything more complicated than sending messages between 2 peers.
ActiveMQ is in the middle ground. Like Zmq, it can be deployed with both broker and P2P topologies. Like RabbitMQ, it’s easier to implement advanced scenarios but usually at the cost of raw performance. It’s the Swiss army knife of messaging :-).
Finally, all 3 products:
have client apis for the most common languages (C++, Java, .Net, Python, Php, Ruby, …)
have strong documentation
are actively supported
Why did you miss Sparrow, Starling, Kestrel, Amazon SQS, Beanstalkd, Kafka, IronMQ ?
Message Queue Servers
Message queue servers are available in various languages, Erlang (RabbitMQ), C (beanstalkd), Ruby (Starling or Sparrow), Scala (Kestrel, Kafka) or Java (ActiveMQ). A short overview can be found here
Sparrow
written by Alex MacCaw
Sparrow is a lightweight queue written in Ruby that “speaks memcache”
Starling
written by Blaine Cook at Twitter
Starling is a Message Queue Server based on MemCached
written in Ruby
stores jobs in memory (message queue)
documentation: some good tutorials, for example the railscast about starling and workling or this blog post about starling
Kestrel
written by Robey Pointer
Starling clone written in Scala (a port of Starling from Ruby to Scala)
Queues are stored in memory, but logged on disk
RabbitMQ
RabbitMQ is a Message Queue Server in Erlang
stores jobs in memory (message queue)
Apache ActiveMQ
ActiveMQ is an open source message broker in Java
Beanstalkd
written by Philotic, Inc. to improve the response time of a Facebook application
in-memory workqueue service mostly written in C
Docu: http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue
Amazon SQS
Amazon Simple Queue Service
Kafka
Written at LinkedIn in Scala
Used by LinkedIn to offload processing of all page and other views
Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)
Supports both on-line as off-line processing
ZMQ
The socket library that acts as a concurrency framework
Faster than TCP, for clustered products and supercomputing
Carries messages across inproc, IPC, TCP, and multicast
Connect N-to-N via fanout, pubsub, pipeline, request-reply
Asynch I/O for scalable multicore message-passing apps
EagleMQ
EagleMQ is an open source, high-performance and lightweight queue manager.
Written in C
Stores all data in memory and support persistence.
It has its own protocol. Supports work with queues, routes and channels.
IronMQ
IronMQ
Written in Go
Fully managed queue service
Available both as cloud version and on-premise
I hope that this will be helpful for us.
source
More information than you would want to know:
http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
UPDATE
Just elaborating what Paul added in comment. The page mentioned above is dead after 2010, so read with a pinch of salt. Lot of stuff has been been changed in 3 years.
It really depends on your use-case.
Comparing 0MQ with ActiveMQ or RabbitMQ is not fair.
ActiveMQ and RabbitMQ are Messaging Systems wich require installation and administration.
They offer featurewise a lot more than ZeroMQ. They have real persistent Queues, Support for transactions etc.
ZeroMQ is a lightweight message orientated socket implementation. It is also suitable for in-process asynchronous programming. It is possible to run a "Enterprise Messaging System" over ZeroMQ, but you would have to implement a lot on your own.
So:
ActiveMQ, RabbitMQ, Websphere MQ & MSMQ are "Enterprise Message Queues"
ZeroMQ is a message orientated IPC Library.
There's a comparison between RabbitMQ and ActiveMQ here. Out of the box, ActiveMQ is configured to guarantee message delivery - which can give the impression its slow compared to less reliable messaging systems. You can always change the configuration for performance if you wish and get at least as good performance as any other messaging system. At least you have that option. There's a lot of information on the forums and the ActiveMQ FAQ for configuration for scaling, performance and high availability. Also, ActiveMQ will support AMQP 1.0 when the spec is finalized, together with other wire formats, like STOMP.
Another plus for ActiveMQ is its an Apache project, so there is diversity in the developer community - and its not tied to one company.
I have not used ActiveMQ or RabbitMQ but have used ZeroMQ. The big difference as I see it between ZeroMQ and ActiveMQ etc. is that 0MQ is brokerless and does not have built in reliabilty for message delivery. If you are looking for an easy to use messaging API supporting many messaging patterns,transports, platforms and language bindings then 0MQ is definitely worth a look. If you are looking for a full blown messaging platform then 0MQ may not fit the bill.
See www.zeromq.org/docs:cookbook for plenty examples of how 0MQ can be used.
I an successfully using 0MQ for message passing in an electricity usage monitoring application (see http://rwscott.co.uk/2010/06/14/currentcost-envi-cc128-part-1/)
I'm using zeroMQ. I wanted a simple message passing system and I don't need the complication of a broker. I also don't want a huge Java oriented enterprise system.
If you want a fast, simple system and you need to support multiple languages (I use C and .net) then I'd recommend looking at 0MQ.
I can only add my 2 cents about ActiveMQ but since this is one of the most popular:
The language you want to write in might be important. Although ActiveMQ does have a client for most, their C# implementation is far from complete compared to the Java Library.
This means some basic functionality is flaky (fail-over protocol that ... well ... fails in some cases, no redelivery support) and other simply isn't there. Since .NET doesn't seem to be all that important to the project, development is rather slowish and there doesn't seem to be any release plan. Trunk is often broken so if you do consider this, you might want to consider contributing to the project if you want things to go ahead.
Then there is ActiveMQ itself which has a lot of nice features but some very strange issues aswell. We use the Fuse (Progress) version of activemq for stability reasons but even then there are a couple of weird "bugs" that you want to keep in mind:
Brokers that stop sending messages in some occasions
Journal Errors making the queue show messages that are not there anymore (they don't get delivered to the consumer but still)
Priority is still not implemented (is on the Issues list since the start of human kind)
etc. etc.
All and all, it is a pretty nice product IF you can live with its issues:
A) are not afraid to actively get involved when using .NET
B) develop in java ;-)
ZeroMQ is really with zero queues! It is a really mistake! It does not hav queues, topics, persistence, nothing! It is only a middleware for sockets API. If it is what you are looking cool! otherwise forget it! it is not like activeMQ or rabbitmq.
There is a comparison of the features and performance of RabbitMQ ActiveMQ and QPID given at
http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/
Personally I have tried all the above three. RabbitMQ is the best performance wise according to me, but it does not have failover and recovery options. ActiveMQ has the most features, but is slower.
Update :
HornetQ is also an option you can look into, it is JMS Complaint, a better option than ActiveMQ if you are looking for a JMS based solution.
I wrote about my initial experience regarding AMQP, Qpid and ZeroMQ here: http://ron.shoutboot.com/2010/09/25/is-ampq-for-you/
My subjective opinion is that AMQP is fine if you really need the persistent messaging facilities and is not too concerned that the broker may be a bottleneck. Also, C++ client is currently missing for AMQP (Qpid didn't win my support; not sure about the ActiveMQ client however), but maybe work in progress. ZeroMQ may be the way otherwise.
I've used ActiveMQ in a production environment for about 3 years now. While it gets the job done, lining up versions of the client libraries that work properly and are bug free can be an issue. Were currently looking to transition to RabbitMQ.
There is some discussion in the comments of this blog post, about Twitter writing their own message queue, which may be interesting.
Steve did extensive load and stress
testing of ActiveMQ, RabbitMQ, etc.
ActiveMQ is actually quite slow (much
slower than Kestrel), RabbitMQ
consistently crashes with too many
producers and too few consumers.
You probably won't have Twitter-like load initially however :)
Few applications have as many tuning configurations as ActiveMQ. Some features that make ActiveMQ stand out are:
Configurable Prefetch size.
Configurable threading.
Configurable failover.
Configurable administrative notification to producers.
...
details at:
http://activemq.net/blog
http://activemq.apache.org
If you are also interested in commercial implementations, you should take a look at Nirvana from my-channels.
Nirvana is used heavily within the Financial Services industry for large scale low-latency trading and price distribution platforms.
There is support for a wide range of client programming languages across the enterprise, web and mobile domains.
The clustering capabilities are extremely advanced and worth a look if transparent HA or load balancing is important for you.
Nirvana is free to download for development purposes.
Abie, it all comes down to your use case. Rather than relying on someone else's account of their use case, feel free to post your use case to the rabbitmq-discuss list. Asking on twitter will get you some responses too. Best wishes, alexis
About ZeroMQ aka 0MQ, as you might already know, it's the one that will get you the most messages per seconds (they were about 4 millions per sec on their ref server last time I checked), but as you might also already know, the documentation is non existent. You will have a hard time finding how to start the server(s), let alone how to use them. I guess that's partly why no one contributed about 0MQ yet.
Have fun!