I am trying to setup a HA amqp client. There are currently 3 amqps brokers. Currently my client config is as below :
<property name="remoteURI" value="amqps://node1:9551?jms.username=XXXXXXXX&jms.password=XXXXXXXXX&transport.trustStoreLocation=etc/keystore.jks" />
Now since I have 2 other AMQP brokers too, im trying to connect to them too. Firstly is it possible ? According to documentation, for python I can try something like :
connection = qpid.messaging.Connection.establish("node1", reconnect=True, reconnect_urls=["node1", "node2", "node3"])
But for JMS related connection, it states :
connectionfactory.qpidConnectionfactory = amqp://guest:guest#clientid/test?brokerlist='tcp://localhost:5672'&failover='failover_exchange'
But I dont see any indication on how to connect to other brokers.
Any idea how this can be achieved from client side ?
My assumption is that you are using the QpidJMS AMQP v1.0 based client from the Apache Qpid project since you've not given any other information to make a better guess. The way that client would handle failover configuration is on the connection URI, something like:
failover://(amqp://host1:5672,amqp://host2:5672)?jms.username....
You can of course find these things out from reading the documentation.
Related
I've seen brokers like RabbitMQ or Apache Pulsar provide a Websocket API to connect directly your browser to the broker.
AFAIK, I've not seen the same for Apache Kafka. You have to implement yourself an intermediary Websocket server.
Why Confluent, which owns Apache Kafka development, does not provide an out-of-the-box websocket API like Rabbitmq or Pulsar ?
Confluent does not own Apache Kafka development. Apache Kafka is a project owned by the Apache Software Foundation (ASF).
There are several examples of how to use WebSockets with Apache Kafka:
https://www.confluent.io/blog/data-stream-processing-with-kafka-streams-bitrock-and-confluent/
https://dev.to/victorgil/kafka-websockets-angular-event-driven-microservices-all-the-way-to-the-frontend-12aa
https://medium.com/swlh/angular-spring-boot-kafka-how-to-stream-realtime-data-the-reactive-way-510a0f1e5881
Your question seems somewhat a rhetorical one, but if you actually would like to see Websockets as part of Apache Kafka then the first step would be to raise a Kafka Improvement Proposal (KIP) for discussion in the community. There is a dev mailing list where you can get further guidance on the process on contributing code.
There is already an open source connecting Kafka with Websocket. https://github.com/b/kafka-websocket
If you have requirements connecting to Kafka from a browser, I suggest to consider to browserfiy this http://github.com/confluentinc/kafka-rest-node
There is a kafka connector to Ably hosted on the Confluent Hub. Ably is essentially a serverless WebSocket option (with pub/sub and message queues). You will also find a kafka rule on Ably's website.
I am trying to find a way to consume messages that being sent by a websocket to a kafka topic (the messages are sent by the websocket to the address 'ws://address:port/topic_name' and I want to add all of those messages to a kafka topic).
I read about kafka connect and tried to find a way to do it with it but it doesnt seem to work...
thanks in advance :)
There is no Kafka Connector to a socket in Confluent Platform.
I work in a team that use Kafka in production and our source is a socket, so your options are to use platforms that support this socket->Kafka producing, or write one by yourself.
About possible platforms, I think most of them will be overkill though you can utilize them for this problem, some options are:
1. NiFi or MiniFi for smaller loads, use PublishKafka Processor
2. StreamSets with Kafka Producer Destination
3. Apache Flume- not very recommended, this project is stops to evolve.
If you wish to write your own producer, you basically have to create a listener on this port, and produce the incoming messages to Kafka; if this is a web socket, just get the payload of the requests and produce them to Kafka.
Example Kafka Producer Code can be copied from tutorialspoint simple producer example*
Here are some open-source projects examples:
1. https://github.com/DataReply/kafka-connect-socket-source
2. https://github.com/kafka-socket/miniature_engine
3. https://github.com/dhanuka84/kafka-connect-tcp
4. https://github.com/krux/tcp-stream-kafka-producer
The idea of Kafka connect is that you have some sort of external integration that serves as storage. This can be SAP, Salesforce, RDBMS, MQ or anything else that has state. You websocket endpoint does not have data, you can not poll it it is someone else that is invoking it and there fore the data is transfered. Now if you know who is actualy holding the data than you can potentialy build a conector using this guide. https://docs.confluent.io/current/connect/devguide.html
For your particular case, the best you can do is either to use Kafka Producer API https://docs.confluent.io/current/clients/producer.html
and from your websocket enpoint use this producer to post a message to the topic, or even better if you are using spring you can use a higher level abstraction, that will be KafkaTemplate https://docs.spring.io/spring-kafka/reference/html/#sending-messages.
Full disclosure: I work for MigratoryData.
You can check out MigratoryData's solution for Kafka. MigratoryData is a scalable WebSocket server. The MigratoryData Source/Sink Connector for Kafka makes use of Kafka Connect API and can be used to stream data in real-time from Kafka to WebSocket clients and vice versa. The main advantage of the solution is it extends Kafka messaging to WebSocket clients while preserving Kafka's key features like guaranteed delivery, message ordering, etc.
Does Apache Active MQ provide Server-To-Server-Connection as IBM Websphere MQ does?
Background:
We currently have Websphere MQ 7.0.1 (CUST) installed to communicate with another remote Websphere MQ Server (GOV), which is not under our control.
The MQs hold a Server-To-Server-Connection via VPN-tunnel. We would like to replace our Websphere MQ with Apache ActiveMQ.
We have one QM-Manager with 4 queues and 3 channels.
Queues:
OUT (type remote): for sending messages to GOV
QOUT (local type transmission) that is used for OUT
IMP (local): used for receiving from GOV
DLQ: (local): dead letter queue
Channels:
conch: server connection
CUST_GOV: sender
GOV_CUST: receiver
I read a lot about bridging ActiveMQ to Websphere MQ (Client API, Resource Adapter, OSGi) with Camel.
Looks pretty easy with a Camel route, but this is at least a Client-Server setup.
E. G. https://www.shaishavparekh.com/2016/01/wmq-to-amq-bridge/
I need something that hides away the IBM proprietary channel stuff.
Now I found this entry: https://serverfault.com/questions/366743/apache-activemq-server-to-server-connection
As this is from 2012 I would like to ask, if anyone successful switched from Websphere MQ to ActiveMQ resp. is it still true that I can only use Client-Server?
ActiveMQ does not support a server-to-server connection to WebsphereMQ. As you mentioned, Camel would probably be the simplest way to get a connection between the two brokers.
To be clear, even if ActiveMQ did implement a bridge with WebsphereMQ it would almost certainly use the WebsphereMQ client so would fundamentally still be a client-server implementation. It's not really clear from your question why you want to avoid this.
I have a 3rd party system pumping data into HornetQ using JMS. I need to replace HornetQ by Kafka but I cannot change the 3rd party system. What is the correct way to get the data into kafka.
I googled around and found JMS-Client and kafka connect. After reading both documentation I'm confused and not sure which one is the right one.
Has anyone any experience with this and can give me some hints on how to do this?
The right way is to use the JMS-Client because it's an implementation of the JMS API specification but with the Kafka wire-protocol. It means that you can use this client in your 3rd party system and using Kafka instead of HornetQ on the other side. It means that at least you need to add this dependency to the 3rd party system in order to use this JMS implementation for Kafka instead of the HornetQ one.
Use the Kafka JMS Client when you want to replace a JMS Broker with Apache Kafka
Use the Kafka JMS Connector when you want to integrate Kafka with a legacy JMS broker and send messages between the two different systems.
I have a jms server running on weblogic and I need another application running on another server (weblogic as well) to listen to JMS topics sent by the JMS server mentioned before. The fact is that I don't know how to do that. I mean, what do I need on the consumer application side? Thansk in advance.
I know it´s a little old, but could help other people trying to achieve the same.
First you need to enable Cross-Domain Security on both domains envolved on your JMS communication. Please see specific documentation here: https://docs.oracle.com/middleware/1221/wls/SECMG/domain.htm#SECMG402
For reading a message from a JMS resource, there are a ton of examples you can search online, but basically you should rely on Weblogic´s t3 protocol. Here is a relativelly recent example using Spring Boot: Connect to remote jms queue with Spring Boot