I'm using MQTT with Spring Integration.
I would like to include a contentType header in the MQTT message.
I'm writing to the MQTT broker in this way:
#Bean
public IntegrationFlow outgoingMqttMsgFlow() {
return IntegrationFlows.from("outgoingMqttMsgChannel")
.enrichHeaders(headers -> headers.header(MqttHeaders.TOPIC, mqttTopic))
.enrichHeaders(headers -> headers.header(MessageHeaders.CONTENT_TYPE, "usp.msg", true))
.handle(new MqttPahoMessageHandler(mqttBroker, UUID.randomUUID().toString())).get();
}
When I see the MQTT messages in RabbitMQ I see this:
This is, it seems that content type header is not being included in the message (only x-mqtt-dup and x-mqtt-publish-qos).
Am I doing something wrong?
Thanks in advance.
MQTT v3 has no concept of "headers"; those headers are just the rabbitmq MQTT plugin's mechanism to handle QOS.
You would need to embed the content type in the payload somehow.
MQTT does not have headers but properties, but they represent a similar concept.
MQTT properties are supported since version v5.
If you want to be able to map Spring Integration headers into MQTT properties you have to use Mqttv5PahoMessageHandler.
There is an example in https://docs.spring.io/spring-integration/reference/html/mqtt.html
Finally, take into account that since RabbitMQ does not support MQTT v5, it neither supports MQTT properties. Use Mosquitto instead.
Related
I am trying to find a way to consume messages that being sent by a websocket to a kafka topic (the messages are sent by the websocket to the address 'ws://address:port/topic_name' and I want to add all of those messages to a kafka topic).
I read about kafka connect and tried to find a way to do it with it but it doesnt seem to work...
thanks in advance :)
There is no Kafka Connector to a socket in Confluent Platform.
I work in a team that use Kafka in production and our source is a socket, so your options are to use platforms that support this socket->Kafka producing, or write one by yourself.
About possible platforms, I think most of them will be overkill though you can utilize them for this problem, some options are:
1. NiFi or MiniFi for smaller loads, use PublishKafka Processor
2. StreamSets with Kafka Producer Destination
3. Apache Flume- not very recommended, this project is stops to evolve.
If you wish to write your own producer, you basically have to create a listener on this port, and produce the incoming messages to Kafka; if this is a web socket, just get the payload of the requests and produce them to Kafka.
Example Kafka Producer Code can be copied from tutorialspoint simple producer example*
Here are some open-source projects examples:
1. https://github.com/DataReply/kafka-connect-socket-source
2. https://github.com/kafka-socket/miniature_engine
3. https://github.com/dhanuka84/kafka-connect-tcp
4. https://github.com/krux/tcp-stream-kafka-producer
The idea of Kafka connect is that you have some sort of external integration that serves as storage. This can be SAP, Salesforce, RDBMS, MQ or anything else that has state. You websocket endpoint does not have data, you can not poll it it is someone else that is invoking it and there fore the data is transfered. Now if you know who is actualy holding the data than you can potentialy build a conector using this guide. https://docs.confluent.io/current/connect/devguide.html
For your particular case, the best you can do is either to use Kafka Producer API https://docs.confluent.io/current/clients/producer.html
and from your websocket enpoint use this producer to post a message to the topic, or even better if you are using spring you can use a higher level abstraction, that will be KafkaTemplate https://docs.spring.io/spring-kafka/reference/html/#sending-messages.
Full disclosure: I work for MigratoryData.
You can check out MigratoryData's solution for Kafka. MigratoryData is a scalable WebSocket server. The MigratoryData Source/Sink Connector for Kafka makes use of Kafka Connect API and can be used to stream data in real-time from Kafka to WebSocket clients and vice versa. The main advantage of the solution is it extends Kafka messaging to WebSocket clients while preserving Kafka's key features like guaranteed delivery, message ordering, etc.
I want to use Reactor RabbitMQ for my existing RabbitMQ project with Spring boot(https://github.com/gotidhavalh/amqpdemo).
So I want to know how can I implement it and when any AMQP message is sent to any queue then how can I receive that AMQP message just like with #RabbitListner annotation in without Reactor case.
There is currently no support for #RabbitListener with the reactive client.
I am currently trying to write an adapter which will consume messages from ActiveMQ and publish it to Kafka.
I am thinking of using spring integration to integrate these two messaging systems.
My problem is that my application will not maintain registry of the Models using which many applications will publish the records to activeMQ. I want to receive these javax jms message and want to perform some transformation like adding jmscorrelationId into kafka message.
ALso, another requirement is to send acknowledgement to active mq only when kafka send/publish is successfull.
Can ack be send back to activemq using spring integration?
Will spring integration be a good option?
Kindly note my tech architect is not in favor of using Camel/Mule. Also, he does not want to use Kafka Connect as i was planning to use Kafka connect source.
Please suggest.
The Spring Integration Kafka extension project has a sync mode for publishing, which will block the thread until Kafka confirms delivery (or throw an exception on a failure).
The JMS inbound gateway can be used to return a reply to a JMS queue.
You can add transformers (or whatever) in the flow to modify the message.
I am using Spring, Spring-Websocket, STOMP for my application, and RabbitMQ as broker. I need to log all messages going through RabbitMQ to Postgresql tables.
I know that I can write #MessageMapping in Spring and log there, but my problem is that some clients talk to RabbitMQ directly through MQTT protocol, and Spring does not support it yet (https://jira.spring.io/browse/SPR-12581). Moreover browser clients talk through Spring to RabbitMQ using STOMP protocol.
RabbitMQ allows to track all messages using Firehose tracer. How to properly listen to amq.rabbitmq.trace topic from Spring? Or do I need to write separate Java app as consumer?
The Spring AMQP is for you!
You bind some custom queue to to that amq.rabbitmq.trace with appropriate pattern (e.g. publish.#) and configure SimpleMessageListenerContainer to receive messages from that queue.
It can be done even with pretty simple config: #EnableRabbit and #RabbitListener on some POJO method. Anyway the Binding #Bean must be there to attache your queue to that exchange.
NEWBIE Question here, maybe I am way off...
When I send a JMS text message to an ActiveMQ topic using the activemq-all jar, is a JMS message actually sent over the wire, or does ActiveMQ actually map/transform the message to pure AMQP?
Thanks.
JMS is an API specification and not a wire level protocol specification. It defines a set of APIs a Messaging provider must implement to call the messaging provider a JMS provider. How the JMS APIs are implemented is internal to the implementer. ActiveMQ might(or does) use AMQP protocol under the hood to implement JMS APIs. Others, for example, IBM MQ uses it's own proprietary protocol.