I am trying to send json with string delimited one
for e.g
"carrierId":"XXX",
"eventProcessedTS":1618261255871,
"eventTS":1618261255871,
"deviceId":"XXX",
"event": "{\"deviceDetailsDataModel\":{\"eventDateTime\":1618261255871,\"vin\":\"XXXX\",\"deviceTimeZone\":\"XXXX\",\"deviceId\":\"XXXXX\"}}"
}
but when I look at the kafka consumer
I am receiving the message as
"carrierId":"XXX",
"eventProcessedTS":1618261255871,
"eventTS":1618261255871,
"deviceId":"XXX",
"event": "{"deviceDetailsDataModel":{"eventDateTime":1618261255871,"vin":"XXXX","deviceTimeZone":"XXXX","deviceId":"XXXXX"}}"
}```
without string delimited.
I have created this as JsonObject and convert event object alone to string using event.toString() in my JSR223sampler but Still, it's not working. is it a limitation? I am using kafkaclient 2.7.0
It's not very clear which "Jmeter kafka producer" you're using, out of box JMeter doesn't provide any Kafka load testing capabilities, if you're using a plugin like Pepper-Box - Kafka Load Generator or kafkameter you should reach out to JMeter plugins developers and/or maintainers with this question.
Also be aware that you can always implement Kafka producer using JSR223 Sampler and Groovy code, example code would be something like:
import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.kafka.clients.producer.ProducerRecord
def props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
def producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("your-topic", "your_JSON_here");
producer.close();
This way you will get confidence that JMeter sending your JSON "as is" without any eventual transformations introduced by the plugins
More information: Apache Kafka - How to Load Test with JMeter
Related
I have requirement where need to have a Spring Boot Rest Service that a client application will call every 30 minutes and service is to return
number of latest messages based on the number specified in query param e.g. http://messages.com/getNewMessages?number=10 in this case should return 10 messages
number of messages based on the number and offset specified in query param e.g. http://messages.com/getSpecificMessages?number=5&start=123 in this case should return 5 messages starting offset 123.
I have simple standalone application and it works fine. Here is what I tested and would lke some direction of incorporating it in the service.
public static void main(String[] args) {
// create kafka consumer
Properties properties = new Properties();
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "my-first-consumer-group");
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, args[0]);
Consumer<String, String> consumer = new KafkaConsumer<>(properties);
// subscribe to topic
consumer.subscribe(Collections.singleton("test"));
consumer.poll(0);
//get to specific offset and get specified number of messages
for (TopicPartition partition : consumer.assignment())
consumer.seek(partition, args[1]);
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(5000));
System.out.println("Total Record Count ******* : " + records.count());
for (ConsumerRecord<String, String> record : records) {
System.out.println("Message: " + record.value());
System.out.println("Message offset: " + record.offset());
System.out.println("Message: " + record.timestamp());
Date date = new Date(record.timestamp());
Format format = new SimpleDateFormat("yyyy MM dd HH:mm:ss.SSS");
System.out.println("Message date: " + format.format(date));
}
consumer.commitSync();
As my consumer will be on-demand wondering in Spring Boot Service how I can achieve this. Where do I specify the properties if I put in application.properties those get's injected at startup time but how do i control MAX_POLL_RECORDS_CONFIG at runtime. Any help appreciated.
MAX_POLL_RECORDS_CONFIG only impact your kafka-client return the records to your spring service, it will never reduce the bytes that the consumer poll from kafka-server
see the above picture, no matter your start offset = 150 or 190, kafka server will return the whole data from (offset=110, offset=190), kafka server even didn't know how many records return to consumer, he only know the byte size = (220 - 110)
so i think you can control the record number by yourself,currently it is controlled by the kafka client jar, they are both occupy your jvm local memory
The answer to your question is here and the answer with code example is this answer.
Both written by the excellent Gary Russell, the main or one of the main person behind Spring Kafka.
TL;DR:
If you want to arbitrarily rewind the partitions at runtime, have your
listener implement ConsumerSeekAware and grab a reference to the
ConsumerSeekCallback.
Question
I need to inject messages in IBM MQ with JMeter.
Format should be RFH2. I need to set Format field to empty and also change content of the header.
Which kind of JMeter object should I used to do that?
Can you help me please? Thanks a lot.
Update n°1
Thank to #DmitriT answer I'm able to send message in queue.
However, it seems that header content is not put in the header but before the message. Please find below an example:
Server logs with message sent with MQ Visual Edit
Header
2020-04-21 11:07:59.913 DEBUG 48093 --- [DefaultMessageListenerContainer-2]
c.b.i.c.listeners.AbstractAgiListener : Receive message on MQ with header : {someargs,
jms_destination=queue:///myqueue, someargs, Sender=mysender, someargs, jms_type=mcd://jms_byte,
someargs}
Message
<Document ...>...</Document>
Server logs with message sent with JMeter
Header
2020-04-21 11:07:59.913 DEBUG 48093 --- [DefaultMessageListenerContainer-2]
c.b.i.c.listeners.AbstractAgiListener : Receive message on MQ with header : {someargs}
Message
RFH ¨ÿÿÿþ ¸ <mcd><Msd>jms_bytes</Msd></mcd> 8<jms><Dst>queue:///myqueue</Dst>
<Pri>0</Pri></jms> <usr><Sender>mysender</Sender></usr><Document ...>...</Document>
Any idea how to solve it please? Thank you.
The "JMeter object" you should use is JSR223 Sampler
Download the relevant version of the com.ibm.mq.allclient library (with dependencies) and drop it to JMeter Classpath
Restart JMeter to pick the libraries up
Add JSR223 Sampler to your Test Plan
Create the message according to your requirements and publish it to the queue. Reference code:
import com.ibm.mq.MQAsyncStatus
import com.ibm.mq.MQMessage
import com.ibm.mq.MQPutMessageOptions
import com.ibm.mq.MQQueueManager
import com.ibm.mq.constants.CMQC
import com.ibm.mq.constants.MQConstants
import com.ibm.mq.headers.MQRFH2
def mqProps = new Hashtable<String, Object>()
mqProps.put(MQConstants.CHANNEL_PROPERTY, 'DEV.APP.SVRCONN')
mqProps.put(MQConstants.PORT_PROPERTY, 1414)
mqProps.put(MQConstants.HOST_NAME_PROPERTY, '192.168.99.100')
def qManager = 'QM1'
def queueName = 'DEV.QUEUE.1'
def qMgr = new MQQueueManager(qManager, mqProps)
def openOptions = MQConstants.MQOO_OUTPUT | MQConstants.MQOO_INPUT_AS_Q_DEF
def queue = qMgr.accessQueue(queueName, openOptions)
def pmo = new MQPutMessageOptions()
pmo.options = MQConstants.MQPMO_ASYNC_RESPONSE
def message = new MQMessage()
def rfh2 = new MQRFH2()
rfh2.setEncoding(CMQC.MQENC_NATIVE)
rfh2.setCodedCharSetId(CMQC.MQCCSI_INHERIT)
rfh2.setFormat(CMQC.MQFMT_STRING)
rfh2.setNameValueCCSID(1208)
rfh2.setFieldValue('your', 'data', 'here')
rfh2.write(message)
queue.put(message, pmo)
queue.close()
MQAsyncStatus asyncStatus = qMgr.getAsyncStatus()
log.info('Successfully published: ' + asyncStatus.putSuccessCount + ' message(s)')
References:
MQRFH2 class
Handling IBM MQ message headers with IBM MQ classes for Java
IBM MQ testing with JMeter - Learn How
I am using kafka streams to consume JSON string from one topic, process and generate response to be stored in another topic. However the message that needs to be produced to the response topic needs to be in avro format.
I have tried using key as string serde and value as SpecificAvroSerde
Following is my Code to create Topology:
StreamsBuilder builder = new StreamsBuilder();
KStream<Object, Object> consumerStream =builder.stream(kafkaConfiguration.getConsumerTopic());
consumerStream = consumerStream.map(getKeyValueMapper(keyValueMapperClassName));
consumerStream.to(kafkaConfiguration.getProducerTopic());
Following is my config
if (schemaRegistry != null && schemaRegistry.length > 0) {
streamsConfig.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, String.join(",", schemaRegistry));
}
streamsConfig.put(this.keySerializerKeyName, StringSerde.class);
streamsConfig.put(this.valueSerialzerKeyName, SpecificAvroSerde.class);
streamsConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
streamsConfig.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
streamsConfig.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, batchSize);
streamsConfig.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, FailOnInvalidTimestamp.class);
streamsConfig.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, processingGuarantee);
streamsConfig.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, Integer.parseInt(commitIntervalMs));
streamsConfig.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, numberOfThreads);
streamsConfig.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, replicationFactor);
streamsConfig.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, DeserializationExceptionHandler.class);
streamsConfig.put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG, ProductionExceptionHandler.class);
streamsConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION,StreamsConfig.OPTIMIZE);
streamsConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionMode);
streamsConfig.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, maxPollRecords);
I am seeing the following error when I try with the example:
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
Problem is with the Key Value Serdes. You should use the correct serdes while consuming the stream and same for while publishing the stream.
In case if your input is JSON and you want to publish as Avro, you can do it as following:
Properties streamsConfig= new Properties();
streamsConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, SpecificAvroSerde.class);
StreamsBuilder builder = new StreamsBuilder();
KStream<Object, Object> consumerStream =builder.stream(kafkaConfiguration.getConsumerTopic(),Consumed.with(Serdes.String(), Serdes.String()));
// Replace AvroObjectClass with your avro object type
KStream<String,AvroObjectClass> consumerAvroStream = consumerStream.map(getKeyValueMapper(keyValueMapperClassName));
consumerAvroStream.to(kafkaConfiguration.getProducerTopic());
How to create a instance of Message with multiple headers using Spring Integration MessageBuilder class
Found the below information but it is only for single header
http://docs.spring.io/autorepo/docs/spring-integration/3.0.0.M3/reference/html/messaging-construction-chapter.html
Please use the current documentation - your link is not even to a released version, but a milestone of the 3.0.0 release. The current version is 4.2.0.RELEASE.
You can add as many headers as you want...
Message<String> message4 = MessageBuilder.withPayload("test4")
.setHeader("foo", 123)
.setHeader("bar", 456)
.build();
You can also use createMessage method. Pass MessageHeaders as second argument.
Map<String,Object> headers = new HashMap<>();
headers.put("foo", 123);
headers.put("bar", 456);
MessageBuilder.createMessage(payload, new MessageHeaders(headers));
everyone. I have an HTTP API for posting messages in a RabbitMQ broker and I need to implement the request-response pattern in order to receive the responses from the server. So I am something like a bridge between the clients and the server. I push the messages to the broker with specific routing-key and there is a Consumer for that messages, which is publishing back massages as response and my API must consume the response for every request. So the diagram is something like this:
So what I do is the following- For every HTTP session I create a temporary responseQueue(which is bound to the default exchange, with routing key the name of that queue), after that I set the replyTo header of the message to be the name of the response queue(where I will wait for the response) and also set the template replyQueue to that queue. Here is my code:
public void sendMessage(AbstractEvent objectToSend, final String routingKey) {
final Queue responseQueue = rabbitAdmin.declareQueue();
byte[] messageAsBytes = null;
try {
messageAsBytes = new ObjectMapper().writeValueAsBytes(objectToSend);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
MessageProperties properties = new MessageProperties();
properties.setHeader("ContentType", MessageBodyFormat.JSON);
properties.setReplyTo(responseQueue.getName());
requestTemplate.setReplyQueue(responseQueue);
Message message = new Message(messageAsBytes, properties);
Message receivedMessage = (Message)requestTemplate.convertSendAndReceive(routingKey, message);
}
So what is the problem: The message is sent, after that it is consumed by the Consumer and its response is correctly sent to the right queue, but for some reason it is not taken back in the convertSendAndReceived method and after the set timeout my receivedMessage is null. So I tried to do several things- I started to inspect the spring code(by the way it's a real nightmare to do that) and saw that is I don't declare the response queue it creates a temporal for me, and the replyTo header is set to the name of the queue(the same what I do). The result was the same- the receivedMessage is still null. After that I decided to use another template which uses the default exchange, because the responseQueue is bound to that exchange:
requestTemplate.send(routingKey, message);
Message receivedMessage = receivingTemplate.receive(responseQueue.getName());
The result was the same- the responseMessage is still null.
The versions of the amqp and rabbit are respectively 1.2.1 and 1.2.0. So I am sure that I miss something, but I don't know what is it, so if someone can help me I would be extremely grateful.
1> It's strange that RabbitTemplate uses doSendAndReceiveWithFixed if you provide the requestTemplate.setReplyQueue(responseQueue). Looks like it is false in your explanation.
2> To make it worked with fixed ReplyQueue you should configure a reply ListenerContainer:
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory);
container.setQueues(responseQueue);
container.setMessageListener(requestTemplate);
3> But the most important part here is around correlation. The RabbitTemplate.sendAndReceive populates correlationId message property, but the consumer side has to get deal with it, too: it's not enough just to send reply to the responseQueue, the reply message should has the same correlationId property. See here: how to send response from consumer to producer to the particular request using Spring AMQP?
BTW there is no reason to populate the Message manually: You can just simply support Jackson2JsonMessageConverter to the RabbitTemplate and it will convert your objectToSend to the JSON bytes automatically with appropriate headers.