My application works fine for the below code when start my application in debug mode but when I run my application it never goes into the loop. Even though the current timestamp becomes greater than ts2 it never goes into the loop.
#KafkaListener( id ="consumerContainer", topics = "topic1", groupId = "Consumer_Test", containerFactory = "kafkaListenerContainerFactory")
public void consume(ConsumerRecord<String, String> record, Acknowledgment acknowledgment) throws Exception {
Long ts1 = record.timestamp();
Long ts2 = ts1 + TimeUnit.MINUTES.toMillis(5);
Long currentDateTime = System.currentTimeMillis();
int b3 = currentDateTime.compareTo(ts2);
if (b3 > 0) {
this.recordPublisher.publish(record.value());
acknowledgment.acknowledge();
System.out.println("record received is "+record.key());
}
}
Also my factory config is :
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
factory.getContainerProperties().setSyncCommits(true);
Related
I've set up a spring-reactive server and an use a while loop inside of a Scheduled job to make fire and forget requests from a httpClient (backend just returns the same simple string via tomcat) it is configured as follows:
public ConnectionProvider getConnectionProvider(){
int maxConnections = 20;
ConnectionProvider connProvider = ConnectionProvider
.builder("webclient-conn-pool")
.maxConnections(maxConnections)
.maxIdleTime(Duration.of(20, ChronoUnit.SECONDS))
.maxLifeTime(Duration.of(1000, ChronoUnit.SECONDS))
.pendingAcquireMaxCount(2000)
.pendingAcquireTimeout(Duration.ofMillis(20000))
.build();
return connProvider;
}
public HttpClient getHttpClient(){
return HttpClient
.create(getConnectionProvider());
//.secure(sslContextSpec -> sslContextSpec.sslContext(webClientSslHelper.getSslContext()))
/*.tcpConfiguration(tcpClient -> {
LoopResources loop = LoopResources.create("webclient-event-loop",
selectorThreadCount, workerThreadCount, Boolean.TRUE);
return tcpClient
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.option(ChannelOption.TCP_NODELAY, true);*/
}
I've also created a job that uses #Autowired kafkaTemplate and makes a while loop that sends a short text string
For http / tomcat calls I am getting around 65 requests a second
For kafka I am maxing out around 50000 requests with a half second lag
application.props
spring.kafka.bootstrap-servers=PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093
host.name=localhost
Job to make calls
#Async
#Scheduled(fixedDelay = 15000)
public void scheduleTaskUsingCronExpression() {
generateCalls();
}
private void generateCalls() {
try{
int i = 0;
System.out.println("start");
long startTime = System.currentTimeMillis();
while(i <= 5000){
//Thread.sleep(5);
String message = "Test Message sadg sad-";
kafkaTemplate.send(TOPIC, message + i);
i++;
}
long endTime = System.currentTimeMillis();
System.out.println((endTime - startTime));
System.out.println("done");
}
catch(Exception e){
e.printStackTrace();
}
System.out.println("RUNNING");
}
Kakfa partition config
#Bean
public KafkaAdmin kafkaAdmin() {
//String bootstrapAddress = "localhost:29092";
String bootstrapAddress = "localhost:9092";
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
#Bean
public NewTopic testTopic() {
return new NewTopic("test-topic", 6, (short) 1);
}
Kafka consumer consuming messge
#KafkaListener(topics = "test-topic", groupId = "one", concurrency = "6" )
public void listenGroupFoo(String message) {
if(message.indexOf("-0") != -1){
startTime = new Date().getTime();
System.out.println("Starting Message in group foo: " + message);
}
else if(message.indexOf("-100000") != -1){
endTime = new Date().getTime();
System.out.println("Received Message in group foo: " + message);
System.out.println(endTime - startTime);
}
}
For hardware I have a 10900k with 64gb ram
5ghz clock speed
970 Evo single nvme disk
10 core 20 thread
All requests are from the same local server to the same local server
I can add more local servers for testing if needed
Is there a better way to organize / optimize the code to make a massive number of requests?
Theories:
Multiple Threads?
Changing configurations of servers such as tomcat configs (receiving or sending side)?
Not use the kafkaTemplate that is autowired or creating multiple?
Modify Hardware to have multiple disks?
Improve server receiving http to allow more connections?
Not use a job to make the requests?
Anything else anyone can think of to help?
I'm trying to use the idle between polls mentioned here to slow down the consumption rate, i also use the max.poll.interval.ms to double the idle between polls, but its always triggering partition rebalance, any idea what is the problem?
[Edit]
I have 5 hosts and i'm setting concurrency level to 1
[Edit 2]
I was setting the idle between polls to 5 min and max.poll.interval.ms to 10 min i also noticed this log "About to close the idle connection from 105 due to being idle for 540012 millis".
I decreased the idle between polls to 10 sec and the issue disappeared, any idea why?
private ConsumerFactory<String, GenericRecord> dlqConsumerFactory() {
Map<String, Object> configurationProperties = commonConfigs();
DlqConfiguration dlqConfiguration = kafkaProperties.getConsumer().getDlq();
final Integer idleBetweenPollInterval = dlqConfiguration.getIdleBetweenPollInterval()
.orElse(DLQ_POLL_INTERVAL);
final Integer maxPollInterval = idleBetweenPollInterval * 2; // two times the idleBetweenPoll, to prevent re-balancing
logger.info("Setting max poll interval to {} for DLQ", maxPollInterval);
overrideIfRequired(DQL_CONSUMER_CONFIGURATION, configurationProperties, ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, maxPollInterval);
dlqConfiguration.getMaxPollRecords().ifPresent(maxPollRecords ->
overrideIfRequired(DQL_CONSUMER_CONFIGURATION, configurationProperties, ConsumerConfig.MAX_POLL_RECORDS_CONFIG, maxPollRecords)
);
return new DefaultKafkaConsumerFactory<>(configurationProperties);
}
<time to process last polled records> + <idle between polls> must be less than max.poll.interval.ms.
EDIT
There is logic in the container to make sure we never exceed the max poll interval:
idleBetweenPolls = Math.min(idleBetweenPolls,
this.maxPollInterval - (System.currentTimeMillis() - this.lastPoll)
- 5000); // NOSONAR - less by five seconds to avoid race condition with rebalance
I can't reproduce the issue with this...
#SpringBootApplication
public class So63411124Application {
public static void main(String[] args) {
SpringApplication.run(So63411124Application.class, args);
}
#KafkaListener(id = "so63411124", topics = "so63411124")
public void listen(String in) {
System.out.println(in);
}
#Bean
public ApplicationRunner runner(ConcurrentKafkaListenerContainerFactory<?, ?> factory,
KafkaTemplate<String, String> template) {
factory.getContainerProperties().setIdleBetweenPolls(300000L);
return args -> {
while (true) {
template.send("so63411124", "foo");
Thread.sleep(295000);
}
};
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so63411124").partitions(1).replicas(1).build();
}
}
logging.level.org.springframework.kafka=debug
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.properties.max.poll.interval.ms=600000
If you can provide a small example like this that exhibits the behavior you describe, I will take a look to see what's wrong.
I am trying to seek offset from a SQL database in my kafka listener method .
I have used registerSeekCallback method in my code but this method gets invoked when we run the consumer (or container is started) . Let's say my consumer is running and last committed offset is 20 in MySql database. I manually change the last committed offset in Mysql database to 11 but my consumer will keep reading from 21 unless i restart my consumer(container restarted) . I am looking out for any option if i can override or seek offset in my listener method itself. Any help would be appreciated.
public class Listen implements ConsumerSeekAware
{
#Override
public void registerSeekCallback(ConsumerSeekCallback callback)
{
// fetching offset from a database
Integer offset = offsetService.getOffset();
callback.seek("topic-name",0,offset);
}
#KafkaListener(topics = "topic-name", groupId = "group")
public void listen(ConsumerRecord record Acknowledgment acknowledgment) throws Exception
{
// processing the record
acknowledgment.acknowledge(); //manually commiting the record
// committing the offset to MySQL database
}
}
Editing with new listener method :-
#KafkaListener(topics = "topic-name", groupId = "group")
public void listen(ConsumerRecord record Acknowledgment acknowledgment,
#Header(KafkaHeaders.CONSUMER) Consumer<?, ?> consumer)) throws Exception {
// seeking old offset stored in database (which is 11 )
consumer.seek(partition,offsetService.getOffset());
log.info("record offset is {} and value is {}" , record.offset(),record.value() );
acknowledgment.acknowledge();
}
In database my last committed offset is 11 and last committed offset on kafka end is 21. When i wrote a new record in kafka topic(i.e on offset 22) , my consumer triggers and processes 22 offset first then it goes back to seek offset 11 & start processing from there.
why is it consuming offset 22 first although i am seeking offset 11 ?
With my above code , every time i write a new message to my kafka top it processes that record first then it seeks the offset present in my database . Is there any way i can avoid that ?
There are several techniques in this answer.
Bear in mind that performing a seek on the consumer will not take effect until the next poll (any records fetched on the last poll will be sent to the consumer first).
EDIT
Here's an example:
#SpringBootApplication
public class So63429201Application {
public static void main(String[] args) {
SpringApplication.run(So63429201Application.class, args).close();
}
#Bean
public ApplicationRunner runner(KafkaTemplate<String, String> template, Listener listener) {
return args -> {
IntStream.range(0, 10).forEach(i -> template.send("so63429201", i % 3, null, "foo" + i));
Thread.sleep(8000);
listener.seekToTime(System.currentTimeMillis() - 11000);
Thread.sleep(8000);
listener.seekToOffset(new TopicPartition("so63429201", 0), 11);
Thread.sleep(8000);
};
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so63429201").partitions(3).replicas(1).build();
}
}
#Component
class Listener extends AbstractConsumerSeekAware {
#KafkaListener(id = "so63429201", topics = "so63429201", concurrency = "2")
public void listen(String in) {
System.out.println(in);
}
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
System.out.println(assignments);
super.onPartitionsAssigned(assignments, callback);
callback.seekToBeginning(assignments.keySet());
}
public void seekToTime(long time) {
getSeekCallbacks().forEach((tp, callback) -> callback.seekToTimestamp(tp.topic(), tp.partition(), time));
}
public void seekToOffset(TopicPartition tp, long offset) {
getSeekCallbackFor(tp).seek(tp.topic(), tp.partition(), offset);
}
}
Starting with spring kafka version 2.5.5, we can apply an initial offset to all assigned partitions:
#KafkaListener( groupId = "group_json", containerFactory = "userKafkaListenerFactory", topicPartitions =
{#org.springframework.kafka.annotation.TopicPartition(topic = "Kafka_Topic", partitions = {"0"},
partitionOffsets = #PartitionOffset(partition = "*", initialOffset = "3")),
#org.springframework.kafka.annotation.TopicPartition(topic = "Kafka_Topic_2", partitions = {"0"},
partitionOffsets = #PartitionOffset(partition = "*", initialOffset = "4"))
})
public void consumeJson(User user, ConsumerRecord<?, ?> consumerRecord, Acknowledgment acknowledgment) throws Exception {
/*
Reading the message into a String variable.
*/
String message = consumerRecord.value().toString();
}
Source: https://docs.spring.io/spring-kafka/docs/2.5.5.RELEASE/reference/html/#reference
We are trying to follow this blog and understand deferredresult . https://www.linkedin.com/pulse/building-async-non-blocking-microservices-using-spring-patnaik/
In the blog, the person has given a normal blocking and non blocking code. I have copied it here.
The person ran jmeter to concurrently test 1000 threads for 5 minutes. He mentioned that latency and tps were different for non blocking with the code. When I try to see it using jmeter with same details, i see the same latency for both non blocking and blocking.
I have already tried decreasing and increasing the times etc.
public SpringBootAppController() {
timer = new Timer();
ses = new ScheduledThreadPoolExecutor(10);
}
#RequestMapping("/blockingprocess")
public String blockingProcessing(#RequestParam(value="processingtime") long processingtime) throws InterruptedException
{
long startTime = System.currentTimeMillis();
Thread.sleep(processingtime);
//add more processing later
long endTime = System.currentTimeMillis();
long timeTaken = endTime-startTime;
return "SUCCESS. Blocking process completed in " + timeTaken + " Ms";
}
#RequestMapping("/nonblockingprocess")
public DeferredResult<String> nonBlockingProcessing(#RequestParam(value="processingtime") long processingtime) throws InterruptedException
{
DeferredResult<String> deferredResult = new DeferredResult<String>();
NewProcess j = new NewProcess(deferredResult, processingtime);
ses.schedule(j,processingtime, TimeUnit.MILLISECONDS);
System.out.println("hello");
return deferredResult;
}
Another class.
public NewProcess(DeferredResult<String> deferredresult, long processingtime)
{
this.deferredresult = deferredresult;
this.processingtime = processingtime;
}
#Override
public void run()
{
String result = "SUCCESS non blocking process completed in " + processingtime + " Ms";
deferredresult.setResult(result);
}
Expected is difference and better performance for non blocking compared to blocking.
I created a spring boot application that sends messages to a Kafka topic. I am using spring spring-integration-kafka:
A KafkaProducerMessageHandler<String,String> is subscribed to a channel (SubscribableChannel) and pushes all messages received to one topic.
The application works fine. I see messages arriving in Kafka via console consumer (local kafka).
I also create an Integrationtest that uses KafkaEmbedded. I am checking the expected messages by subscribing to the channel within the test - all is fine.
But i want the test to check also the messages put into kafka. Sadly Kafka's JavaDoc is not the best. What i tried so far is:
#ClassRule
public static KafkaEmbedded kafkaEmbedded = new KafkaEmbedded(1, true, "myTopic");
//...
#Before
public void init() throws Exception {
mockConsumer = new MockConsumer<>( OffsetResetStrategy.EARLIEST );
kafkaEmbedded.consumeFromAnEmbeddedTopic( mockConsumer,"sikom" );
}
//...
#Test
public void endToEnd() throws Exception {
// ...
ConsumerRecords<String, String> records = mockConsumer.poll( 10000 );
StreamSupport.stream(records.spliterator(), false).forEach( record -> log.debug( "record: " + record.value() ) );
}
The problem is that i don't see any records. I am not sure if my KafkaEmbedded setup is correct.
But messages are receive by the channel.
This works for me. Give it a try
#RunWith(SpringRunner.class)
#SpringBootTest
public class KafkaEmbeddedTest {
private static String SENDER_TOPIC = "testTopic";
#ClassRule
// By default it creates two partitions.
public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, SENDER_TOPIC);
#Test
public void testSend() throws InterruptedException, ExecutionException {
Map<String, Object> senderProps = KafkaTestUtils.producerProps(embeddedKafka);
//If you wish to send it to partitions other than 0 and 1,
//then you need to specify number of paritions in the declaration
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);
producer.send(new ProducerRecord<>(SENDER_TOPIC, 0, 0, "message00")).get();
producer.send(new ProducerRecord<>(SENDER_TOPIC, 0, 1, "message01")).get();
producer.send(new ProducerRecord<>(SENDER_TOPIC, 1, 0, "message10")).get();
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("sampleRawConsumer", "false", embeddedKafka);
// Make sure you set the offset as earliest, because by the
// time consumer starts, producer might have sent all messages
consumerProps.put("auto.offset.reset", "earliest");
final List<String> receivedMessages = Lists.newArrayList();
final CountDownLatch latch = new CountDownLatch(3);
ExecutorService executorService = Executors.newSingleThreadExecutor();
executorService.execute(() -> {
KafkaConsumer<Integer, String> kafkaConsumer = new KafkaConsumer<>(consumerProps);
kafkaConsumer.subscribe(Collections.singletonList(SENDER_TOPIC));
try {
while (true) {
ConsumerRecords<Integer, String> records = kafkaConsumer.poll(100);
records.iterator().forEachRemaining(record -> {
receivedMessages.add(record.value());
latch.countDown();
});
}
} finally {
kafkaConsumer.close();
}
});
latch.await(10, TimeUnit.SECONDS);
assertTrue(receivedMessages.containsAll(Arrays.asList("message00", "message01", "message10")));
}
}
I am using countdown latch because Producer.Send(..) is an async operation. So what i am doing here is waiting in an infinite loop polling kafka every 100 milliseconds, if there is new record and if so adding it to a List for future assertions and then reducing the countdown. And I will wait for 10 seconds in total just to be sure.
You can as well use a simple loop and then exit after a few minutes.(If you don't wish to use CountdownLatch and ExecutorService stuff)