Performance Testing Java Http and Kafka is while loop slowing performance - spring

Question

I've set up a spring-reactive server and an use a while loop inside of a Scheduled job to make fire and forget requests from a httpClient (backend just returns the same simple string via tomcat) it is configured as follows:
public ConnectionProvider getConnectionProvider(){
int maxConnections = 20;
ConnectionProvider connProvider = ConnectionProvider
.builder("webclient-conn-pool")
.maxConnections(maxConnections)
.maxIdleTime(Duration.of(20, ChronoUnit.SECONDS))
.maxLifeTime(Duration.of(1000, ChronoUnit.SECONDS))
.pendingAcquireMaxCount(2000)
.pendingAcquireTimeout(Duration.ofMillis(20000))
.build();
return connProvider;
}
public HttpClient getHttpClient(){
return HttpClient
.create(getConnectionProvider());
//.secure(sslContextSpec -> sslContextSpec.sslContext(webClientSslHelper.getSslContext()))
/*.tcpConfiguration(tcpClient -> {
LoopResources loop = LoopResources.create("webclient-event-loop",
selectorThreadCount, workerThreadCount, Boolean.TRUE);
return tcpClient
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.option(ChannelOption.TCP_NODELAY, true);*/
}
I've also created a job that uses #Autowired kafkaTemplate and makes a while loop that sends a short text string
For http / tomcat calls I am getting around 65 requests a second
For kafka I am maxing out around 50000 requests with a half second lag
application.props
spring.kafka.bootstrap-servers=PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093
host.name=localhost
Job to make calls
#Async
#Scheduled(fixedDelay = 15000)
public void scheduleTaskUsingCronExpression() {
generateCalls();
}
private void generateCalls() {
try{
int i = 0;
System.out.println("start");
long startTime = System.currentTimeMillis();
while(i <= 5000){
//Thread.sleep(5);
String message = "Test Message sadg sad-";
kafkaTemplate.send(TOPIC, message + i);
i++;
}
long endTime = System.currentTimeMillis();
System.out.println((endTime - startTime));
System.out.println("done");
}
catch(Exception e){
e.printStackTrace();
}
System.out.println("RUNNING");
}
Kakfa partition config
#Bean
public KafkaAdmin kafkaAdmin() {
//String bootstrapAddress = "localhost:29092";
String bootstrapAddress = "localhost:9092";
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
#Bean
public NewTopic testTopic() {
return new NewTopic("test-topic", 6, (short) 1);
}
Kafka consumer consuming messge
#KafkaListener(topics = "test-topic", groupId = "one", concurrency = "6" )
public void listenGroupFoo(String message) {
if(message.indexOf("-0") != -1){
startTime = new Date().getTime();
System.out.println("Starting Message in group foo: " + message);
}
else if(message.indexOf("-100000") != -1){
endTime = new Date().getTime();
System.out.println("Received Message in group foo: " + message);
System.out.println(endTime - startTime);
}
}
For hardware I have a 10900k with 64gb ram
5ghz clock speed
970 Evo single nvme disk
10 core 20 thread
All requests are from the same local server to the same local server
I can add more local servers for testing if needed
Is there a better way to organize / optimize the code to make a massive number of requests?
Theories:
Multiple Threads?
Changing configurations of servers such as tomcat configs (receiving or sending side)?
Not use the kafkaTemplate that is autowired or creating multiple?
Modify Hardware to have multiple disks?
Improve server receiving http to allow more connections?
Not use a job to make the requests?
Anything else anyone can think of to help?