Spring-boot batch file read and send data to Kafka - spring-boot

Question

I am having trouble sending data from CSV file to Kafka. Here is my code for writer.java for batch processing
import java.util.List;
import javax.persistence.criteria.CriteriaBuilder.In;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.kafka.core.KafkaTemplate;
import com.codenotfound.kafka.repository.*;
import java.util.*;
import com.codenotfound.kafka.Car;
import com.codenotfound.kafka.producer.Sender;
public class Writer implements ItemWriter<Car>{
private final Repository repo;
public Writer(Repository repo) {
this.repo = repo ;
}
#Override
public void write(List<? extends Car> car) throws Exception {
repo.save(car);
}
}
So instead of repo.save(car), I want this car class details to be sent to Kafka.
Here is my Car class and Repository interface respectively
#Entity
#Table(name = "Car")
public class Car {
private String make;
private String manufacturer;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private long id;
public Car() {
//super();
}
public Car(String make, String manufacturer) {
super();
this.make = make;
this.manufacturer = manufacturer;
}
public String getMake() {
return make;
}
public void setMake(String make) {
this.make = make;
}
public String getManufacturer() {
return manufacturer;
}
public void setManufacturer(String manufacturer) {
this.manufacturer = manufacturer;
}
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
#Override
public String toString() {
return "Car [make=" + make + ", manufacturer=" + manufacturer + ", id=" + id + "]";
}
}
and Repository class
public interface Repository extends CrudRepository<Car, Long>,CustomRepository {
}
My Sender file for Kafka is:
package com.codenotfound.kafka.producer;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.kafka.core.KafkaTemplate;
import com.codenotfound.kafka.Car;
public class Sender {
private static final Logger LOGGER = LoggerFactory.getLogger(Sender.class);
#Value("${topic.json}")
private String jsonTopic;
#Autowired
private KafkaTemplate<String, Car> kafkaTemplate;
public void send(Car car) {
LOGGER.info("sending car='{}'", car.toString());
kafkaTemplate.send(jsonTopic, car);
}
}
Please suggest to me a method to send data from a CSV file to my Kafka.

Answer 1

it seems you already have all the pieces of the puzzle laid out for you. What you need to do is change your ItemWriter using your Sender class, so you'd have something like this:
#Component
public class Writer implements ItemWriter<Car> {
#Value("${topic.json}")
private String jsonTopic;
#Autowired
private KafkaTemplate<String, Car> kafkaTemplate;
#Override
public void write(List<? extends Car> cars) throws Exception {
cars.forEach(car -> kafkaTemplate.send(jsonTopic, car));
}
}
and in your job configuration, you need to autowire it like this (the code below is simplified, just showing how to declare the writer in the step):
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
private Writer carWriter;
#Bean
public Step myStep() {
this.stepBuilderFactory.get("myStep")
.<Car,Car> chunk(10)
.reader(reader())
.writer(carWriter) // don't use new here
.build();
}
}