Can we use Logstash in SpringBoot to sync data from RDBMS - oracle

Question

We can sync data from Oracle/any db to elastic using Logstash-Jdbc plugin. But, I can't find any way to manipulate the data which is from DB in this jdbc plugin. I want to use the Logstash/any plugin in my spring boot application to do the same by which I want to manipulate the data & column names before saving into elastic.

Answer 1

There are many Logstash input plugins and you can do basic stream processing with grok filter inside of Logstash how ever I offer to use Kafka input plugin to have do your stream processing the send you data to Logstash.
Create consumer with your Kafka broker and publish your documents with publisher class inside of your spring project then use Logstash config input to ingest your data to your index more over in this road map you have strong consumer-publisher pipeline with aid of Apache Kafka.
Find an example as follow,
<!--Kafka-->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>2.5.8.RELEASE</version>
</dependency>
After manipulating you documents and data, Create a publisher class to publish this documents
public class DriverProducer {
#Autowired
KafkaTemplate<Integer, String > kafkaTemplate;
#Autowired
ObjectMapper objectMapper;
public void messenger(Object convey) throws JsonProcessingException {
String message=objectMapper.writeValueAsString(convey);
ListenableFuture<SendResult<Integer,String>> listenableFuture=kafkaTemplate.sendDefault(null, message);
listenableFuture.addCallback(new ListenableFutureCallback<SendResult<Integer, String>>() {
#Override
public void onFailure(Throwable throwable) {
failHandler(null, message, throwable);
}
#Override
public void onSuccess(SendResult<Integer, String> result) {
successHandler(result);
}
});
}
private void failHandler(Integer key, String message, Throwable throwable){
//log.error("Unable to send the message for following Error :"+throwable.getMessage());
try {
throw throwable;
}
catch (Throwable anotherThrowable){
//log.error("**Supreme Error on throwing the throwable**"+anotherThrowable.getMessage());
}
}
private void successHandler (SendResult<Integer,String> result){
//log.info("Message sent successfully :"+ result);
}
#AllArgsConstructor
#NoArgsConstructor
#Getter
#Setter
public static class Convey{
private SagaSequence sequence;
private Integer key;
private Date date;
}
And your Logstash config file may look like this
input {
kafka{
group_id => "35834"
topics => ["Second-Topic"]
bootstrap_servers => "localhost:9092"
codec => json
}
}
filter {
}
output {
file {
path => "/SOMEPATH"
}
elasticsearch {
hosts => ["localhost:9200"]
document_type => "_doc"
index => "logger"
}
stdout { codec => rubydebug
}
}