Jackson ObjectWriter only writes first entry from stream - spring

I want to create a Spring Boot controller which creates a CSV file using data from a stream. I use Jackson CSV (jackson-dataformat-csv 2.12.1) to write the data stream from the DB to a StreamingResponseBody.
To keep it simple, I replaced the actual data from the DB with a list containing 1, 2, 3. I want a CSV file which looks like this:
1
2
3
But it only contains the first entry (1). Can someone help me to identify the problem?
Please not that I don't want to create the file somewhere on the server, I want to stream the content directly to the user.
My code looks like this:
import com.fasterxml.jackson.dataformat.csv.CsvMapper
import org.springframework.http.MediaType
import org.springframework.http.ResponseEntity
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.RequestMapping
import org.springframework.web.bind.annotation.RestController
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody
import javax.servlet.http.HttpServletResponse
#RestController
#RequestMapping(value = ["/test"], produces = [MediaType.TEXT_HTML_VALUE])
class TestController {
#GetMapping("/download", produces = [MediaType.TEXT_EVENT_STREAM_VALUE])
fun download(response: HttpServletResponse): ResponseEntity<StreamingResponseBody>? {
response.setHeader("Content-Disposition", "attachment;filename=download.csv")
response.status = HttpServletResponse.SC_OK
val mapper = CsvMapper()
val schema = mapper.schemaFor(Int::class.java)
val writer = mapper.writer(schema)
val input = listOf(1, 2, 3).stream()
val stream = StreamingResponseBody { outputStream ->
input.forEach { entity ->
writer.writeValue(outputStream, entity)
}
}
return ResponseEntity
.ok()
.header("Content-Disposition", "attachment;filename=download.csv")
.body(stream)
}
}

With the help of Andriy's comment I was able to find the cause and the solution. Jackson closes the stream when it's finished writing to it, see: ObjectMapper._writeValueAndClose.
To change this behavior you have to set JsonGenerator.Feature.AUTO_CLOSE_TARGET to false like this:
val jsonFactory = CsvFactory().configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false)
val mapper = ObjectMapper(jsonFactory)
val writer = mapper.writer(CsvMapper().schemaFor(Int::class.java))
Note: There is no AUTO_CLOSE_TARGET option for the CsvGeneratorbut using the JsonGenerator setting also works for the CsvFactory.

Didn't see the column formatter assigned while writing to the stream in your code. Can you try the following :-
CsvMapper mapper = new CsvMapper();
CsvSchema schema = mapper.schemaFor(Dymmy.class);
schema = schema.withColumnSeparator('\t');

There is a better alternative then setting AUTO_CLOSE_TARGET to false, which is to use SequenceWritter.
val stream = StreamingResponseBody { outputStream ->
val mapper = CsvMapper()
val sequenceWriter = mapper.writer(mapper.schemaFor(Int::class.java).withHeader())
.writeValues(outputStream)
input.forEach { entity ->
sequenceWriter.write(entity)
}
}

Related

aggregator spring cloud stream with timeout

I want to make an application that receives messages, stores those messages in a list, and later with and schedule releases those messages every x amount of time.
I know spring cloud stream has an aggregator that already does this, but I think I need it to be done manually because I need to keep a unique message based upon a key and only replace the old message if it matches a specific condition ( I think of it as a Set aggregator with conditions)
what I have tried so far.
also in this link https://github.com/chalimbu/AggregatorQuestionStack
Processor.
import org.springframework.cloud.stream.annotation.EnableBinding
import org.springframework.cloud.stream.annotation.Input
import org.springframework.cloud.stream.annotation.Output
import org.springframework.cloud.stream.messaging.Processor
import org.springframework.scheduling.annotation.Scheduled
#EnableBinding(Processor::class)
class SetAggregatorProcessor(val storageService: StorageService) {
#Input
public fun inputMessage(input: Map<String,Any>){
storageService.messages.add(input)
}
#Output
#Scheduled(fixedDelay = 20000)
public fun produceOutput():List<Map<String,Any>>{
val message= storageService.messages
storageService.messages.clear()
return message;
}
}
Memory storage.
import org.springframework.stereotype.Service
#Service
class StorageService {
public var messages: MutableList<Map<String,Any>> = mutableListOf()
}
This code generates the following error when I start pushing messages.
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139) ~[spring-integration-core-5.5.8.jar:5.5.8]
The idea is to deploy this app as part of the spring cloud stream (dataflow) platform.
I prefer the declarative approach(over the functional approach), but if somebody knows how to do it with the reactor way, I could settle for it.
Thanks for any help or advice.
thanks to this example(https://github.com/spring-cloud/spring-cloud-stream-samples/blob/main/processor-samples/sensor-average-reactive-kafka/src/main/java/sample/sensor/average/SensorAverageProcessorApplication.java) I was able to figure something out using flux in case someone else needs it
#Configuration
class SetAggregatorProcessor : Function<Flux<Map<String, Any>>, Flux<MutableList<Map<String, Any>>>> {
override fun apply(data: Flux<Map<String, Any>>):Flux<MutableList<Map<String, Any>>> {
return data.window(Duration.ofSeconds(20)).flatMap { window: Flux<Map<String, Any>> ->
this.aggregateList(window)
}
}
private fun aggregateList(group: Flux<Map<String, Any>>): Mono<MutableList<Map<String, Any>>>? {
return group.reduce(
mutableListOf(),
BiFunction<MutableList<Map<String, Any>>, Map<String, Any>, MutableList<Map<String, Any>>> {
acumulator: MutableList<Map<String, Any>>, element: Map<String, Any> ->
acumulator.add(element)
acumulator
}
)
}
}
update https://github.com/chalimbu/AggregatorQuestionStack/tree/main/src/main/kotlin/com/project/co/SetAggregator

How to read Spring Boot application log files into Splunk? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed last year.
This post was edited and submitted for review 10 months ago and failed to reopen the post:
Original close reason(s) were not resolved
Improve this question
I am looking to send log data from the application to Splunk. I came to know that there is nothing to do with spring, it's just Splunk needs some configurations to read Application's Logs files. I want to know how we can make Splunk read Applications Log files.
Please help me out with Splunk integration with Spring Boot. It will be great if you provided any code snippets or references.
In terms of integration, what are you after? Are you looking to bring data in from Splunk for use in your Sprint Boot application, or are you looking to send data from your application into Splunk?
For logging into Splunk, I suggest you look at the following:
https://github.com/splunk/splunk-library-javalogging
https://docs.spring.io/autorepo/docs/spring-integration-splunk/0.5.x-SNAPSHOT/reference/htmlsingle/
https://github.com/barrycommins/spring-boot-splunk-sleuth-demo
If you are looking to interact with the Splunk application and run queries against it, look at the Splunk Java SDK, https://dev.splunk.com/enterprise/docs/java/sdk-java/howtousesdkjava/
Here are the steps which I have followed to integrate Splunk successfully into my Spring Boot application:
Set up the repository in the pom.xml file by adding the following:
<repositories>
<repository>
<id>splunk-artifactory</id>
<name>Splunk Releases</name>
<url>https://splunk.jfrog.io/splunk/ext-releases-local</url>
</repository>
</repositories>
Add the maven dependency for Splunk jar, within the dependencies tags, which will download and setup the Splunk jar file in the project (In my case the jar file is splunk-1.6.5.0.jar):
<dependency>
<groupId>com.splunk</groupId>
<artifactId>splunk</artifactId>
<version>1.6.5.0</version>
</dependency>
Configure and run the Splunk query from your controller / service / main class:
package com.my.test;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.HashMap;
import java.util.Map;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.splunk.Args;
import com.splunk.HttpService;
import com.splunk.Job;
import com.splunk.SSLSecurityProtocol;
import com.splunk.Service;
#SpringBootApplication
public class Main {
public static String username = "your username";
public static String password = "your password";
public static String host = "your splunk host url like - splunk-xx-test.abc.com";
public static int port = 8089;
public static String scheme = "https";
public static Service getSplunkService() {
HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2);
Map<String, Object> connectionArgs = new HashMap<>();
connectionArgs.put("host", host);
connectionArgs.put("port", port);
connectionArgs.put("scheme", scheme);
connectionArgs.put("username", username);
connectionArgs.put("password", password);
Service splunkService = Service.connect(connectionArgs);
return splunkService;
}
/* Take the Splunk query as the argument and return the results as a JSON
string */
public static String getQueryResultsIntoJsonString(String query) throws IOException {
Service splunkService = getSplunkService();
Args queryArgs = new Args();
//set "from" time of query. 1 = from beginning
queryArgs.put("earliest_time", "1");
//set "to" time of query. now = till now
queryArgs.put("latest_time", "now");
Job job = splunkService.getJobs().create(query);
while(!job.isDone()) {
try {
Thread.sleep(500);
} catch(InterruptedException ex) {
ex.printStackTrace();
}
}
Args outputArgs = new Args();
//set format of result set as json
outputArgs.put("output_mode", "json");
//set offset of result set (how many records to skip from the beginning)
//Default is 0
outputArgs.put("offset", 0);
//set no. of records to get in the result set.
//Default is 100
//If you put 0 here then it would be set to "no limit"
//(i.e. get all records, don't truncate anything in the result set)
outputArgs.put("count", 0);
InputStream inputStream = job.getResults(outputArgs);
//Now read the InputStream of the result set line by line
//And return the final result into a JSON string
//I am using Jackson for JSON processing here,
//which is the default in Spring boot
BufferedReader in = new BufferedReader(new InputStreamReader(inputStream));
String resultString = null;
String aLine = null;
while((aLine = in.readLine()) != null) {
//Convert the line from String to JsonNode
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNode = mapper.readTree(aLine);
//Get the JsonNode with key "results"
JsonNode resultNode = jsonNode.get("results");
//Check if the resultNode is array
if (resultNode.isArray()) {
resultString = resultNode.toString();
}
}
return resultString;
}
/*Now run your Splunk query from the main method (or a RestController or a Service class)*/
public static void main(String[] args) {
try {
getQueryResultsIntoJsonString("search index=..."); //your Splunk query
} catch (IOException e) {
e.printStackTrace();
}
}
}

Apache Beam - Unable to infer a Coder on a DoFn with multiple output tags

I am trying to execute a pipeline using Apache Beam but I get an error when trying to put some output tags:
import com.google.cloud.Tuple;
import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.joda.time.Duration;
import java.lang.reflect.Type;
import java.util.Map;
import java.util.stream.Collectors;
/**
* The Transformer.
*/
class Transformer {
final static TupleTag<Map<String, String>> successfulTransformation = new TupleTag<>();
final static TupleTag<Tuple<String, String>> failedTransformation = new TupleTag<>();
/**
* The entry point of the application.
*
* #param args the input arguments
*/
public static void main(String... args) {
TransformerOptions options = PipelineOptionsFactory.fromArgs(args)
.withValidation()
.as(TransformerOptions.class);
Pipeline p = Pipeline.create(options);
p.apply("Input", PubsubIO
.readMessagesWithAttributes()
.withIdAttribute("id")
.fromTopic(options.getTopicName()))
.apply(Window.<PubsubMessage>into(FixedWindows
.of(Duration.standardSeconds(60))))
.apply("Transform",
ParDo.of(new JsonTransformer())
.withOutputTags(successfulTransformation,
TupleTagList.of(failedTransformation)));
p.run().waitUntilFinish();
}
/**
* Deserialize the input and convert it to a key-value pairs map.
*/
static class JsonTransformer extends DoFn<PubsubMessage, Map<String, String>> {
/**
* Process each element.
*
* #param c the processing context
*/
#ProcessElement
public void processElement(ProcessContext c) {
String messagePayload = new String(c.element().getPayload());
try {
Type type = new TypeToken<Map<String, String>>() {
}.getType();
Gson gson = new Gson();
Map<String, String> map = gson.fromJson(messagePayload, type);
c.output(map);
} catch (Exception e) {
LOG.error("Failed to process input {} -- adding to dead letter file", c.element(), e);
String attributes = c.element()
.getAttributeMap()
.entrySet().stream().map((entry) ->
String.format("%s -> %s\n", entry.getKey(), entry.getValue()))
.collect(Collectors.joining());
c.output(failedTransformation, Tuple.of(attributes, messagePayload));
}
}
}
}
The error shown is:
Exception in thread "main" java.lang.IllegalStateException: Unable to
return a default Coder for Transform.out1 [PCollection]. Correct one
of the following root causes: No Coder has been manually specified;
you may do so using .setCoder(). Inferring a Coder from the
CoderRegistry failed: Unable to provide a Coder for V. Building a
Coder using a registered CoderProvider failed. See suppressed
exceptions for detailed failures. Using the default output Coder from
the producing PTransform failed: Unable to provide a Coder for V.
Building a Coder using a registered CoderProvider failed.
I tried different ways to fix the issue but I think I just do not understand what is the problem. I know that these lines cause the error to happen:
.withOutputTags(successfulTransformation,TupleTagList.of(failedTransformation))
but I do not get which part of it, what part needs a specific Coder and what is "V" in the error (from "Unable to provide a Coder for V").
Why is the error happening? I also tried to look at Apache Beam's docs but they do not seems to explain such a usage nor I understand much from the section discussing about coders.
Thanks
First, I would suggest the following -- change:
final static TupleTag<Map<String, String>> successfulTransformation =
new TupleTag<>();
final static TupleTag<Tuple<String, String>> failedTransformation =
new TupleTag<>();
into this:
final static TupleTag<Map<String, String>> successfulTransformation =
new TupleTag<Map<String, String>>() {};
final static TupleTag<Tuple<String, String>> failedTransformation =
new TupleTag<Tuple<String, String>>() {};
That should help the coder inference determine the type of the side output. Also, have you properly registered a CoderProvider for Tuple?
Thanks to #Ben Chambers' answer, Kotlin is:
val successTag = object : TupleTag<MyObj>() {}
val deadLetterTag = object : TupleTag<String>() {}

How to use StanfordNLP Chinese segmentor in Java?

I have tried the following code, however the code does not work and only outputs null.
String text = "我爱北京天安门。";
StanfordCoreNLP pipeline = new StanfordCoreNLP();
Annotation annotation = pipeline.process(text);
String result = annotation.get(CoreAnnotations.ChineseSegAnnotation.class);
System.out.println(result);
The result:
...
done [0.6 sec].
Using mention detector type: rule
null
How to use StanfordNLP Chinese segmentor correctly?
Some sample code:
import edu.stanford.nlp.pipeline.*;
import edu.stanford.nlp.ling.CoreAnnotations;
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.util.StringUtils;
import java.util.*;
public class ChineseSegmenter {
public static void main (String[] args) {
// set the properties to the standard Chinese pipeline properties
Properties props = StringUtils.argsToProperties("-props", "StanfordCoreNLP-chinese.properties");
StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
String text = "...";
Annotation annotation = new Annotation(text);
pipeline.annotate(annotation);
List<CoreLabel> tokens = annotation.get(CoreAnnotations.TokensAnnotation.class);
for (CoreLabel token : tokens)
System.out.println(token);
}
}
Note: Make sure the Chinese models jar is on your CLASSPATH. That file is available here: http://stanfordnlp.github.io/CoreNLP/download.html
The above code should print out the tokens created after the Chinese segmenter is run.

How to Convert gson to LinkedHashMap<String, List<String>>?

i'm new to gson and i wonder how convert json data to LinkedHashMap<String, List<String>>
my json data is show like below:
{ "data":
{
"data1": ["asdf", "qwer"],
"data2": ["xczv", "aweqrfds123", "sfdgq234"],
"data3": ["dsafasd", "xcvr123", "sdfa324123"]
}
}
field names of json data of data are dynamic, so i want to convert json data of data to LinkedHashMap<String, List<String>>
how can i do that ?
You can use TypeToken to convert it into expected type with Gson#fromJson(Reader,Type)
As per JSON string it is LinkedHashMap<String,LinkedHashMap<String,ArrayList<String>>>
Sample code:
BufferedReader reader = new BufferedReader(new FileReader(new File("json.txt")));
Type type = new TypeToken<LinkedHashMap<String,LinkedHashMap<String,ArrayList<String>>>>() {}.getType();
LinkedHashMap<String,LinkedHashMap<String,ArrayList<String>>> data = new Gson().fromJson(reader, type);
LinkedHashMap<String,ArrayList<String>> innerMap = data.get("data");
System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(innerMap));
This is not how it works in Gson world - you can't convert JSON to any Java class you want, unless you want to do all of that manually. The common approach works as described below:
Create a Java class, which matches your JSON format, e.g. you can use a Java class generator described here: http://jsongen.byingtondesign.com/
Use GsonBuilder to read your Json from a file and to import it to the generated class
I've used that approach and the Java file that has been generated (after I've fixed a minor syntax error in your initial JSON) looks like this:
package com.json;
import java.util.List;
public class Data{
private List data1;
private List data2;
private List data3;
public List getData1(){
return this.data1;
}
public void setData1(List data1){
this.data1 = data1;
}
public List getData2(){
return this.data2;
}
public void setData2(List data2){
this.data2 = data2;
}
public List getData3(){
return this.data3;
}
public void setData3(List data3){
this.data3 = data3;
}
}
To start working with the newly created class you can use the template below:
is = new InputStreamReader(new FileInputStream(new File('<path-to-json>')), "UTF-8")/;
Gson gson = new GsonBuilder().create();
Data d = gson.fromJson(is, Data.class);
// Start using your d instance here

Resources