I have the following code:
File overitimeFile = new File(filePath+overtimeFileName);
FlatFileItemWriter<OvertimeSAPExport> overtimeItemWriter =
new FlatFileItemWriter<OvertimeSAPExport>();
overtimeItemWriter.setResource(new FileSystemResource(overitimeFile));
overtimeItemWriter.setShouldDeleteIfExists(true);
PassThroughLineAggregator<OvertimeSAPExport> lineAggregator =
new PassThroughLineAggregator<OvertimeSAPExport>();
overtimeItemWriter.setLineAggregator(lineAggregator);
overtimeItemWriter.open(new ExecutionContext());
List<OvertimeSAPExport> overtimeList = overtimeDAO.getSapOvertimeData(locationId, month);
overtimeItemWriter.write(overtimeList);
I have implemented the toString method for OvertimeSAPExport and when I debug I can see that it enters the toString once for each record in the list and gets te correct string from it.
It also creates the file without problems and throws no exceptions my way, but when I look at the file, it's empty.
Could someone PLEASE show me where my mistake is?
Try overtimeItemWriter.close(); and see if the file is flushed on disk. You also need to validate if a transaction is ongoing that postponed the writing.
Related
Please help
We use WebServiceTemplate.marshalSendAndReceive, where do we pass the url, pojo, and WebServiceMessageCallback
And use interceptor where
ByteArrayTransportOutputStream byteArrayTransportOutputStream = new ByteArrayTransportOutputStream();
webServiceMessage.writeTo(byteArrayTransportOutputStream);
String request = EOL + IntegrationUtils.deleteSensitiveData(new String(byteArrayTransportOutputStream.toByteArray()));
Why before writeTo webServiceMessage is saving soapRequest in formatted form in webServiceMessage.envelope.element
But after writeTo I get message in one line without any formatting
What is problem? What does it depend on?
I am reading a csv file using Spring batch. I am reading the content of csv and writing this content to the database in accordance with one of the entity class.
Now there could be certain lines in csv that are wrong and don't match the POJO attributes. To handle this I configured my Step as follows:
Step step = stepBuilderFactory.get("CSV-Step")
.<Book, Book>chunk(100)
.faultTolerant()
.skip(FlatFileParseException.class)
.skipLimit(1)
.reader(itemReader)
.writer(itemWriter)
.build();
It basically skips the line which causes FlatFileParseException and goes on with subsequent lines. Now I also want to log the lines for which parsing could not be done. For this in my GlobalExceptionHandler annotated with #ControllerAdvice I made following method:
#OnReadError
public void handleCsvParseException(FlatFileParseException ex, Throwable throwable) {
logger.error("! FlatFileParseException, line is: " + ex.getLineNumber());
logger.error("! FlatFileParseException, input is: " + ex.getInput());
logger.error("! Message: " + throwable.getMessage());
logger.error("! Cause: " + throwable.getCause());
}
The thing is that this method is not being called because i have the skip configuration in my Step. How can i ignore the unwanted lines i.e. skipping unwanted line but at the same time log information about them. I would appreciate any kind of help.
The SkipListener is what you need. This listener will be called whenever the configured skippable exception occurs during reading, processing or writing items.
In your case, you can implement the logging logic in the SkipListener#onSkipInRead(Throwable t); method. The FlatFileParseException will be passed as a parameter and will give you the necessary context (the line number and the raw input).
Hope this helps.
I have a function that preprocess a complex object before sending it to server to save some space, so it creates a copy of the object, and got this error on submission (AJAX).
It was working before I decided to create a "clean" copy of the object.
Why is this error thrown?
Found out that you can´t copy the file object, the spec doesn´t allow it, so, in my function, I had to make every file point to the original file object in order to accomplish the submission of every file:
prepareData = function(originalObject){
var data = clone originalObject;
data.id_bs = data.bs.id;
delete data.bs;
data.id_Cc= data.cc.id;
delete data.cc;
//Added this to make it work
for ( var kDoc = 0; kDoc < originalObject.docs.length; kDoc++ ){
data.docs[kDoc] = originalObject.docs[kDoc];
}
return data;
}
Regarding above subject, is there any way to get the value of a field from a pipe. And use that value outside the pipe's scope in Hadoop Cascading? The data has delimiter as '|':
first_name|description
Binod|nothing
Rohit|nothing
Ramesh|abc
From above pipe I need to get a value from the description, whatever that is 'nothing' or 'abc'.
Hadoop Cascading is developed with a concept of creating real case scenario by flowing data between pipe and executing parallely it over Map-Reduce Hadoop system.
Execution of java program is unnecessary to depend with rest of the cascading flow (from creating source tap to sink tap), and what Hadoop Cascading does is: it executes those two different processes in different independent JVM instances and they will be unable to share their values.
Following code and its output shows brief hints:
System.out.println("Before Debugging");
m_eligPipe = new Each(m_eligPipe, new Fields("first_name"), new Debug("On Middle", true));
System.out.println("After Debugging");
Expected ouput:
Before Debugging
On Middle: ['first_name']
On Middle: ['Binod']
On Middle: ['Rohit']
On Middle: ['Ramesh']
After Debugging
Actual output:
Before Debugging
After Debugging
...
...
On Middle: ['first_name']
On Middle: ['Binod']
On Middle: ['Rohit']
On Middle: ['Ramesh']
I don't understand what you are trying to say. Do you to mean to extract the value of field ${description} outside the scope of the pipe. If possible something like this in pseudo code.
str = get value of description in inputPipe (which is in the scope of the job rather than function or buffer)
I assume this is what you want: you have a pipe with one field, that is the concatenation of ${first_name} and ${description}. And you want the output to be a pipe with field that is ${description}.
If so, this is what I'd do: implement a function that extracts description and have your flow execute it.
You function (let's call it ExtractDescriptionFunction) should override method operate with something like this:
#Override
public void operate(FlowProcess flowProcess, FunctionCall<Tuple> functionCall) {
TupleEntry arguments = functionCall.getArguments();
String concatenation = arguments.getString("$input_field_name");
String[] values = concatenation.split("\\|"); // you might want to have some data sanity check here
String description = values[1];
Tuple tuple = functionCall.getContext();
tuple.set(0, description);
functionCall.getOutputCollector().add(tuple);
}
Then, in your flow definition, add this:
Pipe outputPipe = new Each(inputPipe, new ExtractDescriptionFunction());
Hope this helps.
I am using Velocity 1.7 to format string and I had some trouble with default values. Velocity by itself has no special syntax for case when value is not set and we want to use some another, default value.
By the means of Velocity it looks like:
#if(!${name})Default John#else${name}#end
which is unconveniant for my case.
After googling I've found DisplayTool, according to documentation it will look like:
$display.alt($name,"Default John")
So I added maven dependency but not sure how to add DisplayTool to my method and it is hard to found instructions for this.
Maybe somebody can help with advice or give useful links?..
My method:
public String testVelocity(String url) throws Exception{
Velocity.init();
VelocityContext context = getVelocityContext();//gets simple VelocityContext object
Writer out = new StringWriter();
Velocity.evaluate(context, out, "testing", url);
logger.info("got first results "+out);
return out.toString();
}
When I send
String url = "http://www.test.com?withDefault=$display.alt(\"not null\",\"exampleDefaults\")&truncate=$display.truncate(\"This is a long string.\", 10)";
String result = testVelocity(url);
I get "http://www.test.com?withDefault=$display.alt(\"not null\",\"exampleDefaults\")&truncate=$display.truncate(\"This is a long string.\", 10)" without changes, but should get
"http://www.test.com?withDefault=not null&truncate=This is...
Please tell me what I am missing. Thanks.
The construction of the URL occurs in your Java code, before you invoke Velocity, so Velocity isn't going to evaluate $display.alt(\"not null\",\"exampleDefaults\"). That syntax will be valid only in a Velocity template (which typically have .vm extensions).
In the Java code, there's no need to use the $ notation, you can just call the DisplayTool methods directly. I've not worked with DisplayTool before, but it's probably something like this:
DisplayTool display = new DisplayTool();
String withDefault = display.alt("not null","exampleDefaults");
String truncate = display.truncate("This is a long string.", 10);
String url = "http://www.test.com?"
+ withDefault=" + withDefault
+ "&truncate=" + truncate;
It might be better, though, to call your DisplayTool methods directly from the Velocity template. That's what is shown in the example usage.