I am trying to read an Excel file in manipulate it or add new data to it and write it back out. I am also trying to do this a complete reactive process using Flux and Mono. The Idea is to return the resulting file or bytearray via a webservice.
My question is how do I get a InputStream and OutputStream in a non blocking way?
I am using the Apache Poi library to read and generate the Excel File.
I currently have a solution based around a mix of Mono.fromCallable() and Blocking code getting the Input Stream.
For example the webservice part is as follows.
#GetMapping(value = API_BASE_PATH + "/download", produces = "application/vnd.ms-excel")
public Mono<ByteArrayResource> download() {
Flux<TimeKeepingEntry> createExcel = excelExport.createDocument(false);
return createExcel.then(Mono.fromCallable(() -> {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
excelExport.getWb().write(outputStream);
return new ByteArrayResource(outputStream.toByteArray());
}).subscribeOn(Schedulers.elastic()));
}
And the Processing of the file:
public Flux<TimeKeepingEntry> createDocument(boolean all) {
Flux<TimeKeepingEntry> entries = null;
try {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
log.info("Created document");
if (all) {
//all entries
} else {
entries = service.findByMonth(currentMonthName).log("Excel Export - retrievedMonths").sort(Comparator.comparing(TimeKeepingEntry::getDateOfMonth)).doOnNext(timeKeepingEntry-> {
this.populateEntry(sheet, timeKeepingEntry);
});
}
} catch (IOException e) {
log.error("Error Importing File", e);
}
return entries;
}
This works well enough but not very in line with Flux and Mono. Some guidance here would be good. I would prefer to have the whole sequence non-blocking.
Unfortunately the WorkbookFactory.create() operation is blocking, so you have to perform that operation using imperative code. However fetching each timeKeepingEntry can be done reactively. Your code would looks something like this:
public Flux<TimeKeepingEntry> createDocument() {
return Flux.generate(
this::getWorkbookSheet,
(sheet, sink) -> {
sink.next(getNextTimeKeepingEntryFrom(sheet));
},
this::closeWorkbook);
}
This will keep the workbook in memory, but will fetch each entry on demand when the elements of the Flux are requested.
Related
I am from the old world that think webapi should return a strong typed object and let json serialization return data.
However, recently we got this requirement:
We have a sql table which has more than 500 columns.
The customer always want to return all the columns.
Our c# code does nothing other than reading the SqlDatareader, convert the reader to a c# object and return result.
In this case, wouldn't better to do this (example copied from another stackoverflow post). Basically just return a stream? Does returning a stream still considered to be anti-pattern?
public HttpResponseMessage SomeMethod(List<string> someIds)
{
HttpResponseMessage resp = new HttpResponseMessage();
resp.Content = new PushStreamContent(async (responseStream, content, context) =>
{
await CopyBinaryValueToResponseStream(responseStream, someIds);
});
return resp;
}
private static async Task CopyBinaryValueToResponseStream(Stream responseStream, int imageId)
{
// PushStreamContent requires the responseStream to be closed
// for signaling it that you have finished writing the response.
using (responseStream)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
await connection.OpenAsync();
using (SqlCommand command = new SqlCommand("SELECT 500 columns FROM [StupidWideTable] WHERE ....", connection))
{
.....
using (SqlDataReader reader = await command.ExecuteReaderAsync(CommandBehavior.SequentialAccess))
{
if (await reader.ReadAsync())
{
if (!(await reader.IsDBNullAsync(0)))
{
using (Stream data = reader.GetStream(0))
{
// Asynchronously copy the stream from the server to the response stream
await data.CopyToAsync(responseStream);
}
}
}
}
}
}
}// close response stream
}
Does returning a stream still considered to be anti-pattern?
Well, that depends on what you want to do. For example, if you want to return a 500 if the SQL server fails partway through, then you shouldn't return a stream.
Streaming results works fine on ASP.NET, but it's important to note that all headers (including the response status code) are sent before the stream begins. So you'll send an immediate 200 when you start streaming the result, and if there's an error later on there's no way to go back in time and change that to a 500. Or add some kind of Continue header.
In other words, yes it's supported; but you lose all the benefits of model binding, content negotiation, exception handlers, etc., because you're bypassing that whole pipeline.
I am aware of the TransferManager and the .uploadFileList() and .uploadFileDirectory() methods, however they accept java.io.File types as arguments. I have a collection of byte array input streams containing jpeg image data. I don't want to create in-memory files to store this data before I upload it either.
So what I need is essentially what the S3 client's PutObjectRequest does but for a collection of InputStream objects. Also, if one upload fails, I want to abort the whole thing and not upload anything, much like how a database transaction will reverse the changes if something goes wrong along the way.
Is this possible with the Java SDK?
Before I share an answer, please consider upgrading...
fyi - TransferManager is deprecated, now supported as TransferManagerBuilder in JAVA AWS SDK, please consider upgrading if TransferManagerBuilder Object suits your needs.
now since you asked about TransferManager, you could either 1) copy the code below and replace the functionality/arguments with your custom in memory handling of the input stream and handle it in your custom function... or; 2) further below is another sample, try to use this as-is...
Github source modify with with inputstream and issue listed here
private def uploadFile(is: InputStream, s3ObjectName: String, metadata: ObjectMetadata) = {
try {
val putObjectRequest = new PutObjectRequest(bucketName, s3ObjectName,
is, metadata)
// TransferManager supports asynchronous uploads and downloads
val upload = transferManager.upload(putObjectRequest)
upload.addProgressListener(ExceptionReporter.wrap(UploadProgressListener(putObjectRequest)))
} catch {
case e: Exception => throw new RuntimeException(e)
}
}
Bonus, Nice custom answer here using sequence input streams
public void combineFiles() {
List<String> files = getFiles();
long totalFileSize = files.stream()
.map(this::getContentLength)
.reduce(0L, (f, s) -> f + s);
try {
try (InputStream partialFile = new SequenceInputStream(getInputStreamEnumeration(files))) {
ObjectMetadata resultFileMetadata = new ObjectMetadata();
resultFileMetadata.setContentLength(totalFileSize);
s3Client.putObject("bucketName", "resultFilePath", partialFile, resultFileMetadata);
}
} catch (IOException e) {
LOG.error("An error occurred while combining files. {}", e);
}
}
private Enumeration<? extends InputStream> getInputStreamEnumeration(List<String> files) {
return new Enumeration<InputStream>() {
private Iterator<String> fileNamesIterator = files.iterator();
#Override
public boolean hasMoreElements() {
return fileNamesIterator.hasNext();
}
#Override
public InputStream nextElement() {
try {
return new FileInputStream(Paths.get(fileNamesIterator.next()).toFile());
} catch (FileNotFoundException e) {
System.err.println(e.getMessage());
throw new RuntimeException(e);
}
}
};
}
I am reading a csv file input from the client browser and code is below
#RequestMapping(value = "/file-upload", method = {RequestMethod.PUT, RequestMethod.POST}, consumes =
MediaType.APPLICATION_JSON_UTF8_VALUE, produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
#ResponseStatus(HttpStatus.OK)
public Set<SessionBulkUploadLine> fileUpload(#RequestBody final FileUploadInfo fileUploadInfo) {
final CsvFileParser parser = new CsvFileParser();
try (final ByteArrayInputStream stream = new ByteArrayInputStream(settings.getBytes())) {
final Spreadsheet sheet = parser.parse(stream, true);
}
Do i need to close the stream in the above??
Please advise.
Thanks
1) No, you don't need to close it by explicitly calling .close() because it will be automatically closed after the try catch finish its execution. (you're using a try-catch with resources)
2) If you're not using a try-catch with resources then any stream should be closed after you're done using it so the garbage collector can remove it from memory. (A stream often uses a lot of resources)
try{
final ByteArrayInputStream stream = ... ;
// logic here
}catch(Exception e)
{
// print error
}finally{
stream.close();
}
Please note that the stream is initialized INSIDE the try block.
I'm trying to write a custom Nifi processor which will take in the contents of the incoming flow file, perform some math operations on it, then write the results into an outgoing flow file. Is there a way to dump the contents of the incoming flow file into a string or something? I've been searching for a while now and it doesn't seem that simple. If anyone could point me toward a good tutorial that deals with doing something like that it would be greatly appreciated.
The Apache NiFi Developer Guide documents the process of creating a custom processor very well. In your specific case, I would start with the Component Lifecycle section and the Enrich/Modify Content pattern. Any other processor which does similar work (like ReplaceText or Base64EncodeContent) would be good examples to learn from; all of the source code is available on GitHub.
Essentially you need to implement the #onTrigger() method in your processor class, read the flowfile content and parse it into your expected format, perform your operations, and then re-populate the resulting flowfile content. Your source code will look something like this:
#Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
FlowFile flowFile = session.get();
if (flowFile == null) {
return;
}
final ComponentLog logger = getLogger();
AtomicBoolean error = new AtomicBoolean();
AtomicReference<String> result = new AtomicReference<>(null);
// This uses a lambda function in place of a callback for InputStreamCallback#process()
processSession.read(flowFile, in -> {
long start = System.nanoTime();
// Read the flowfile content into a String
// TODO: May need to buffer this if the content is large
try {
final String contents = IOUtils.toString(in, StandardCharsets.UTF_8);
result.set(new MyMathOperationService().performSomeOperation(contents));
long stop = System.nanoTime();
if (getLogger().isDebugEnabled()) {
final long durationNanos = stop - start;
DecimalFormat df = new DecimalFormat("#.###");
getLogger().debug("Performed operation in " + durationNanos + " nanoseconds (" + df.format(durationNanos / 1_000_000_000.0) + " seconds).");
}
} catch (Exception e) {
error.set(true);
getLogger().error(e.getMessage() + " Routing to failure.", e);
}
});
if (error.get()) {
processSession.transfer(flowFile, REL_FAILURE);
} else {
// Again, a lambda takes the place of the OutputStreamCallback#process()
FlowFile updatedFlowFile = session.write(flowFile, (in, out) -> {
final String resultString = result.get();
final byte[] resultBytes = resultString.getBytes(StandardCharsets.UTF_8);
// TODO: This can use a while loop for performance
out.write(resultBytes, 0, resultBytes.length);
out.flush();
});
processSession.transfer(updatedFlowFile, REL_SUCCESS);
}
}
Daggett is right that the ExecuteScript processor is a good place to start because it will shorten the development lifecycle (no building NARs, deploying, and restarting NiFi to use it) and when you have the correct behavior, you can easily copy/paste into the generated skeleton and deploy it once.
My script fetches xml via httpConnection and saves to persistent store. No problems there.
Then I loop through the saved data to compose a list of image url's to fetch via queue.
Each of these requests calls the httpConnection thread as so
...
public synchronized void run()
{
HttpConnection connection = (HttpConnection)Connector.open("http://www.somedomain.com/image1.jpg");
connection.setRequestMethod("GET");
String contentType = connection.getHeaderField("Content-type");
InputStream responseData = connection.openInputStream();
connection.close();
outputFinal(responseData, contentType);
}
public synchronized void outputFinal(InputStream result, String contentType) throws SAXException, ParserConfigurationException, IOException
{
if(contentType.startsWith("text/"))
{
// bunch of xml save code that works fine
}
else if(contentType.equals("image/png") || contentType.equals("image/jpeg") || contentType.equals("image/gif"))
{
// how to save images here?
}
else
{
//default
}
}
What I can't find any good documentation on is how one would take the response data and save it to an image stored on the device.
Maybe I just overlooked something very obvious. Any help is very appreciated.
Thanks
I tried following this advise and found the same thing I always find when looking up BB specific issues: nothing.
The problem is that every example or post assumes you know everything about the platform.
Here's a simple question: What line of code writes the read output stream to the blackberry device? What path? How do I retrieve it later?
I have this code, which I do not know if it does anything because I don't know where it is supposedly writing to or if that's even what it is doing at all:
** filename is determined on a loop based on the url called.
FileOutputStream fos = null;
try
{
fos = new FileOutputStream( File.FILESYSTEM_PATRIOT, filename );
byte [] buffer = new byte [262144];
int byteRead;
while ((byteRead = result.read (buffer ))!=- 1)
{
fos.write (buffer, 0, byteRead);
}
fos.flush();
fos.close();
}
catch(IOException ieo)
{
}
finally
{
if(fos != null)
{
fos.close();
}
}
The idea is that I have some 600 images pulled from a server. I need to loop the xml and save each image to the device so that when an entity is called, I can pull the associated image - entity_id.png - from the internal storage.
The documentation from RIM does not specify this, nor does it make it easy to begin figuring it out.
This issue does not seem to be addressed on this forum, or others I have searched.
Thanks
You'll need to use the Java FileOutputStream to do the writing. You'll also want to close the connection after reading the data from the InputStream (move outputFinal above your call to close). You can find all kinds of examples regarding FileOutputStream easily.
See here for more. Note that in order to use the FileOutputStream your application must be signed.