I'm processing a csv file through my springboot app, the file is to download it, in my case I use streams but there is a problem what I don't know what's wrong in my code because some rows is complete with the columns but next row only write some columns and leftover columns are write below as if were a new row. I hope you understand what I mean. I hope you give a hand, thank you in advance.
This code below is the controller
.....
#RequestMapping(value="/stream/csv/{grupo}/{iduser}", method = RequestMethod.GET)
public void generateCSVUsingStream(#PathVariable("grupo") String grupo,
#PathVariable("iduser") String userId,HttpServletResponse response) {
response.addHeader("Content-Type", "application/csv");
response.addHeader("Content-Disposition", "attachment; filename=\""+userId+"_Reporte_PayCash"+grupo.replaceAll("\\s", "")+".csv");
response.setCharacterEncoding("UTF-8");
try (Stream<ReportePayCashDTO> streamPaycashdatos = capaDatosDao.ReportePayCashStream(userId, grupo);PrintWriter out = response.getWriter();) {
//PrintWriter out = response.getWriter();
out.write(String.join(",", "Cuenta" , "Referencia", "Referencia_paycash","Distrito","Plaza","Cartera"));
out.write("\n");
streamPaycashdatos.forEach(streamdato -> {
out.write(streamdato.getAccount()+","+streamdato.getReferencia()+","+streamdato.getReferenciapaycash()
+","+streamdato.getCartera()+","+streamdato.getState()+","+streamdato.getCity());
out.append("\r\n");
});
out.flush();
out.close();
streamPaycashdatos.close();
} catch (IOException ix) {
throw new RuntimeException("There is an error while downloading file", ix);
}
}
The method on DAO is this
...
#Override
public Stream<ReportePayCashDTO> ReportePayCashStream(String userId, String grupo) {
// TODO Auto-generated method stub
Stream<ReportePayCashDTO > stream = null ;
String query ="";
//more code
try {
stream = getJdbcTemplate().queryForStream(query, (rs, rowNum) -> {
return new ReportePayCashDTO(Utils.valnull(rs.getString("account")),
Utils.valnull(rs.getString("reference")),
Utils.valnull(rs.getString("referencepaycash")),
Utils.valnull(rs.getString("state")),
Utils.valnull(rs.getString("city")),
Utils.valnull(rs.getString("cartera"))
);
});
}catch(Exception e) {
e.printStackTrace();
logger.error(e.getMessage());
}
return stream;
}
Example: This is what I hoped will write into csv file
55xxxxx02,88xxxx153,1170050202662,TAMAULIPAS,TAMPICO,AmericanExpre
58xxxxx25,88xxx899,1170050202662,TAMAULIPAS,TAMPICO,AmericanClasic
but some rows was written like this
55xxxxx02,88xxxx153,1170050202662
,TAMAULIPAS,TAMPICO,AmericanExpre
58xxxxx25,88xxx899,1170050202662
,TAMAULIPAS,TAMPICO,AmericanClasic
Related
An attempt was made to implement file downloads through the SpringBoot MVC structure. There is no error, it says it has run normally, but the download does not proceed.
All information about the file is entered correctly, and also the path and name of the file are entered correctly.
I'd like to know why the download doesn't proceed even though there's no error.
#RestController
public class Controller {
#PostMapping("/fileDownload")
public void fileDownload(#RequestBody BoardFileDTO dto,HttpServletRequest request,HttpServletResponse response) throws UnsupportedEncodingException {
//File contains all stored paths, names, and extensions
Path fileNamePath = Paths.get(Directory + dto.getFile_save_name()).toAbsolutePath();
String filename = dto.getFile_save_name(); //The name of the saved file
String downname = dto.getFile_name(); //The name of the file to be saved
if (filename == null || "".equals(filename)) {
filename = downname;
}
try {
String browser = request.getHeader("User-Agent");
//File Encoding
if (browser.contains("MSIE") || browser.contains("Trident")
|| browser.contains("Chrome")) {
filename = URLEncoder.encode(filename, "UTF-8").replaceAll("\\+",
"%20");
} else {
filename = new String(filename.getBytes("UTF-8"), "ISO-8859-1");
}
} catch (UnsupportedEncodingException ex) {
System.out.println("UnsupportedEncodingException");
}
System.out.println(fileNamePath);
File file1 = new File(fileNamePath.toString());
if (!file1.exists()) {
return ;
}
// Specifying a File
response.setContentType("application/octer-stream");
response.setHeader("Content-Transfer-Encoding", "binary;");
response.setHeader("Content-Disposition", "attachment; filename=\"" + filename + "\"");
try {
OutputStream os = response.getOutputStream();
FileInputStream fis = new FileInputStream(fileNamePath.toString());
int ncount = 0;
byte[] bytes = new byte[512];
while ((ncount = fis.read(bytes)) != -1 ) {
os.write(bytes, 0, ncount);
}
fis.close();
os.close();
} catch (FileNotFoundException ex) {
System.out.println("FileNotFoundException");
} catch (IOException ex) {
System.out.println("IOException");
}
}
}
Your code is a bit convoluted imho. A couple of issues I see with your code
Using Path.toString to convert to a File, use the proper factory methods instead or use java.nio.Files to check the existence.
Your content-type is wrong application/octer-stream isn't a known content-type (you probably want application/octet-stream.
Copying from a Path or File is better done with either the StreamUtils from Spring or the java.nio.Files class (if you already have a Path use that).
#PostMapping("/fileDownload")
public void fileDownload(#RequestBody BoardFileDTO dto, HttpServletRequest request, HttpServletResponse response) throws UnsupportedEncodingException {
//File contains all stored paths, names, and extensions
Path fileNamePath = Paths.get(Directory, dto.getFile_save_name()).toAbsolutePath();
if (!Files.exists(fileNamePath)) {
return;
}
String filename = determineFilename(dto, request);
// Specifying a File
response.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);
response.setHeader("Content-Transfer-Encoding", "binary;");
response.setHeader("Content-Disposition", "attachment; filename=\"" + filename + "\"");
try {
Files.copy(fileNamePath, response.getOutputStream());
} catch (IOException ex) {
System.out.println("IOException");
}
}
private static String determineFilename(BoardFileDTO dto, HttpServletRequest request) {
String filename = dto.getFile_save_name(); //The name of the saved file
if (filename == null || "".equals(filename)) {
filename = dto.getFile_name();
}
String browser = request.getHeader("User-Agent");
//File Encoding
if (browser.contains("MSIE") || browser.contains("Trident") || browser.contains("Chrome")) {
filename = URLEncoder.encode(filename, StandardCharsets.UTF_8).replaceAll("\\+", "%20");
} else {
filename = new String(filename.getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1);
}
return filename;
}
It would write it something like that. As you have a path use the java.nio.Files to check for existence and copying. Use constants for mediatypes and charsets.
Your error handling is quite basic (I would say non-existing and at least not proper) as the processing just stops and returns an empty 200 to the client. No information what so ever.
I took the liberty to factor out the logic to determine the filename, which should make your code more readable.
I have a spring boot application and for a particular feature I have to prepare a CSV everyday for another service to use. The job runs everyday at 6 AM. And dumps the csv on the server. The issue is the data list is big. It's Around 7.8 millions of rows.I am using spring JPA to fetch all the records. Is their any better way to make it more efficient? Here's my code....
#Scheduled(cron = "0 1 6 * * ?")
public void saveMasterChildList() {
log.debug("running write job");
DateFormat dateFormatter = new SimpleDateFormat("dd_MM_yy");
String currentDateTime = dateFormatter.format(new Date());
String fileName = currentDateTime + "_Master_Child.csv";
ICsvBeanWriter beanWriter = null;
List<MasterChild> masterChildren = masterChildRepository.findByMsisdnIsNotNull();
try {
beanWriter = new CsvBeanWriter(new FileWriter(new File("/u01/edw_bill/", fileName)),
CsvPreference.STANDARD_PREFERENCE);
String[] header = {"msisdn"};
String[] nameMapping = {"msisdn"};
beanWriter.writeHeader(header);
for (MasterChild masterChild : masterChildren) {
beanWriter.write(masterChild, nameMapping);
}
} catch ( IOException e) {
log.debug("Error writing the CSV file {}", e.toString());
} finally {
if (beanWriter != null) {
try {
beanWriter.close();
} catch (IOException e) {
log.debug("Error closing the writer {}", e.toString());
}
}
}
} here
You could use pagination to separate data and load chunk by chunk. See this.
I am trying to implement an export to excel function via a web service which uses webflux as the other api and controllers work well. My problem is that calling the function that generates the excel file is accessed after retrieving data from repository as a Flux (no problem there). I have sorted the results and am trying to call another populate methid via flatMap, I am having a number of issues trying to get this to work and to make sure that the code in the flatMap runs before the code in the webservice to return the file.
Below is the code for the webservice:
#GetMapping(API_BASE_PATH + "/download")
public ResponseEntity<byte[]> download() {
Mono<Void> createExcel = excelExport.createDocument(false);
Mono.when(createExcel).log("Excel Created").then();
Workbook workbook = excelExport.getWb();
OutputStream outputStream = new ByteArrayOutputStream();
try {
workbook.write(outputStream);
} catch (IOException e) {
e.printStackTrace();
}
byte[] media = ((ByteArrayOutputStream) outputStream).toByteArray();
HttpHeaders headers = new HttpHeaders();
headers.setCacheControl(CacheControl.noCache().getHeaderValue());
headers.setContentType(MediaType.valueOf("text/html"));
headers.set("Content-disposition", "attachment; filename=filename.xlsx");
ResponseEntity<byte[]> responseEntity = new ResponseEntity<>(media, headers, HttpStatus.OK);
return responseEntity;
}
And the code for the exelExport class:
public Mono<Void> createDocument(boolean all) {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
try {
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
Row row = sheet.getRow(1);
Cell cell = row.getCell(3);
if (cell == null)
cell = row.createCell(3);
cell.setCellType(CellType.STRING);
cell.setCellValue("a test");
log.info("Created document");
Flux<TimeKeepingEntry> entries = service.findByMonth(LocalDate.now().getMonth().getDisplayName(TextStyle.FULL, Locale.ENGLISH)).log("Excel Export - retrievedMonths");
entries.subscribe();
return entries.groupBy(TimeKeepingEntry::getDateOfMonth).flatMap(Flux::collectList).flatMap(timeKeepingEntries -> this.populateEntry(sheet, timeKeepingEntries)).then();
} catch (IOException e) {
log.error("Error Creating Document", e);
}
//should never get here
return Mono.empty();
}
private void populateEntry(Sheet sheet, List<TimeKeepingEntry> timeKeepingEntries) {
int rowNum = 0;
for (int i = 0; i < timeKeepingEntries.size(); i++) {
TimeKeepingEntry timeKeepingEntry = timeKeepingEntries.get(i);
if (i == 0) {
rowNum = calculateFirstRow(timeKeepingEntry.getDay());
}
LocalDate date = timeKeepingEntry.getFullDate();
Row row2 = sheet.getRow(rowNum);
Cell cell2 = row2.getCell(1);
cell2.setCellValue(date.toString());
if (timeKeepingEntry.getDay().equals(DayOfWeek.FRIDAY.getDisplayName(TextStyle.FULL, Locale.ENGLISH))) {
rowNum = +2;
} else {
rowNum++;
}
}
}
The workbook is never update because the populateEntry is never executed. As I said I have tried a number of differnt methods including Mono.just and Mono.when, but I cant seem to get the correct combination to get it to process before the webservice method tries to return the file.
Any help would be great.
Edit1: Shows the ideal crateDocument Method.
public Mono<Void> createDocument(boolean all) {
try {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
log.info("Created document");
if (all) {
//all entries
} else {
service.findByMonth(currentMonthName).log("Excel Export - retrievedMonths").collectSortedList(Comparator.comparing(TimeKeepingEntry::getDateOfMonth)).doOnNext(timeKeepingEntries -> {
this.populateEntry(sheet, timeKeepingEntries);
});
}
} catch (IOException e) {
log.error("Error Importing File", e);
}
return Mono.empty();
}
There are several problems in the implementation of your webservice.
When to subscribe
First off, in reactive programming you must generally try to build a single processing pipeline (by calling Mono and Flux operators and returning the end result as a Mono and Flux). In any case, you should either let the framework do the subscribe or at least only subscribe once, at the end of that pipeline.
Here instead you are mixing two approaches: your createDocument method correctly returns a Mono, but it also does the subscribe. Even worse, the subscription is done on an intermediate step, and nothing subscribes to the whole pipeline in the webservice method.
So in effect, nobody sees the second half of the pipeline (starting with groupBy) and thus it never gets executed (this is a lazy Flux, also called a "cold" Flux).
Mixing synchronous and asynchronous
The other problem is again an issue of mixing two approaches: your Flux are lazy and asynchronous, but your webservice is written in an imperative and synchronous style.
So the code starts an asynchronous Flux from the DB, immediately return to the controller and tries to load the file data from disk.
Option 1: Making the controller more Flux-oriented
If you use Spring MVC, you can still write these imperative style controllers yet sprinkle in some WebFlux. In that case, you can return a Mono or Flux and Spring MVC will translate that to the correct asynchronous Servlet construct. But that would mean that you must turn the OutputStream and bytes handling into a Mono, to chain it to the document-writing Mono using something like then/flatMap/etc... It is a bit more involved.
Option 2: Turning the Flux into imperative blocking code
The other option is to go back to imperative and blocking style by calling block() on the createDocument() Mono. This will subscribe to it and wait for it to complete. After that, the rest of your imperative code should work fine.
Side Note
groupBy has a limitation where if it results in more than 256 open groups it can hang. Here the groups cannot close until the end of the file has been reached, but fortunately since you only process data for a single month, the Flux wouldn't exceed 31 groups.
Thanks to #SimonBasie for the pointers, my working code is now as follows.
#GetMapping(value = API_BASE_PATH + "/download", produces = "application/vnd.ms-excel")
public Mono<Resource> download() throws IOException {
Flux<TimeKeepingEntry> createExcel = excelExport.createDocument(false);
return createExcel.then(Mono.fromCallable(() -> {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
excelExport.getWb().write(outputStream);
return new ByteArrayResource(outputStream.toByteArray());
}));
}
public Flux<TimeKeepingEntry> createDocument(boolean all) {
Flux<TimeKeepingEntry> entries = null;
try {
InputStream inputStream = new ClassPathResource("Timesheet Template.xlsx").getInputStream();
wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheetAt(0);
log.info("Created document");
if (all) {
//all entries
} else {
entries = service.findByMonth(currentMonthName).log("Excel Export - retrievedMonths").sort(Comparator.comparing(TimeKeepingEntry::getDateOfMonth)).doOnNext(timeKeepingEntry-> {
this.populateEntry(sheet, timeKeepingEntry);
});
}
} catch (IOException e) {
log.error("Error Importing File", e);
}
return entries;
}
I have researched and tried to refer back to my fileChooser.getSeletedFile() in my save as action but can not work out how to check whether or not a file has been created. Here is my attempted code so far:
Save as code(works well):
public void Save_As() {
fileChooserTest.setApproveButtonText("Save");
int actionDialog = fileChooserTest.showOpenDialog(this);
File fileName = new File(fileChooserTest.getSelectedFile() + ".txt");
try {
if (fileName == null) {
return;
}
BufferedWriter outFile = new BufferedWriter(new FileWriter(fileName));
outFile.write(this.jTextArea2.getText());//put in textfile
outFile.flush(); // redundant, done by close()
outFile.close();
} catch (IOException ex) {
}
}
"Save" code doesn't work:
private void SaveActionPerformed(java.awt.event.ActionEvent evt) {
File f = fileChooserTest.getSelectedFile();
try {
if (f.exists()) {
BufferedWriter bw1 = new BufferedWriter(new FileWriter(fileChooserTest.getSelectedFile() + ".txt"));
bw1 = new BufferedWriter(new FileWriter(fileChooserTest.getSelectedFile() + ".txt"));
String text = ((JTextArea) jTabbedPane1.getSelectedComponent()).getText();
bw1.write(text);
bw1.close();
} else {
Save_As();
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
Instead of storing an instance to the JFileChooser rather store an instance to the File (wich will be null before any save has been performed). In your SaveActionPerformed method check if the file is null. If it is null then do a Save_As and store the selected file in your file variable, if it is not null then do a normal save into the file.
I want to get the XML in atom format of a GoogleDocs spreadsheet using the [generateAtom(..,..)][1] method of the class BaseEntry which a SpreadsheetEntry inherits. But I don't understand the the second parameter in the method, ExtensionProfile. What is it and will this method call suffice if I just want to get the XML in atom format?
XmlWriter x = new XmlWriter();
spreadSheetEntry.generateAtom(x,new ExtensionProfile());
[1]: http://code.google.com/apis/gdata/javadoc/com/google/gdata/data/BaseEntry.html#generateAtom(com.google.gdata.util.common.xml.XmlWriter, com.google.gdata.data.ExtensionProfile)
From the JavaDoc for ExtensionProfile:
A profile is a set of allowed
extensions for each type together with
additional properties.
Usually if you've got a service, you can ask that for its extension profile using Service.getExtensionProfile().
Elaborating Jon Skeet's answer, you need to instanciate a service like this:
String developer_key = "mySecretDeveloperKey";
String client_id = "myApplicationsClientId";
YouTubeService service = new YouTubeService(client_id, developer_key);
Then you can write to a file using the extension profile of your service:
static void write_video_entry(VideoEntry video_entry) {
try {
String cache_file_path = Layout.get_cache_file_path(video_entry);
File cache_file = new File(cache_file_path);
Writer writer = new FileWriter(cache_file);
XmlWriter xml_writer = new XmlWriter(writer);
ExtensionProfile extension_profile = service.getExtensionProfile();
video_entry.generateAtom(xml_writer, extension_profile);
xml_writer.close();
writer.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Analogously, you can read a file using the extension profile of your service:
static VideoFeed read_video_feed(File cache_file_file) {
VideoFeed video_feed = new VideoFeed();
try {
InputStream input_stream = new FileInputStream(cache_file_file);
ExtensionProfile extension_profile = service.getExtensionProfile();
try {
video_feed.parseAtom(extension_profile, input_stream);
} catch (ParseException e) {
e.printStackTrace();
}
input_stream.close();
} catch (IOException e) {
e.printStackTrace();
}
return video_feed;
}