I'm outputting a tab-delimited file from my webapp that should be opened in Excel. The problem is that .xls seems not good for opening and editing it, then Excel required some other format, and if I change the extension to .tsv then the file becomes unknown for Excel (on Windows 7) and .csv is for comma-separated. Can you advice me which the file extension should be?
This is the code that outputs the file and it works. It's just that I should choose the most suitable extension for tab-separated values.
#RequestMapping(value = "/export", method = RequestMethod.GET)
#ResponseBody
public ModelAndView export(HttpServletResponse response) {
try {
String str = "";
Iterator<Individual> iterator = customerAccountService.getAllIndividuals().iterator();
while(iterator.hasNext()){
Individual individual = iterator.next();
str = str + individual.getId() + "\t" +individual.getIndividualName().getName() + "\t" + individual.getAddress().getStreetName() + "\n";
}
InputStream is = new ByteArrayInputStream(str.getBytes());
IOUtils.copy(is, response.getOutputStream());
response.setContentType("application/xls");
response.setHeader("Content-Disposition","attachment; filename=export.tsv");
response.flushBuffer();
} catch (IOException ex) {
//logger.info("Error writing file to output stream. Filename was '" + fileName + "'");
throw new RuntimeException("IOError writing file to output stream");
}
ModelAndView modelAndView = new ModelAndView(ViewName.MENU);
modelAndView.addObject(ObjectName.ADD_FORM, new LoginForm());
return modelAndView;
}
Put
sep=\t
as the first line in your .csv-file (yes, you can name it .csv then). That tells excel what the delimiter character should be.
Note, that actually if you open the .csv with a text editor, it should read like
sep= (an actual tabulator character here, it's just not visible...)
Related
We have large csv file with 100 millions records, and used spring batch to load, read and write to database by splitting file with 1 million records using "SystemCommandTasklet". Below is snippet,
#Bean
#StepScope
public SystemCommandTasklet splitFileTasklet(#Value("#{jobParameters[filePath]}") final String inputFilePath) {
SystemCommandTasklet tasklet = new SystemCommandTasklet();
final File file = BatchUtilities.prefixFile(inputFilePath, AppConstants.PROCESSING_PREFIX);
final String command = configProperties.getBatch().getDataLoadPrep().getSplitCommand() + " " + file.getAbsolutePath() + " " + configProperties.getBatch().getDataLoad().getInputLocation() + System.currentTimeMillis() / 1000;
tasklet.setCommand(command);
tasklet.setTimeout(configProperties.getBatch().getDataLoadPrep().getSplitCommandTimeout());
executionContext.put(AppConstants.FILE_PATH_PARAM, file.getPath());
return tasklet;
}
and batch-config:
batch:
data-load-prep:
input-location: /mnt/mlr/prep/
split-command: split -l 1000000 --additional-suffix=.csv
split-command-timeout: 900000 # 15 min
schedule: "*/60 * * * * *"
lock-at-most: 5m
With above config, I could able to read load and write successfully to database. However, found a bug with below snippet that, after splitting the file, only first file will have headers, but next splitted file does not have hearders in the first line. So, I have to either disable or avoid linesToSkip(1) config for FlatFileItemReader(CSVReader).
#Configuration
public class DataLoadReader {
#Bean
#StepScope
public FlatFileItemReader<DemographicData> demographicDataCSVReader(#Value("#{jobExecutionContext[filePath]}") final String filePath) {
return new FlatFileItemReaderBuilder<DemographicData>()
.name("data-load-csv-reader")
.resource(new FileSystemResource(filePath))
.linesToSkip(1) // Need to avoid this from 2nd splitted file onwards as splitted file does not have headers
.lineMapper(lineMapper())
.build();
}
public LineMapper<DemographicData> lineMapper() {
DefaultLineMapper<DemographicData> defaultLineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setNames("id", "mdl65DecileNum", "mdl66DecileNum", "hhId", "dob", "firstName", "middleName",
"lastName", "addressLine1", "addressLine2", "cityName", "stdCode", "zipCode", "zipp4Code", "fipsCntyCd",
"fipsStCd", "langName", "regionName", "fipsCntyName", "estimatedIncome");
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(new DemographicDataFieldSetMapper());
return defaultLineMapper;
}
}
Note: Loader should not skip first row from second file while loading.
Thank you in advance. Appreciate any suggestions.
I would do it in the SystemCommandTasklet with the following command:
tail -n +2 data.csv | split -l 1000000 --additional-suffix=.csv
If you really want to do it with Java in your Spring Batch job, you can use a custom reader or an item processor that filters the header. But I would not recommend this approach as it introduces an additional test for each item (given the large number of lines in your input file, this could impact the performance of your job).
I tried to upload files to dynamic directory to SFTP. When I uploaded some files, the first file always uploaded to the last directory. Then after that rest file will be uploaded to the correct directory. When I did debug mode, I saw that every first file would be uploaded to temporaryDirectory which is the code already set up by spring. I don't know how to set the value of this temporaryDirectory to the right value. Please, help me to solve the problem.
Or maybe you guys have other way to upload and create proper dynamic directory. Please let me know.
Here is the code:
private String sftpRemoteDirectory = "documents/"
#MessagingGateway
public interface UploadGateway {
#Gateway(requestChannel = "toSftpChannel")
void upload(File file, #Header("dirName") String dirName);
}
#Bean
#ServiceActivator(inputChannel = "toSftpChannel")
public MessageHandler handler() {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
SimpleDateFormat formatter = new SimpleDateFormat("yyMMdd");
String newDynamicDirectory = "E" + formatter.format(new Date())+String.format("%04d",Integer.parseInt("0001") + 1);
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory + newDynamicDirectory));
handler.setFileNameGenerator(message -> {
String dirName = (String) message.getHeaders().get("dirName");
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory + dirName));
handler.setAutoCreateDirectory(true);
if (message.getPayload() instanceof File) {
return (((File) message.getPayload()).getName());
} else {
throw new IllegalArgumentException("File expected as payload!");
}
});
return handler;
}
You are using a LiteralExpression, evaluated just once, you need an expression that's evaluated at runtime.
handler.setRemoteDirectoryExpressionString("'" + sftpRemoteDirectory/ + "'" + headers['dirName']);
I am doing a portlet to create banners. I preferences I made the form with: input type="file" and the form nctype='multipart/form-data'
In the processAction I get the image, but I don't know how save it in the server, because I only get save in temporal instance portlet, but if I restart the server I lose the image.
This is my code to save the image:
private boolean uploadFile( ActionRequest request, ActionResponse response) throws ValidatorException, IOException, ReadOnlyException {
try {
// Si la request es del tipo multipart ...
if (PortletFileUpload.isMultipartContent(request)) {
DiskFileItemFactory diskFileItemFactory = new DiskFileItemFactory();
PortletFileUpload servletFileUpload = new PortletFileUpload(diskFileItemFactory);
servletFileUpload.setSizeMax(81920); // bytes
List fileItemsList = servletFileUpload.parseRequest(request);
Iterator it = fileItemsList.iterator();
while (it.hasNext()){
FileItem fileItem = (FileItem)it.next();
if (fileItem.isFormField()){
}
else{
String nombreCampo = fileItem.getFieldName();
String nombreArchivo = fileItem.getName();
String extension = nombreArchivo.substring(nombreArchivo.indexOf("."));
PortletContext context = request.getPortletSession().getPortletContext();
String path = context.getRealPath("/images");
File archivo = new File(path + "/" + nombreArchivo);
PortletContext pc = request.getPortletSession().getPortletContext();
fileItem.write(archivo);
}
}
}
} catch (Exception e) {}
return true;
}
I don't know if I am doing something wrong or this isn't the correct way.
Any idea?
Thanks in advance
EDIT:
Finally I tried do it with DLFolderLocalServiceUtil and DLFileEntryLocalServiceUtil, but it doesn't work correctly. When I load the page you can see the image, but after, when the page is load completely, the image disappears.
I don't know if it is because I don't create fine the fileEntry or the url is wrong.
This is my code:
long folderId = CounterLocalServiceUtil.increment(DLFolder.class.getName());
DLFolder folder = DLFolderLocalServiceUtil.createDLFolder(folderId);
long userId = themeDisplay.getUserId();
long groupId = themeDisplay.getScopeGroupId();
folder.setUserId(userId);
folder.setGroupId(groupId);
folder.setName("Banner image " + nombreArchivo+String.valueOf(folderId));
DLFolderLocalServiceUtil.updateDLFolder(folder);
ServiceContext serviceContext= ServiceContextFactory.getInstance(DLFileEntry.class.getName(), request);
File myfile = new File(nombreArchivo);
fileItem.write(myfile);
List<DLFileEntryType> tip = DLFileEntryTypeLocalServiceUtil.getFileEntryTypes(DLUtil.getGroupIds(themeDisplay));
DLFileEntry DLfileEntry = DLFileEntryLocalServiceUtil.addFileEntry(userId, groupId, 0, folderId, null, MimeTypesUtil.getContentType(myfile), nombreArchivo, "Image banner_"+nombreArchivo, "", tip.get(0).getFileEntryTypeId(), null, myfile, fileItem.getInputStream(), myfile.getTotalSpace(), serviceContext);
FileVersion fileVersion = null;
//FileEntry fileEntry = DLAppServiceUtil.getFileEntry(groupId, folderId, nombreArchivo);
//String path = DLUtil.getPreviewURL(fileEntry, fileVersion, themeDisplay, "&imagePreview=1");
String path1 = themeDisplay.getPortalURL()+"/c/document_library/get_file?uuid="+DLfileEntry.getUuid()+"&groupId="+themeDisplay.getScopeGroupId();
String path = "/documents/" + DLfileEntry.getGroupId() + "/" + DLfileEntry.getFolderId() + "/" + DLfileEntry.getTitle()+"/"+DLfileEntry.getUuid();
System.out.println("path " + path);
System.out.println("path " + path1);
prefs.setValue(nombreCampo, path);
And this is the output:
path /documents/10180/0/cinesa888.png/f24e6da2-0be8-47ad-a3b5-a4ab0d41d17f
path http://localhost:8080/c/document_library/get_file?uuid=f24e6da2-0be8-47ad-a3b5-a4ab0d41d17f&groupId=10180
I tried to get the url like lpratlong said (DLUtil) but when I tried to get the FileEntry with DLAppServiceUtil.getFileEntry(..) I have an error that says no exist FileEntry.
I don't know what I am doing wrong.. Any idea?
Thanks.
You can use Liferay API to store the file in the Document Library : take a look in DLFolder and DLFileEntry API (for exemple, DLFileEntryLocalServiceUtil will show you allowed local operations).
These API will allowed you to store your file in your file system (in the "data" folder of your Liferay installation) and to store reference of your file in Liferay database.
I am using following code to export the content to .cvs file
which also support chinese and japanese characters.
public ActionResult Download(strng accnumber)
{
string csvContent = "东西，东西，东西, hi";
var data = Encoding.UTF32.GetBytes(csvContent );
string filename = "CSV_" + accnumber + ".csv";
return File(data, "text/csv", filename);
}
when i export my file i am not getting proper chinese or japanese characters. what is missing?
i have used UTF32 encoding to support it.
Edited:
i have noticed that opening my .csv file in notepad shows perfect characters but ms-excel doesn't.
I am also got same problem, solve it by using UTF-8-BOM.
Encoding.UTF8.GetPreamble().Concat(Encoding.UTF8.GetBytes(stringData)).ToArray()
As you are on asp.net serving that file you also have to deal with the encoding of the http pipleline. I din't spot that earlier, sorry.
Instead of having a plain ActionResult you should use one of the derived ActionResults, I've used FileContentResult. Please pay note to the special ContentType I'm constructing to tell the browsers an UTF-32 file is coming...
public ActionResult Download(string accnumber)
{
string csvContent = "东西，东西，东西, hi";
var data = Encoding.UTF8.GetBytes(csvContent);
// add byte order mark
var bom = new byte[] { 0xEF, 0xBB, 0xBF };
// hold it all
var all = new byte[bom.Length + data.Length];
// copy over BOM
Array.Copy(bom, all, bom.Length);
// copy over data
Array.Copy(data, 0, all, bom.Length, data.Length);
string filename = "CSV_" + accnumber + ".csv";
var file = new FileContentResult( all, "text/csv" )
{
FileDownloadName = filename
};
return file;
}
I encountered this problem too. I fix it by adding the following line just before "return file ;" and it works for me.
Response.Write("<meta http-equiv=Content-Type content=text/html;charset=utf-8>");
return file;
If I'm using the InputStream to receive a file, like
HttpContext.Current.Request.InputStream
How can get more information about the file?
I can easily convert a Stream into a phisical File, but for example, how would I know the file extension in use?
string fileIn = #"C:\Temp\inputStreamedFile.xxx"; // What extension?
using (FileStream fs = System.IO.File.Create(fileIn))
{
Stream f = HttpContext.Current.Request.InputStream;
byte[] bytes = new byte[f.Length];
f.Read(bytes, 0, (int)f.Length);
fs.Write(bytes, 0, bytes.Length);
}
The idea behind this is because using HttpPostedFileBase I always get null:
public ContentResult Send(HttpPostedFileBase fileToUpload, string email)
{
// Get file stream and save it
// Get File in stream
string fileIn = Path.Combine(uploadsPath, uniqueIdentifier),
fileOut = Path.Combine(convertedPath, uniqueIdentifier + ".pdf");
// Verify that the user selected a file
if (fileToUpload != null && fileToUpload.ContentLength > 0)
{
// extract only the fielname
string fileExtension = Path.GetExtension(fileToUpload.FileName);
fileIn = String.Concat(fileIn, fileExtension);
fileToUpload.SaveAs(fileIn);
}
// TODO: Add Convert File to Batch
return Content("File queued for process with id: " + uniqueIdentifier);
}
and this is what I'm sending from the command line:
$ curl --form email='mail#domain.com' --form fileToUpload='C:\temp\MyWord.docx' http://localhost:64705/send/
File queued for process with id: 1d777cc7-7c08-460c-8412-ddab72408123
the variable email is filled up correctly, but fileToUpload is always null.
P.S. This does not happen if I use a form to upload the same data.
I'm sorry if this doesn't help, but why use InputStream to get uploaded file(s) ?
This is what I usually do:
[HttpPost]
public ActionResult Upload(HttpPostedFileBase[] files) {
String physicalPath = "c:\\whatever";
foreach (var file in files) {
String extension = Path.GetExtension(file.FileName);
file.SaveAs(physicalPath + "\\" + file.FileName);
}
return View();
}
The only problem I found was using curl... I was forgetting the # sign that mention that the uploaded form would be encoded as multipart/form-data.
The correct curl command to use HttpPostedFileBase would be:
$ curl --form email='mail#domain.com'
--form fileToUpload=#'C:\temp\MyWord.docx'
http://localhost:64705/send/
You can get the info about posted file from <input type="file" />. But actually it's used a bit different way to upload files in asp.net mvc check out here