In Excel you can open a CSV file and click on "Text to Columns" (under "Data" tab) to have a column delimited file.
Is there any way to simulate this using X++? A class that would open the CSV file and apply the exact same Excel "Text to Columns" action?
You can use the InFieldDelimiter method of the CommaIo class to use the correct delimiters.
Or use str2con to split a string.
You can even read an Excel file using ExcelIo.
Update:
As your file is not CSV use str2con:
for (con = commaIo.read(); commaIo.status() == IO_Status::Ok; con = commaIo.read())
{
lineNum++;
if (lineNum > 1 && conlen(con))
con = str2con(conPeek(con,1));
info(str2con(con));
}
This may not remove quotes and trailing spaces, do this for interesting string fields:
purchId = strRtrim(strRem(conPeek(con,3),'"'));
Related
I have CSV file which has some blank lines.
Is there is setting in JMeter CSV data config to exclude blank lines and read only those lines which has values.
Please help in this regard.
I don't think it's possible, the options are in:
Remove blank lines from the CSV file using setUp Thread Group and JSR223 Sampler, example code:
def content = new File('test.csv').readLines()
def notblank = content.findAll { it != '' }
org.apache.commons.io.FileUtils.writeLines(new File('test.csv'), notblank)
Use If Controller to skip the blank lines via __jexl3() function
${__jexl3("${variable_from_csv}" != "",)}
I'm going to export all of my data in mySQL using PHPExcel with CSV Format. Here is my code:
function exportHartatoCSV() {
$memberid = $this->input->post('memberid');
$tahun = $this->input->post('tahunpajak');
$this->db->where('taxYear', $tahun);
$this->db->where('memberID', $memberid);
$query = $this->db->get('list_harta');
$this->load->library("excel");
$objPHPExcel = new PHPExcel();
$objPHPExcel->setActiveSheetIndex(0)
->setCellValue('A1', 'Tahun Pajak')
->setCellValue('B1', 'Pembetulan')
->setCellValue('C1', 'Kode Harta')
->setCellValue('D1', 'Jenis Harta')
->setCellValue('E1', 'Tahun Perolehan')
->setCellValue('F1', 'Harga Perolehan')
->setCellValue('G1', 'Keterangan');
$row = 2;
$workbookName = $this->MemberModel->getNPWP($memberid);
header('Content-Type: application/vnd.ms-excel');
header('Content-Disposition: attachment;filename="'.$workbookName.'.csv"');
header('Cache-Control: max-age=0');
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'CSV');
$objWriter->setDelimiter("\t");
$objWriter->setEnclosure('');
$total = 0;
foreach ($query->result() as $key) {
$objPHPExcel->getActiveSheet()->setCellValue('A'.$row, $key->taxYear);
$objPHPExcel->getActiveSheet()->setCellValue('B'.$row, $key->pembetulan);
$objPHPExcel->getActiveSheet()->setCellValue('C'.$row, $key->hartaID);
$objPHPExcel->getActiveSheet()->setCellValue('D'.$row, $key->hartaName);
$objPHPExcel->getActiveSheet()->setCellValue('E'.$row, $key->hartaYear);
$objPHPExcel->getActiveSheet()->setCellValue('F'.$row, $key->hartaPrice);
$objPHPExcel->getActiveSheet()->setCellValue('G'.$row, $key->hartaDesc);
$total += $key->hartaPrice;
$row++;
}
$objPHPExcel->getActiveSheet()->setCellValue('E'.$row, "Total Harta:");
$objPHPExcel->getActiveSheet()->setCellValue('F'.$row, $total);
$objWriter->save('php://output');
}
I want to change the delimiter from , to tab (like writing data to the next column. But it doesn't work. The strings are stick to each other. The output is like this:
Output
But I want the output is like this:
Expected Output
What I really want is a normal-looking excel table, but with .csv format.
So you need tab-delimited text (csv) file or you just want "normal"-looking excel table?
You must remember that there is a system list delimiter that is used in excel by default. In windows you can set that delimiter in your Region and Language, in Additional settings. It's called as List separator there. In some countries default value is ";", in others ",".
If you don't want to change your default delimiter you have an option to choose delimiter in Excel when you open file.
Update
You can use both options, but it looks like it's easier to set the delimiter in PHPExcel export settings to that delimiter that is set by default in your client's OS, than explain them to set custom delimiter in excel when opening file.
$objWriter->setDelimiter(";");
or
$objWriter->setDelimiter(",");
Update 2
If you have clients from different countries and/or they use different operating systems (mac/linux, windows), so they may have different system separators, then you'll need to save 2 versions of each file (with both , and ; delimiters). Or you'll need to explain them how to work with it.
I'm putting this system setting screenshots just for example:
Mac OS
Windows
As #teeyo mentioned, you might need to make sure you import it using the delimiter in Excel (or other spreadsheet editor). It might already be correct in the data generated by CI:
To verify the data is correct in the file itself, you can open it in eg. Notepad++ and use the Show All Characters feature:
I need to read the filename of a csv file to extract some information about what to do with the data inside the file. How do I read the filename?
e.g. I will get a file called 200_SomeTestName.csv. The file will contain details of the test to be created. I'm required to create a test called SomeTestName for the student who has an id of 200.
You can list all the CSV files in the dropbox folder and check if they match the pattern. Then extract the information you need and create the test:
Dir.glob('yourdropboxfolder/*.csv').each do |filename|
name = File.basename(filename, '.csv')
if (match = /(\d+)_(.*).csv/.match(name))
student_id = match[1]
test_name = match[2]
create_test_for(student_id, test_name, File.read(filename))
end
end
Not sure if you need to extract the values from the filename or if this information is also in the file.
I need to parse several csv files from a given folder. As each csv has different columns, there are separate tables in DB for each csv. I need to know
Does spring batch provide any mechanism which scans through the given folder and then I can pass those files one by one to the reader.
As I am trying to make the reader/writer generic, is it possible to just get the column header for each csv, based upon that I am trying to build tokenizer and also the insert query.
Code sample
public ItemReader<Gdp> reader1() {
FlatFileItemReader<Gdp> reader1 = new FlatFileItemReader<Gdp>();
reader1.setResource(new ClassPathResource("datagdp.csv"));
reader1.setLinesToSkip(1);
reader1.setLineMapper(new DefaultLineMapper<Gdp>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(new String[] { "region", "gdpExpend", "value" });
}
});
setFieldSetMapper(new BeanWrapperFieldSetMapper<Gdp>() {
{
setTargetType(Gdp.class);
}
});
}
});
return reader1;
}
Use a MultiResourceItemReader to scan all files.
I think you need a sort of classified ItemReader as MultiResourceItemReader.delegate but SB doesn't offer that so you have to write your own.
For ItemProcessor and ItemWriter SB offers a classifier-aware implementation (ClassifierCompositeItemProcessor and ClassifierCompositeItemWriter).
Obviously more different input file you have more XML config must be write,but it should be straightforward to do.
I suppose you are expecting this kind of implementation.
During the Partition Step Builder, read all the files names, file header, insert query for the writer and save the same in the Execution Context.
In the slave step, for every reader and writer, pass on the Execution context, get the file to read, file header to the tokenizer, insert query that needs to be inserted for that writer.
This resolves your question.
Answers for your questions:
I don't know about a specific mechanism on spring batch to scan files.
You can use opencsv as generic CSV reader, there are a lot of mechanisms reading files.
About OpenCSV:
If you are using maven project, try to import this dependency:
<dependency>
<groupId>net.sf.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>2.0</version>
</dependency>
You can read your files making an object for specific formats, or generic headers like this below:
private static List<DadosPeople> extrairDadosPeople() throws IOException {
CSVReader readerPeople = new CSVReader(new FileReader(people));
List<PeopleData> listPeople = new ArrayList<PeopleData>();
String[] nextLine;
while ((nextLine = readerPeople.readNext()) != null) {
PeopleData people = new PeopleData();
people.setIncludeData(nextLine[0]);
people.setPartnerCode(Long.valueOf(nextLine[1]));
listPeople.add(people);
}
readerPeople.close();
return listPeople;
}
There are a lot of other ways to read CSV files using opencsv:
If you want to use an Iterator style pattern, you might do something like this:
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"));
String [] nextLine;
while ((nextLine = reader.readNext()) != null) {
// nextLine[] is an array of values from the line
System.out.println(nextLine[0] + nextLine[1] + "etc...");
}
Or, if you might just want to slurp the whole lot into a List, just call readAll()...
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"));
List myEntries = reader.readAll();
which will give you a List of String[] that you can iterate over. If all else fails, check out the Javadocs here.
If you want to customize quote characters and separators, you'll find constructors that cater for supplying your own separator and quote characters. Say you're using a tab for your separator, you can do something like this:
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), '\t');
And if you single quoted your escaped characters rather than double quote them, you can use the three arg constructor:
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), '\t', '\'');
You may also skip the first few lines of the file if you know that the content doesn't start till later in the file. So, for example, you can skip the first two lines by doing:
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), '\t', '\'', 2);
Can I write csv files with opencsv?
Yes. There is a CSVWriter in the same package that follows the same semantics as the CSVReader. For example, to write a tab separated file:
CSVWriter writer = new CSVWriter(new FileWriter("yourfile.csv"), '\t');
// feed in your array (or convert your data to an array)
String[] entries = "first#second#third".split("#");
writer.writeNext(entries);
writer.close();
If you'd prefer to use your own quote characters, you may use the three arg version of the constructor, which takes a quote character (or feel free to pass in CSVWriter.NO_QUOTE_CHARACTER).
You can also customise the line terminators used in the generated file (which is handy when you're exporting from your Linux web application to Windows clients). There is a constructor argument for this purpose.
Can I dump out SQL tables to CSV?
Yes you can. There is a feature on CSVWriter so you can pass writeAll() a ResultSet.
java.sql.ResultSet myResultSet = ....
writer.writeAll(myResultSet, includeHeaders);
Is there a way to bind my CSV file to a list of Javabeans?
Yes there is. There is a set of classes to allow you to bind a CSV file to a list of JavaBeans based on column name, column position, or a custom mapping strategy. You can find the new classes in the com.opencsv.bean package. Here's how you can map to a java bean based on the field positions in your CSV file:
ColumnPositionMappingStrategy strat = new ColumnPositionMappingStrategy();
strat.setType(YourOrderBean.class);
String[] columns = new String[] {"name", "orderNumber", "id"}; // the fields to bind do in your JavaBean
strat.setColumnMapping(columns);
CsvToBean csv = new CsvToBean();
List list = csv.parse(strat, yourReader);
I want to convert text file which is having tab and comma separated values into fully comma separated value in PIG. I am using Apache Pig version 0.11.1., I have tried with the following code and tried with FLATTEN, TOKENIZE. But I cannot make it into fully CSV file.
a = load '/home/mansoor/Documents/ip.txt' using PigStorage(',') as (key:chararray, val1:chararray, val2:chararray );
b = FOREACH a {
key= STRSPLIT(key,'\t');
GENERATE key;
}
Following is my text file input:
M12345 M123456,M234567,M987653
M23456 M23456,M123456,M234567
M34567 M234567,M765678,M987643
I need a file which is having fully CSV file like the following output:
M12345,M123456,M234567,M987653
M23456,M23456,M123456,M234567
M34567,M234567,M765678,M987643
How can I do this?
With pig 0.13, just using load without PigStorage made the csv be well loaded.
a = load '/home/mansoor/Documents/ip.txt';
dump a
gives me
(M12345,M123456,M234567,M987653)
(M23456,M23456,M123456,M234567)
(M34567,M234567,M765678,M987643 )
If that's not what you want, you might want to consider the REPLACE function.
Here is a quick and dirty solution to dispose of a usable csv :
a = load '/home/mansoor/Documents/ip.txt' using PigStorage('\n');
b = foreach a generate FLATTEN(REPLACE($0, '\t', ','));
store b into 'tmp.csv';
You can then use the csv as intended :
c = load 'tmp.csv' using PigStorage(',') as (key:chararray, val1:chararray, val2:chararray, val3:chararray);
describe c
gives c: {key: chararray,val1: chararray,val2: chararray, val3:chararray}
Try this,
a = load '/home/mansoor/Documents/ip.txt';
store a into '/home/mansoor/Documents/op' using PigStorage(',');
Now the file is fully converted into csv file.