Parsing Pipe delimited file and Storing data into DB using Spring/Java - spring

I have a pipe delimited file (excel xlsx) that I need to parse for certain data. the data is all in column A. the first row has the date, the last row has the row count, and everything in between is the row data. I want to take the first three fields of each row and the date from the header and store it into my H2 Table. There is extra data in my file in each row. I Need help creating code that will parse the file and insert it into my db. I have a Entity and some code written but am stuck now.
My file
20200310|
Mn1223|w01192|windows|extra|extra|extra||
Sd1223|w02390|linux|extra|extra|extra||
2
My table
DROP TABLE IF EXISTS Xy_load ;
CREATE TABLE Xy_load (
account_name VARCHAR(250) NOT NULL,
command_name VARCHAR(250) NOT NULL,
system_name VARCHAR (250) NOT NULL,
CREATE_DT date (8) DEFAULT NULL
);
entity class
public class ZyEntity {
#Column(name="account_name")
private String accountName;
#Column(name="command_name")
private String commandName;
#Column(name="system_name")
private String systemName;
#Column(name="CREATE_DT")
private int createDt;
public ZyEntity(String accountName, String commandName, String systemName){
this.accountName=accountName;
this.commandName=commandName;
this.systemName=systemName;
}
public String getAccountName() {
return accountName;
}
public void setAccountName(String accountName) {
this.accountName = accountName;
}
public String getCommandName() {
return commandName;
}
public void setCommandName(String commandName) {
this.commandName = commandName;
}
public String getSystemName() {
return systemName;
}
public void setSystemName(String systemName) {
this.systemName = systemName;
}
public int getCreateDt() {
return createDt;
}
public void setCreateDt(int createDt) {
this.createDt = createDt;
}
}

i was able to figure it out with some help
List<DataToInsert> parseData(String filePath) throws IOException {
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new DataToInsert(val[0], val[1], val[2])).collect(Collectors.toList());
}
public void insertZyData(List<ZyEntity> parseData) {
String sql = "INSERT INTO Landing.load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
for (ZyEntity zyInfo : parseData){
SqlParameterSource source = new MapSqlParameterSource("account_name", zInfo.getAccountName())
.addValue("command_name", zyInfo.getCommandName())
.addValue("system_name", zyInfo.getSystemName());
jdbcTemplate.update(sql, source);
}
}

Related

FlatFileItemWriterBuilder-headerCallback() get number of rows written

Is it possible to get the total number of rows written from FlatFileItemWriter.headerCallback()?
I am a spring-batch nubee and I looked at putting count of lines into header of flat file and Spring Batch - Counting Processed Rows.
However I can't seem to implement the logic using the advice given there. It makes sense the writer count will only be available after the file is processed. However I am trying to get the row-count just before the file is officially written.
I tried to look for a hook like #AfterStep and grab the total rows, but I keep going in circles.
#Bean
#StepScope
public FlatFileItemWriter<MyFile> generateMyFileWriter(Long jobId,Date eventDate) {
String filePath = "C:\MYFILE\COMPLETED";
Resource file = new FileSystemResource(filePath);
DelimitedLineAggregator<MyFile> myFileLineAggregator = new DelimitedLineAggregator<>();
myFileLineAggregator.setDelimiter(",");
myFileLineAggregator.setFieldExtractor(getMyFileFieldExtractor());
return new FlatFileItemWriterBuilder<MyFile>()
.name("my-file-writer")
.resource(file)
.headerCallback(new MyFileHeaderWriter(file.getFilename()))
.lineAggregator(myFileLineAggregator)
.build();
}
private FieldExtractor<MyFile> getMyFileFieldExtractor() {
final String[] fieldNames = new String[]{
"typeRecord",
"idSystem"
};
return item -> {
BeanWrapperFieldExtractor<MyFile> extractor = new BeanWrapperFieldExtractor<>();
extractor.setNames(fieldNames);
return extractor.extract(item);
};
}
Notice I am using the MyFileHeaderWriter.java class(below) in the headerCallback(new MyFileHeaderWriter(file.getFilename())) (above). I am trying to initialize the value of qtyRecordsCreated below.
class MyFileHeaderWriter implements FlatFileHeaderCallback {
private final String header;
private String dtxCreated;
private String tmxCreated;
private String fileName;//15 byte file name private String qtyRecordsCreated;//number of rows in file including the header row
MyFileHeaderWriter(String sbfFileName) {
SimpleDateFormat dateCreated = new SimpleDateFormat("YYDDD");
SimpleDateFormat timeCreated = new SimpleDateFormat("HHMM");
Date now = new Date();
this.dtxCreated = dateCreated.format(now);
this.tmxCreated = timeCreated.format(now);
this.fileName = sbfFileName; this.qtyRecordsCreated="";
String[] headerValues = {dtxCreated,tmxCreated,fileName,qtyRecordsCreated};
this.header = String.join(",", headerValues);
}
#Override
public void writeHeader(Writer writer) throws IOException {
writer.write(header);
}
}
How can I get the number of rows in the header row?
Can the FlatFileFooterCallback be used to fetch the number of rows and then update the header with number of rows in the file afterwards?
You can achieve this in ItemProcessor, try this it work for me
public class EmployeeProcessor implements ItemProcessor<Employee, Employee> {
#Override
public Employee process(Employee employee) throws Exception {
return employee;
}
#AfterStep
public void afterStep(StepExecution stepExecution) {
ExecutionContext stepContext = stepExecution.getExecutionContext();
stepContext.put("count", stepExecution.getReadCount());
System.out.println("COUNT" + stepExecution.getReadCount());
}
}
And in you writer to get value
int count = stepContext.getInt("count");
Hope work for you

Dynamic Filtering of ArrayList with another ArrayList Java 8

I have a few Search Results Objects as given below:
public class TradeSearchResult{
private String tradeRefNo;
private String relatedTradeId;
private String custodyDate;
private String orderNumber;
private String odrQty;
private String price : 500;
}
public class CollateralTradesSearchResult{
private String excludeTradeOUT;
private String settlementStatus;
private String fundId;
private String altFundId;
private String apNumber;
private String collateralOrderNumber;
private String componenetIdentifier;
}
Now I have a Search Filter Criteria object
public class CRITERION {
protected String field; //The field denotes any field name of the either
SearchResult object
protected String operator; its will be EQUALS or NOT_EQUALS
protected String value; Value of the field.
}
Now I need to write a Dynamic Filter Method where I will pass the List of Criterion object and can pass List of either SearchResult like below
public static List<Object> applyFilter(List<CRITERION> comp, List<?> objectList){
//The CRITERION.fiedName can be same in more than one in the list
return filteredList;
}
here is one example: Consider the below List of
TradeSearchResult{
tradeRefNo : W12343;
relatedTradeId: N993093;
custodyDate : 2018-12-14;
orderNumber : 0000342343;
String odrQty : 12;
String price : 500;
},
{
tradeRefNo : W12344;
relatedTradeId: N993093;
custodyDate : 2018-12-14;
orderNumber : 0000342344;
String odrQty : 18;
String price : 600;
},
{
tradeRefNo : W12345;
relatedTradeId: N993094;
custodyDate : 2018-12-14;
orderNumber : 0000342345;
String odrQty : 20;
String price : 700;
}
Now the Criterion class is like
CRITERION{
field :relatedTradeId;
operator : EQUALS;
value :N993093;
}
{
field :relatedTradeId;
operator : EQUALS;
value :N993094;
}
{
field :orderNumber ;
operator : EQUALS;
value :0000342344;
}
It will only return one result even relatedTradeId has two filter
TradeSearchResult{
tradeRefNo : W12344;
relatedTradeId: N993093;
custodyDate : 2018-12-14;
orderNumber : 0000342344;
String odrQty : 18;
String price : 600;
}
Now in the same applyFIlter Method I can send a list of Criterion and a list of CollateralTradesSearchResult and returns filtered result.
Here is something I tried
public static List<Object> applyFilter(List<CRITERION> criList, List<?> objectList){
long startTime = Calendar.getInstance().getTimeInMillis();
Set<Object> objectSet = new HashSet<>();
for(CRITERION cri : criList){
String fieldName = cri.getFIELD();
objectList.stream().filter(p->beanProperties(p).get(fieldName).equals(cri.getVALUE())).forEachOrdered(objectSet::add);
//objectList.retainAll(objectSet);
//objectSet.clear();
}
List<Object> ret = new ArrayList<>(objectSet);
long endTime = Calendar.getInstance().getTimeInMillis();
System.out.println("Size"+ ret.size());
System.out.println("Time Taken to Search"+ String.valueOf(endTime-startTime));
return ret;
}
Hereis the beanProperties() method
public static Map<String, Object> beanProperties(Object bean) {
try {
Map<String, Object> map = new HashMap<>();
Arrays.asList(Introspector.getBeanInfo(bean.getClass(), Object.class)
.getPropertyDescriptors())
.stream()
// filter out properties with setters only
.filter(pd -> Objects.nonNull(pd.getReadMethod()))
.forEach(pd -> { // invoke method to get value
try {
Object value = pd.getReadMethod().invoke(bean);
if (value != null) {
map.put(pd.getName(), value);
}
} catch (Exception e) {
// add proper error handling here
}
});
return map;
} catch (IntrospectionException e) {
// and here, too
return Collections.emptyMap();
}
}
Any help using Stream or by any means will be helpful.
Create an interface SearchResult and make both your classes implement it. Then create this class:
public class Filter<T extends SearchResult> {
public List<T> applyFilter(List<Criterion> criteria, List<T> list) {
Map<String, Set<String>> allowedValues = new HashMap<>();
Map<String, Set<String>> prohibitedValues = new HashMap<>();
populateValues(criteria, "EQUALS", allowedValues);
populateValues(criteria, "NOT_EQUALS", prohibitedValues);
prohibitedValues.forEach((k, v) -> list.removeIf(t -> v.contains(getFieldValue(k, t))));
allowedValues.forEach((k, v) -> list.removeIf(t -> !v.contains(getFieldValue(k, t))));
return list;
}
private static void populateValues(List<Criterion> criteria, String operator, Map<String, Set<String>> values) {
criteria.stream()
.filter(c -> c.getOperator().equals(operator))
.forEach(c -> {
values.merge(c.getField(), Stream.of(c.getValue()).collect(Collectors.toSet()),
(set1, set2) -> Stream.concat(set1.stream(), set2.stream()).collect(Collectors.toSet()));
});
}
private String getFieldValue(String fieldName, T object) {
Field field;
try {
field = object.getClass().getDeclaredField(fieldName);
} catch (NoSuchFieldException e) {
e.printStackTrace();
return null;
}
field.setAccessible(true);
try {
return (String) field.get(object);
} catch (IllegalAccessException e) {
e.printStackTrace();
return null;
}
}
}
And use it as:
Filter<TradeSearchResult> filter = new Filter<>(); //or CollateralTradesSearchResult
List<TradeSearchResult> filteredList = filter.applyFilter(criteria, searchResults);

Compare column of csv in java

I'm reading a csv file with my 'CsvReader' like this:
public class CsvReader {
public static void main(String[] args) {
Map<String,Employee> empl = new HashMap <String,Employee>();
String csvFile= "C:/Users/rapurohi/Desktop/employee.csv";
BufferedReader br=null;
String line= "";
String csvSplitBy=",";
try{
br=new BufferedReader(new FileReader(csvFile));
while ((line =br.readLine())!=null){
String [] employee=line.split(csvSplitBy);
Employee e=new Employee(employee[0], employee[1], employee[2]);
empl.put((employee[0]),e);
System.out.println("Employee[id="+employee[0]+"name="+employee[1]+"salary="+employee[2]+"]" );
}
}catch (FileNotFoundException|IOException e){
e.printStackTrace();
}
}finally{
if(br!=null){
try{
br.close();
}catch(IOException e){
e.printStackTrace();
}
}
}
}
}
The csv i'm using looks like this:
empid,name,salary
"456","John","4000"
"457","James","5000"
"458","Jack","3000"
"459","Jenny","6000"
The dataVO for employee looks like
package com.cg.dto;
public class Employee {
String empId;
String empName;
String empSalary;
public String getEmpId() {
return empId;
}
public void setEmpId(String empId) {
this.empId = empId;
}
public String getEmpName() {
return empName;
}
public void setEmpName(String empName) {
this.empName = empName;
}
public String getEmpSalary() {
return empSalary;
}
public void setEmpSalary(String empSalary) {
this.empSalary = empSalary;
}
public Employee(String empId, String empName, String empSalary) {
super();
this.empId = empId;
this.empName = empName;
this.empSalary = empSalary;
}
I need to find the maximum salary and write the entry in a Csv file as output.
Assuming you only need help with finding the maximum salary, here's a quick example:
int maxSal = 0;
Employee highestPayedEmployee = null;
for (Employee employee : empl.values()) {
if(employee.getSalary() > maxSal){
maxSal = employee.getSalary();
highestPayedEmployee = employee;
}
}
This assumes that your Employee class has a getSalary() method that returns the salary as a number.
If needed you can convert a string to a number with:
Integer.parseInt(salaryString);

Insertion operation in SqlCe in windows phone

I am trying to insert some data into mydatabase but getting error like "Can't perform Create, Update, or Delete operations on 'Table(Dic)' because it has no primary key."
My database name is "condrokotha_new.sdf" and it has a table named "dic" which have 2 columns named "english" and "bangla". I made this database in another C# project in vs 2010. Then i used this database into my windowsphone project. I can show data from database but when i try to insert data i am getting error.
Here is my code:
public partial class MainPage : PhoneApplicationPage
{
condrokotha_newContext db = null;
// Constructor
public MainPage()
{
InitializeComponent();
db = new condrokotha_newContext(condrokotha_newContext.ConnectionString);
db.CreateIfNotExists();
db.LogDebug = true;
}
private void fav_Click(object sender, RoutedEventArgs e)
{
add_new_words("cintakhela","ami");
}
private void add_new_words(string e_word,string b_word)
{
using (condrokotha_newContext context = new condrokotha_newContext(condrokotha_newContext.ConnectionString))
{
Dic d = new Dic();
d.English = e_word;
d.Bangla = b_word;
context.Dics.InsertOnSubmit(d);
context.SubmitChanges();
}
}
}
My data context code like these :
public static string ConnectionString = "Data Source=isostore:/condrokotha_new.sdf";
public static string ConnectionStringReadOnly = "Data Source=appdata:/condrokotha_new.sdf;File Mode=Read Only;";
public static string FileName = "condrokotha_new.sdf";
public condrokotha_newContext(string connectionString) : base(connectionString)
{
OnCreated();
}
#region Extensibility Method Definitions
partial void OnCreated();
#endregion
public System.Data.Linq.Table<Dic> Dics
{
get
{
return this.GetTable<Dic>();
}
}
}
[global::System.Data.Linq.Mapping.TableAttribute(Name="dic")]
public partial class Dic
{
private string _English;
private string _Bangla;
public Dic()
{
}
[global::System.Data.Linq.Mapping.ColumnAttribute(Name="english", Storage="_English", DbType="NVarChar(1000)")]
public string English
{
get
{
return this._English;
}
set
{
if ((this._English != value))
{
this._English = value;
}
}
}
[global::System.Data.Linq.Mapping.ColumnAttribute(Name="bangla", Storage="_Bangla", DbType="NText", UpdateCheck=UpdateCheck.Never)]
public string Bangla
{
get
{
return this._Bangla;
}
set
{
if ((this._Bangla != value))
{
this._Bangla = value;
}
}
}
}
`
How can i insert my data into my database??
Is there anyone who can help in this??
For example you have this
SampleDataContextDataContext db = new SampleDataContextDataContext();
Employee emp = new Employee() {
FirstName = "Experts",
Lastname = "Comment",
Address = "rs.emenu#gmail.com"
}
db.Employees.InsertOnSubmit(emp);
db.SubmitChanges();
The above code will give you same error when u try to insert a new row. The reason is that LINQ does not provide the facility to insert data into a table without a primary key. At this point you have two options.
1.Create a store procedure and call it from LINQ.
SampleDataContextDataContext db = new SampleDataContextDataContext();
db.InsertEmployeeData("Experts","Comment", "rs.emenu#gmail.com");
Here InsertEmployeeData is a a stored procedure, and called it from the code.
2.Create an insert statement and execute using LINQ.
SampleDataContextDataContext db = new SampleDataContextDataContext();
string insertStatement = "Insert into Employee values('Experts', 'Comment','rs.emenu#gmail.com')";
db.ExecuteQuery<Employee>(insertStatement);
Here insert query is normal sql query and executed using the the LINQ ExecuteQuery method.

Trying to string.Join an IList and outputting the results to console

Using "string.Join(",", test);" works but for some reason I get an output of:
"Ilistprac.Location, Ilistprac.Location, Ilistprac.Location"
I tried ToString, Convert.ToString, etc and I still get that output.
All the IList interfaces are implemented with the IEnurmerable too (just not listed here unless someone wants me to).
class IList2
{
static void Main(string[] args)
{
string sSite = "test";
string sBldg = "test32";
string sSite1 = "test";
string sSite2 = "test";
Locations test = new Locations();
Location loc = new Location();
test.Add(sSite, sBldg)
test.Add(sSite1)
test.Add(sSite2)
string printitout = string.Join(",", test); //having issues outputting whats on the list
}
}
string printitout = string.Join(",", test.ToArray<Location>);
public class Location
{
public Location()
{
}
private string _site = string.Empty;
public string Site
{
get { return _site; }
set { _site = value; }
}
}
public class Locations : IList<Location>
{
List<Location> _locs = new List<Location>();
public Locations() { }
public void Add(string sSite)
{
Location loc = new Location();
loc.Site = sSite;
loc.Bldg = sBldg;
_locs.Add(loc);
}
private string _bldg = string.Empty;
public string Bldg
{
get { return _bldg; }
set { _bldg = value; }
}
}
You need to supply a useful ToString implementation for Location as Join is calling that for each element. The default implementation will just return the name of the type. See documentation.
So if you have a type like
class SomeType
{
public string FirstName { get; private set; }
public string LastName { get; private set; }
public SomeType(string first, string last)
{
FirstName = first;
LastName = last;
}
public override string ToString()
{
return string.Format("{0}, {1}", LastName, FirstName);
}
}
You need to specify how that should be represented as a string. If you do that, you can use string.Join like this to produce the output below.
var names = new List<SomeType> {
new SomeType("Homer", "Simpson"),
new SomeType("Marge", "Simpson")
};
Console.WriteLine(string.Join("\n", names));
Output:
Simpson, Homer
Simpson, Marge
You have to override ToString() inc your Location class to provide some meaningful output if you want to keep your current approach, E.g.:
public override string ToString()
{
return Site;
}

Resources