I am building a spring batch job that will be invoked through a webservice. The webservice will take a list of select and delete statement pairs. The records returned by the select statement will be saved as a CSV on the filesystem and then those same records will be deleted by executing the supplied delete statement.
I have seen a number of ColumnRowMapper examples but that requires me to create a POJO for each table entity. I am looking for a solution that will handle any column from any table. Any suggestions on approach?
****UPDATE****
Since writing this post, I've landed on the following solution.
#Bean
#StepScope
public JdbcCursorItemReader<Map<String, ?>> getRowsOfDataForExportFromTable(){
JdbcCursorItemReader<Map<String, ? extends Object>> databaseReader = new JdbcCursorItemReader<>();
databaseReader.setDataSource(jdbcTemplate.getDataSource());
databaseReader.setSql("select * from SOME_TABLE where last_updated_date < DATE_SUB(NOW(), INTERVAL 10 DAY);");
databaseReader.setRowMapper(new RowMapper<Map<String, ? extends Object>>() {
#Override
public Map<String, ? extends Object> mapRow(ResultSet resultSet, int i) throws SQLException {
Map<String,String> resultMap = new LinkedHashMap<>();
int numOfColumns = resultSet.getMetaData().getColumnCount();
for (int j = 1; j < numOfColumns+1; j++){
String columnName = resultSet.getMetaData().getColumnName(j);
String value = resultSet.getString(j);
resultMap.put(columnName,value);
}
return resultMap;
}
});
return databaseReader;
}
The above ItemReader will build a LinkedHashMap row mapper where the column name is the key and the column value is the value.
Did you try to use Map instead of POJO? You can dynamically fill it in Reader, and then create CSV file from this Map.
Related
Im inserting a CSV file in a database table using Spring Batch and Spring Data (and Hibernate).
each time I insert the CSV I have to delete the previous data in the table using the data-jpa deleteAll() method. the problem is that the ids of the table are incremented automatically and continuously (#GeneratedValue(strategy = GenerationType.IDENTITY)) after each delete/insert statement.
I want that after each delete the ids start on 1. the only way that I found to do that is by altering the index (i know its not the best way, so your suggestions are welcomed)
the Question is :
is there any method to run this SQL request
ALTER TABLE res AUTO_INCREMENT=1;
in Java object using Spring Data or Hibernate?
Thanks
Is it possible to generate id on java side and do not use embedded db autoincrement feature?
So the best way will be to generate id explicitly and set it to entity.
Other cases are:
Truncate Table
TRUNCATE TABLE table_name;
This will reset the auto increment on the table as well as deleting all records from that table.
Drop and Recreate
Table DROP TABLE table_name;
CREATE TABLE table_name { ... };
So I think, second is what are you looking for
Instead of Altering the table, I have customized the way that Hibernate Generates the Ids.
instead of using :
#GeneratedValue(strategy = GenerationType.IDENTITY)
I have implemented a custom id generator :
#GenericGenerator(name = "sequence_id", strategy =
"com.xyz.utils.CustomIdGenerator",
parameters = {
#org.hibernate.annotations.Parameter(
name = "table_name", value = "myTable")
})
#GeneratedValue(generator = "sequence_id")
the CustomIdGenerator class :
public class CustomIdGenerator implements IdentifierGenerator,Configurable{
private String table_name ;
#Override
public Serializable generate(SharedSessionContractImplementor session, Object object)
throws HibernateException {
Connection connection = session.connection();
try {
Statement statement=connection.createStatement();
ResultSet rs=statement.executeQuery("select count(id) as Id from "
+table_name );
if(rs.next())
{
int id=rs.getInt(1)+1;
Integer generatedId = new Integer(id);
System.out.println("Generated Id: " + generatedId);
return generatedId;
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
#Override
public void configure(Type type, Properties params, ServiceRegistry serviceRegistry)
throws MappingException {
setTable_name(params.getProperty("table_name")); }
//getters and setters
}
the problem of this solution is the execution of select for each id so it seems generating load on the DBMS and its slow.
and the rs is looping twice for the first id ()
any suggestion for optimization is welcomed
I'm trying to persist a SparseArray in a Room database and can not get it to compile. I keep getting the "Not sure how to convert a Cursor to this method's return type" error message along with "The query returns some columns [plannerLineData] which are not use by android.util.SparseArray."
I have tried using a single field in the PlannerLine Entity alone with a separate PlannerLineData class.
I have data converters to convert SparseArray to String and to convert String back to SparseArray.
I have checked several questions on stackoverflow and have successfully used the Date to Long and the Long to Date converters in other projects, but I seem to be missing something somewhere.
Data Files:
#Entity
public class PlannerLine implements Serializable {
private static final long serialVersionUID = 1L;
#TypeConverters(Converters.class)
#PrimaryKey
#SerializedName("planner_line")
#NonNull
public SparseArray plannerLineData;
public SparseArray getPlannerLineData() {
return plannerLineData;
}
public void setPlannerLineData(SparseArray plannerLineData) {
this.plannerLineData = plannerLineData;
}
public class PlannerLineData implements Serializable {
#SerializedName("lineId")
public int lineId;
#SerializedName("plan_text")
public String planText;
public int getLineId() {
return lineId;
}
public void setLineId(int lineId) {
this.lineId = lineId;
}
public String getPlanText() {
return planText;
}
public void setPlanText(String planText) {
this.planText = planText;
}
}
DAO problem area:
#Dao
public interface PlannerDao {
#Query("SELECT * from PlannerLine")
public SparseArray getPlannerLine(); <---Doesn't like this line
I have also tried returning SparseArray<PlannerLine> and SparseArray<PlannerLineData>, but no joy.
Converters class:
public class Converters {
#TypeConverter
public static String sparseArrayToString(SparseArray sparseArray) {
if (sparseArray == null) {
return null;
}
int size = sparseArray.size();
if (size <= 0) {
return "{}";
}
StringBuilder buffer = new StringBuilder(size * 28);
buffer.append('{');
for (int i = 0; i < size; i++) {
if (i > 0) {
buffer.append("-,- ");
}
int key = sparseArray.keyAt(i);
buffer.append(key);
buffer.append("-=-");
Object value = sparseArray.valueAt(i);
buffer.append(value);
}
buffer.append('}');
return buffer.toString();
}
#TypeConverter
public static SparseArray stringToSparseArray(String string) {
if (string == null) {
return null;
}
String entrySeparator = "-=-";
String elementSeparator = "-,-";
SparseArray sparseArray = new SparseArray();
String[] entries = StringUtils.splitByWholeSeparator(string, elementSeparator);
for (int i = 0; i < entries.length; i++) {
String[] parts = StringUtils.splitByWholeSeparator(entries[i], entrySeparator);
int key = Integer.parseInt(parts[0]);
String text = parts[1];
sparseArray.append(key, text);
}
return sparseArray;
}
Suggestions would be appreciated. Thanks
Edit:
My original vision for this app was to store all the plan lines in a single SparseArray, along with two additional SparseIntArrays (which I did not mention before because the solution would be similar to the SparseArray) to hold info on how the plan lines interact with each other.
After reading through #dglozano's helpful responses, I have decided to re-design the app to just store regular DB files in Room and load the data into the SparseArray (and the two SparseIntArrays) at startup, use only the in memory SparseArray and SparseIntArrays while the app is active, then write changes in the Sparse Arrays to the DB during onStop(). I am also considering updating the DB in the background as I work through app.
Because the answers and suggestions provided by #dglozano led me to the re-design decision, I am accepting his answer as the solution.
Thanks for the help.
It seems that you are doing the Conversion properly. However, the problem is in your DAO Query:
#Query("SELECT * from PlannerLine") // This returns a List of PlannerLine, not a SparseArray
public SparseArray getPlannerLine(); // The return type is SparseArray, not a List of PlannerLine
Therefore, you can try two different things:
1 - Change the Query to #Query("SELECT plannerLineData FROM PlannerLine WHERE lineId == :lineId") , so that the query returns the SparseArray inside the PlannerLine with id lineId. You should change the method signature so it accepts the parameter lineId
#Query("SELECT plannerLineData FROM PlannerLine WHERE lineId == :lineId")
public SparseArray getPlannerLine(int lineId);
2 - If you want to return the full PlannerLine object and then access to its SparseArray field, then you should change the return type. You should also add the lineId parameter to return just one record and not a list of all the PlannerLine stored in the database table.
#Query("SELECT * FROM PlannerLine WHERE lineId == :lineId")
public PlannerLine getPlannerLine(int lineId);
UPDATE
If you want to get a List<PlannerLine> with all the PlannerLine stored in the database, use the following query in your Dao.
#Query("SELECT * FROM PlannerLine")
public List<PlannerLine> getAllPlannerLines();
Then you can access to the SparseArray of each PlannerLine in the list as usual.
How can I read a subset of fields from an avro-parquet file in java?
I thought I could define an avro schema which is a subset of the stored records and then read them...but I get an exception.
here is how i tried to solve it
I have 2 avro schemas:
classA
ClassB
The fields of ClassB are a subset of ClassA.
final Builder<ClassB> builder = AvroParquetReader.builder(files[0].getPath());
final ParquetReader<ClassB> reader = builder.build();
//AvroParquetReader<ClassA> readerA = new AvroParquetReader<ClassA>(files[0].getPath());
ClassB record = null;
final List<ClassB> list = new ArrayList<>();
while ((record = reader.read()) != null) {
list.add(record);
}
But I get a ClassCastException on line (record=reader.read()): Cannot convert ClassA to ClassB
I suppose the reader is reading the schema from the file.
I tried to send in the model (i.e. builder.withModel) but since classB extends org.apache.avro.specific.SpecificRecordBase it throws an exception.
I event tried to set the schema in the configuration and set it through builder.withConfig but no cigar...
So...
Couple of things:
AvroReadSupport.setRequestedProjection(hadoopConf, ClassB.$Schema) can be used to set a projection for the columns that are selected.
The reader.readNext method still will return a ClassA object but will null out the fields that are not present in ClassB.
To use the reader directly you can do the following:
AvroReadSupport.setRequestedProjection(hadoopConf, ClassB.SCHEMA$);
final Builder<ClassB> builder = AvroParquetReader.builder(files[0].getPath());
final ParquetReader<ClassA> reader = builder.withConf(hadoopConf).build();
ClassA record = null;
final List<ClassA> list = new ArrayList<>();
while ((record = reader.read()) != null) {
list.add(record);
}
Also if you're planning to use an inputformat to read the avro-parquet file, there is a convenience method - here is a spark example:
final Job job = Job.getInstance(hadoopConf);
ParquetInputFormat.setInputPaths(job, pathGlob);
AvroParquetInputFormat.setRequestedProjection(job, ClassB.SCHEMA$);
#SuppressWarnings("unchecked")
final JavaPairRDD<Void, ClassA> rdd = sc.newAPIHadoopRDD(job.getConfiguration(), AvroParquetInputFormat.class,
Void.class, ClassA.class);
I worked on both row mapper and resultset extractor call back interfaces.I found difference i.e.,
1.Row mapper can be processing per row basis.But Resultset extractor we can naviagte all rows and return type is object.
Is there any difference other than above?.How the works Rowmapper internal and return type is list?.
Basic difference is with ResultsetExtractor you will need to iterate through the result set yourself, say in while loop.
This interface provides you processing of the entire ResultSet at once. The implemetation of Interface method extractData(ResultSet rs) will contain that manual iteration code.
See one implementation of ResultsetExtractor
while some callback handlers like RowCallbackHandler, the interface method processRow(ResultSet rs) loops for you.
RowMapper can be used both was for mapping each row, or entire rows.
For entire rows Object (by template method jdbcTemplate.query())
public List findAll() {
String sql = "SELECT * FROM EMPLOYEE";
return jdbcTemplate.query(sql, new EmployeeRowMapper());
}
without casting will work
For individual object (with Template method jdbcTemplate.queryForObject())
#SuppressWarnings({ "unchecked", "rawtypes" })
public Employee findById(int id) {
String sql = "SELECT * FROM EMPLOYEE WHERE ID = ?";
// jdbcTemplate = new JdbcTemplate(dataSource);
Employee employee = (Employee) jdbcTemplate.queryForObject(sql, new EmployeeRowMapper(), id );
// Method 2 very easy
// Employee employee = (Employee) jdbcTemplate.queryForObject(sql, new Object[] { id }, new BeanPropertyRowMapper(Employee.class));
return employee;
}
#SuppressWarnings("rawtypes")
public class EmployeeRowMapper implements RowMapper {
public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
Employee employee = new Employee();
employee.setId(rs.getInt("ID"));
employee.setName(rs.getString("NAME"));
employee.setAge(rs.getInt("AGE"));
return employee;
}
}
Best Use cases:
Row Mapper: When each row of a ResultSet maps to a domain Object, can be implemented as private inner class.
RowCallbackHandler: When no value is being returned from callback method for each row, e.g. writing row to a file, converting rows to a XML, Filtering rows before adding to collection. Very efficient as ResultSet to Object mapping is not done here.
ResultSetExtractor: When multiple rows of ResultSet map to a single Object. Like when doing complex joins in a query one may need to have access to entire ResultSet instead of single row of rs to build complex Object and you want to take full control of ResultSet. Like Mapping the rows returned from the join of TABLE1 and TABLE2 to an fully-reconstituted TABLE aggregate.
ParameterizedRowMapper is used to create complex objects
JavaDoc of ResultSetExtractor:
This interface is mainly used within the JDBC framework itself. A RowMapper is usually a simpler choice for ResultSet processing, mapping one result object per row instead of one result object for the entire ResultSet.
ResultSetExtractor is suppose to extract the whole ResultSet (possibly multiple rows), while RowMapper is feeded with row at a time.
Most the time, ResultSetExtractor will loop the ResultSet and use RowMapper, snippet example of Spring RowMapperResultSetExtractor:
List<T> results = (this.rowsExpected > 0 ? new ArrayList<T>(this.rowsExpected) : new ArrayList<T>());
int rowNum = 0;
while (rs.next()) {
results.add(this.rowMapper.mapRow(rs, rowNum++));
}
return results;
Pay attention, ALL results will be transformed, this can create Out Of Memory exception.
See also
RowMapperResultSetExtractor
RowMapper: To process one record of ResultSet at a time.
ResultSetExtractor: To process multiple records of ResultSet at a time.
I think one place where a ResultSetExtractor could be advantageous is when you have a result set (like from a call to a stored procedure) and a row mapper, and want to process them like is done under the covers in the jdbcTemplate methods, such as query(String sql, RowMapper rowMapper). In this case you can save yourself from having to manually iterate over the result set by using the ResultSetExtractor instead of just the RowMapper.
For example:
RowMapper
ResultSet resultSet = cs.executeQuery();
int row = 0;
DateRowMapper dateRowMapper = new DateRowMapper();
List<String> dates = new ArrayList<>();
while (resultSet.next()) {
dates.add(dateRowMapper.mapRow(resultSet, ++row));
}
return dates;
ResultSetExtractor
ResultSet resultSet = callableStatement.executeQuery();
return new RowMapperResultSetExtractor<>(new DateRowMapper()).extractData(resultSet);
Can anyone help me with an example of ColumnMapRowMapper? How to use it?
I've written an answer in my blog, http://selvam2day.blogspot.com/2013/06/singlecolumnrowmapper.html, but here it is for your convenience below:
SingleColumnRowMapper & ColumnMapRowMapper examples in Spring
Spring JDBC includes two default implementations of RowMapper - SingleColumnRowMapper and ColumnMapRowMapper. Below are sample usages of those row mappers.
There are lots of situations when you just want to select one column or only a selected set of columns in your application, and to write custom row mapper implementations for these scenarios doesn't seem right. In these scenarios, we can make use of the spring-provided row mapper implementations.
SingleColumnRowMapper
This class implements the RowMapper interface. As the name suggests, this class can be used to retrieve a single value from the database as a java.util.List. The list contains the column values one per each row.
In the code snippet below, the type of the result value for each row is specified by the constructor argument. It can also be specified by invoking the setRequiredType(Class<T> requiredType) method.
public List getFirstName(int userID)
{
String sql = "select firstname from users where user_id = " + userID;
SingleColumnRowMapper rowMapper = new SingleColumnRowMapper(String.class);
List firstNameList = (List) getJdbcTemplate().query(sql, rowMapper);
for(String firstName: firstNameList)
System.out.println(firstName);
return firstNameList;
}
More information on the class and its methods can be found in the spring javadoc link below.
http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/jdbc/core/SingleColumnRowMapper.html
ColumnMapRowMapper
ColumnMapRowMapper class can be used to retrieve more than one column from a database table. This class also implements the RowMapper interface. This class creates a java.util.Map for each row, representing all columns as key-value pairs: one entry for each column, with the column name as key.
public List<Map<String, Object>> getUserData(int userID)
{
String sql = "select firstname, lastname, dept from users where userID = ? ";
ColumnMapRowMapper rowMapper = new ColumnMapRowMapper();
List<Map<String, Object>> userDataList = getJdbcTemplate().query(sql, rowMapper, userID);
for(Map<String, Object> map: userDataList){
System.out.println("FirstName = " + map.get("firstname"));
System.out.println("LastName = " + map.get("lastname"));
System.out.println("Department = " + map.get("dept"));
}
return userDataList;
}
More information on the class and its methods can be found in the spring javadoc link below.
http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/jdbc/core/ColumnMapRowMapper.html