How do I set batch size in spring JDBC batch update to improve performance?
Listed below is my code snippet.
public void insertListOfPojos(final List<Student> myPojoList) {
String sql = "INSERT INTO " + "Student " + "(age,name) " + "VALUES "
+ "(?,?)";
try {
jdbcTemplateObject.batchUpdate(sql,
new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i)
throws SQLException {
Student myPojo = myPojoList.get(i);
ps.setString(2, myPojo.getName());
ps.setInt(1, myPojo.getAge());
}
#Override
public int getBatchSize() {
return myPojoList.size();
}
});
} catch (Exception e) {
System.out.println("Exception");
}
}
I read that with Hibernate you can provide your batch size in the
configuration xml.
For example,
<property name="hibernate.jdbc.batch_size" value="100"/>.
Is there something similar in Spring's jdbc?
There is no option for jdbc that looks like Hibernate; I think you have to get a look to specif RDBMS vendor driver options when preparing connection string.
About your code you have to use
BatchPreparedStatementSetter.getBatchSize()
or
JdbcTemplate.batchUpdate(String sql, final Collection<T> batchArgs, final int batchSize, final ParameterizedPreparedStatementSetter<T> pss)
if you use JDBC directly, you decide yourself how much statements are used in one commit, while using one of the provided JDBCWriters you decide the batch* size with the configured commit-rate
*afaik the actual spring version uses the prepared statement batch methods under the hood, see https://github.com/SpringSource/spring-framework/blob/master/spring-jdbc/src/main/java/org/springframework/jdbc/core/JdbcTemplate.java#L549
Related
I'm working on an spring-boot/jpa/ mysql project. Now so far everything worked with DateTime objects when fetching/storing objects with the repository.
The problem has now occured when I use the Jdbc Template to execute a custom sql query.
org.springframework.beans.ConversionNotSupportedException: Failed to convert property
value of type 'java.sql.Timestamp' to required type java.time.LocalDateTime' for
property 'from_time': no matching editors or conversion strategy found
The idea is to fetch Time slots (has a start time and duration in minutes) that are overlapping with a new incoming entry.
To get back my objects I was first using a BeanPropertyMapper and then switched to a custom NestedRowMapper.
The resulting conflicting time slots I want to get look like this:
{
id: 1
comment: "i worked 60minutes"
from_time: "2018-06-16 13:00"
duration_minutes: 60
task: {
name: "My task"
...
}
}
This is the method where I run into the issue:
public List<TimeSlot> getOverlappingEntries(TimeSlot timeslot) throws SQLException {
String sql = "SELECT time_slot.comment, time_slot.from_time,"
+ "DATE_ADD(from_time, INTERVAL duration_minutes MINUTE) AS end_time, "
+ " task.name as `task.name`, task.category as `task.category` "
+ " FROM `time_slot` " + " INNER JOIN task on task.id = time_slot.task_id "
+ " WHERE person_id = ? "
+ " HAVING ? < end_time AND DATE_ADD(? ,INTERVAL ? MINUTE) > from_time;";
PreparedStatementCreator prepared = (con) -> {
PreparedStatement prep = con.prepareStatement(sql);
prep.setObject(1, timeslot.person.id);
prep.setObject(2, timeslot.from_time);
prep.setObject(3, timeslot.from_time);
prep.setObject(4, timeslot.durationMinutes);
logger.info(prep.toString());
return prep;
};
return this.connector.query(prepared, NestedRowMapper.get(TimeSlot.class));
}
Now I would imagine spring is capable of converting those objects easily. And anyway there is the simple way of timestamp.toLocalDateTime() to do so. The problem seems more how to register this as a converter service or how to fix spring-boot configuration to do so.
I tried already a custom converter service but that didn't help:
#javax.persistence.Converter
public class SqlTimestampToLocalDateTimeConverter implements Converter<Timestamp,
LocalDateTime>, AttributeConverter<Timestamp, LocalDateTime> {
#Convert
#Override
public LocalDateTime convert(Timestamp source) {
return source.toLocalDateTime();
}
#Override
public LocalDateTime convertToDatabaseColumn(Timestamp attribute) {
return attribute.toLocalDateTime();
}
#Override
public Timestamp convertToEntityAttribute(LocalDateTime dbData) {
return Timestamp.valueOf(dbData);
}
}
Also many other answers on the internet mentioned that this was already implemented with spring framework 4.x.
The dependencies in the project look like this (build.gradle):
dependencies {
compile "org.springframework.boot:spring-boot-starter-thymeleaf:2.0.2.RELEASE"
compile "org.springframework.boot:spring-boot-starter-web:2.0.2.RELEASE"
compile "org.springframework.boot:spring-boot-starter-security:2.0.2.RELEASE"
compile "org.springframework.boot:spring-boot-starter-data-jpa:2.0.2.RELEASE"
compile "mysql:mysql-connector-java:5.1.46"
compileOnly "org.springframework.boot:spring-boot-devtools:2.0.2.RELEASE"
compile 'org.springframework.data:spring-data-rest-webmvc:3.0.7.RELEASE'
compile 'com.querydsl:querydsl-jpa:4.1.4'
compile 'com.querydsl:querydsl-apt:4.1.4:jpa'
testCompile("junit:junit")
testCompile("org.springframework.boot:spring-boot-starter-test")
testCompile("org.springframework.security:spring-security-test")
}
Thank you for any hints, how to solve this!
/edit:
I think I see a possible workaround now. What I could do is just to fetch the id's of all time slots and then use the repository to fetch the actual objects with their data (also their task objects).
But that feels definitely not like the optimal solution...
This is the NestedRowMapper I use:
import org.springframework.beans.*;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.jdbc.support.JdbcUtils;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
public class NestedRowMapper<T> implements RowMapper<T> {
private Class<T> mappedClass;
public static <T> NestedRowMapper<T> get(Class<T> mappedClass) {
return new NestedRowMapper<>(mappedClass);
}
public NestedRowMapper(Class<T> mappedClass) {
this.mappedClass = mappedClass;
}
#Override
public T mapRow(ResultSet rs, int rowNum) throws SQLException {
try {
T mappedObject = this.mappedClass.newInstance();;
BeanWrapper bw = PropertyAccessorFactory.forBeanPropertyAccess(mappedObject);
bw.setAutoGrowNestedPaths(true);
ResultSetMetaData meta_data = rs.getMetaData();
int columnCount = meta_data.getColumnCount();
for (int index = 1; index <= columnCount; index++) {
try {
String column = JdbcUtils.lookupColumnName(meta_data, index);
Object value = JdbcUtils.getResultSetValue(rs, index, Class.forName(meta_data
.getColumnClassName(index)));
bw.setPropertyValue(column, value);
} catch (TypeMismatchException | NotWritablePropertyException
| ClassNotFoundException e) {
e.printStackTrace();
}
}
return mappedObject;
} catch (InstantiationException | IllegalAccessException e1) {
throw new RuntimeException(e1);
}
}
}
You're on the right lines that you can define a RowMapper that tells your app what type of object each column needs to be mapped to. I would recommend trying to use JdbcTemplate.query method: https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jdbc/core/JdbcTemplate.html#query-java.lang.String-java.lang.Object:A-org.springframework.jdbc.core.RowMapper-
You will need to define a RowMapper (not necessarily a NestedRowMapper, you could try ParameterizedRowMapper), then pass that into query with your SQL and WHERE conditions mapped as args.
I think the bast way to use BeanPropertyRowMapper.newInstance(TimeSlot.class) in your getOverlappingEntries method
try this on NestedRowMapper.mapRow
if (value instanceof Timestamp) value = ((Timestamp) value).toLocalDateTime();
I have a spring boot application which is using gradle as build tool and jooq for dao class generation and db connection. Previously my application was connecting to single mysql instance. Below are the configuration we used for connecting to single db instance:
spring.datasource.username=user
spring.datasource.password=password
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.name=ds_name
spring.datasource.schema=ds_schema
spring.jooq.sql-dialect=MYSQL
Current project structure is
a) Main application project MainApp having application.properties with above key-value pairs.
b) Separate application project as DBProject which has jooq's generated DAO classes. MainApp include DBProject as a jar.
I am using gradle as build tool for this.
Everything is working fine till here. But now I have to connect to one more instance of MySQL. So, I have created another db project as DBProject2 which also contains dao classes generated by jooq using another mysql schema. I have created DBProject2 exactly as DBProject is created.
Now, my question is if I include both DBProjects in MainApp as jar then both will use same db configuration as in application.properties. How I can make separate db jars to point to their respective db schemas. I googled alot about this but couldn't find helpful solution.
This is what I do to connect to multiple (additional) data sources in my Play app. I am not sure if it is the best approach, but it works great for me. I have changed names below to be generic.
// In my application.conf
// default data source
db.default.driver=com.mysql.jdbc.Driver
db.default.url="jdbc:mysql://localhost:3306/myDb?useSSL=false"
db.default.username=myuser
db.default.password="mypassword"
// additional data source
db.anothersource.driver=com.mysql.jdbc.Driver
db.anothersource.url="jdbc:mysql://localhost:3306/myothersource?useSSL=false"
db.anothersource.username=myuser
db.anothersource.password="mypassword"
// Then in Java, I create a JooqContextProvider class to expose both connections.
public class JooqContextProvider {
#Inject
Database db;
#Inject
play.Configuration config;
public JooqContextProvider(){}
/**
* Creates a default database connection for data access.
* #return DSLConext.
*/
public DSLContext dsl() {
return DSL.using(new JooqConnectionProvider(db), SQLDialect.MYSQL);
}
/**
* Creates an anothersource database connection for data access.
* #return DSLConext for anothersource.
*/
public DSLContext anotherDsl() {
return DSL.using(
new JooqAnotherSourceConnectionProvider(
config.getString("db.anothersource.url"),
config.getString("db.anothersource.username"),
config.getString("db.anothersource.password")),
SQLDialect.MYSQL);
}
}
// Then I needed to implement my JooqAnotherSourceConnectionProvider
public class JooqAnotherSourceConnectionProvider implements ConnectionProvider {
private Connection connection = null;
String url;
String username;
String password;
public JooqAnotherSourceConnectionProvider(String url, String username, String password){
this.url = url;
this.username = username;
this.password = password;
}
#Override
public Connection acquire() throws DataAccessException {
try {
connection = DriverManager.getConnection(url, username, password);
return connection;
}
catch (java.sql.SQLException ex) {
throw new DataAccessException("Error getting connection from data source", ex);
}
}
#Override
public void release(Connection releasedConnection) throws DataAccessException {
if (connection != releasedConnection) {
throw new IllegalArgumentException("Expected " + connection + " but got " + releasedConnection);
}
try {
connection.close();
connection = null;
}
catch (SQLException e) {
throw new DataAccessException("Error closing connection " + connection, e);
}
}
}
// Then in Java code where I need to access one or the other data sources...
jooq.dsl().select().from().where()...
jooq.anotherDsl().select().from().where()...
I am trying to execute a very simple query from Spring JDBCTemplate. I am retrieving one attribute from a record that is identified by primary key. The entirely of the code is shown below. When I do this with a query constructed by concatenation (dangerous and ugly, and currently uncommented) it executes in 0.1 second. When I change my comments and use the parameterized query it executes in 50 seconds. I would much prefer to get the protection that comes with the parameterized query, however 50 seconds seems like a steep price to pay. Any hints how this could be made more reasonable.
public class JdbcEventDaoImpl {
private static JdbcTemplate jtemp;
private static PreparedStatement getJsonStatement;
private static final Logger logger = LoggerFactory.getLogger(JdbcEventDaoImpl.class);
#Autowired
public void setDataSource(DataSource dataSource) {
JdbcEventDaoImpl.jtemp = new JdbcTemplate(dataSource);
}
public String getJdbcForPosting(String aggregationId){
try {
return (String) JdbcEventDaoImpl.jtemp.queryForObject("select PostingJson from PostingCollection where AggregationId = '" + aggregationId + "'", String.class);
//return (String) JdbcEventDaoImpl.jtemp.queryForObject("select PostingJson from PostingCollection where AggregationId = ?", aggregationId, String.class);
} catch (EmptyResultDataAccessException e){
return "Not Available";
}
}
}
I'm new to spring. I'm working on a batch job that loads the data from a csv file to a sybase table. The input file is more than 5GB & has few millions of records.
I'm doing batch update using spring jdbctemplate. Below is my code snippet.
public int[] batchUpdate(final CsvReader products) throws IOException,
SQLException {
int[] updateCounts = getJdbcTemplate().batchUpdate(sqlStatement,
new AbstractInterruptibleBatchPreparedStatementSetter() {
#Override
public boolean setValuesIfAvailable(PreparedStatement pstmt, int i)
throws SQLException {
// logic to set the values to prepared statement etc...
#Override
public int getBatchSize() {
return batchSize;
}
});
}
}
I'm using apache dbcp datasource. I set the batchsize to 2000. I did not change anything else in the defaults for auto-commits etc..
Now, when I run the job.. it takes 4.5 min on an avg to insert 2000 records & the job runs for 2 days(didn't completed yet).
Can anyone suggest how this can be optimized? Thanks in advance.
I'm newbie with spring framework. I used spring.xml to define the datasouce and DataSourceTransactionManager, so that I could insert data with jdbctemplate object.
And now I want to add the rollback to the transaction.
Unfortunately this rollback only works for JdbcTemplate.updata (String SQL), not for JdbcTemplate.update(PreparedStatementCreator, Keyholder), which I used to get the generated ID by insert.
#Override
#Transactional("txManagerTest")
public SQLQueryObjectIF process(SQLQueryObjectIF queryObj) {
KeyHolder keyHolder = new GeneratedKeyHolder();
for (final String query : queryObj.getQueries()) {
System.out.println(query);
// Rollback works fine for the "update" below.
//jdbcTemplate.update(query);
// Rollback doesn't work for the "update" below. Don't why...
jdbcTemplate.update(new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(Connection con) throws SQLException {
jdbcTemplate.getDataSource().getConnection().setAutoCommit(false);
PreparedStatement ps = jdbcTemplate.getDataSource().getConnection().prepareStatement(query,Statement.RETURN_GENERATED_KEYS);
return ps;
}
}, keyHolder);
//log.info(keyHolder.getKeys().toString());
}
//just for rollback test
if (keyHolder.toString().length()>-1){
throw new RuntimeException("Test Error");
}
return queryObj;
}
That code should be used like this (you need to use the connection given as parameter), otherwise with your code you will get a connection that Spring doesn't know about, by directly accessing the DataSource instance (if Spring doesn't know about it, it will not know to rollback in case of exception):
public PreparedStatement createPreparedStatement(Connection con) throws SQLException {
PreparedStatement ps = con.prepareStatement(query,Statement.RETURN_GENERATED_KEYS);
return ps;
}