The following ItemReader get a list of thousands accounts that need to be retrieved from MD database.
In this approach I am limited to the number of accounts that I can use:
#StepScope
#Bean
public ItemReader<OmsDto> itemReader(#Value("#{stepExecutionContext[accOms]}") List<String> notLoadedFiles) {
StringBuffer buffer = new StringBuffer();
notLoadedFiles.forEach(accountNumber -> buffer.append("'"+accountNumber+"',"));
buffer.replace(buffer.length()- 1, buffer.length(), "");
DriverManagerDataSource mdDataSource = new DriverManagerDataSource();
mdDataSource.setDriverClassName("prestosql");
mdDataSource.setUrl("jdbc:presto:....");
mdDataSource.setUsername(".....");
mdDataSource.setPassword("....");
String sql ="SELECT DISTINCT "
.....
.....
+ "FROM MD.ONLINE WHERE acct IN ";
JdbcCursorItemReader<OmsDto> reader = new JdbcCursorItemReader<OmsDto>();
reader.setVerifyCursorPosition(false);
reader.setDataSource(mdDataSource);
reader.setSql(sql);
reader.open(new ExecutionContext());
BeanPropertyRowMapper<OmsDto> rowMapper = new BeanPropertyRowMapper<>(OmsDto.class);
rowMapper.setPrimitivesDefaultedForNullValue(true);
reader.setRowMapper(rowMapper);
return reader;
}
What is the correct way to create dynamic IN Query (WHERE A IN (…, .., …)) ?
Thank you
Here is an example to generate IN query dynamically,
Example Query: SELECT * FROM USER WHERE ID IN (?,?,?,?,?)
List ids = List.of(1,2,3,4,5);
String inParams = String.join(",", ids.stream().map(id -> "?").collect(Collectors.toList()));
String query = String.format("SELECT * FROM USER WHERE ID IN (%s)", inParams);
Note that, if your query IN clause parameters limit more than 1000, it's better to use TEMP tables. Here some examples on github
Related
I'm using JpaPagingItemReader with Spring batch job to read the data from Database. I have a requirement where I can only write 10k records to external server in each try. I set chunk size and page size to 10000. Will this help to read, process and write only 10k records in each try until the JpaPagingItemReader is exhausted with response null and all the data from DB is read?
I'm confused between Pagesize and setMaxItemCount values. Which method will ensure to read the 10k records from DB?
Batch config Step:
public Step step1(StepBuilderFactory stepBuilderFactory, #Qualifier("itemReader") ItemReader<BatchRecords> ir,
#Qualifier("itemProcessor") ItemProcessor<BatchRecords, Map<String, Object>> ip,
#Qualifier("itemWriter") ItemWriter<Map<String, Object>> iw) throws Exception {
logger.info("Step1");
return stepBuilderFactory.get("step1").<BatchRecords, Map<String, Object>>chunk(10000).reader(ir).processor(ip)
.writer(iw).build();
}
ItemReader:
#Bean(destroyMethod="")
#StepScope
public JpaPagingItemReader<BatchRecords> itemReader(){
Map<String, Object> dates = batchUtil.dateFormat();
String startDate = (String) dates.get("StartDate");
String endDate = (String) dates.get("EndDate");
logger.info("Batch settlement Start date : {}, End Date: {}",startDate,endDate);
JpaPagingItemReader<BatchRecords> reader = new JpaPagingItemReader<BatchRecords>();
reader.setEntityManagerFactory(localContainerdbEntityManagerFactory.getNativeEntityManagerFactory());
reader.setQueryString("select b from BatchRecords b where b.status = 'O' and b.ordTimestamp between '" + startDate + "' and '" + endDate+ "' ");
reader.setPageSize(10000);
// reader.setMaxItemCount(50*reader.getPageSize());
try {
reader.afterPropertiesSet();
} catch (Exception e) {
logger.error("Exception in paging Item reader: "+e.getMessage());
}
return reader;
}```
You need to use maxItemCount if you want to limit the total number of items to read. I think it's even better to set the limit in you SQL query. PageSize will tell the reader how many items to read in each page (which is not a limit on the total number of items but a limit on the number of items to read when a new page is requested).
Any help would be greatly appreciated. I am working on a project using Spring JDBC for data access and am performing a simple query with an order by column expression, I am currently getting inconsistent results meaning the order by doesn't seem to be working. I have tried more than one database still no avail.
String sql = "select * from account where upper(name) like upper(:query) order by name asc";
MapSqlParameterSource params = new MapSqlParameterSource().addValue("query", "%" + query + "%");
List<Account> accountsSearched = namedParameterJdbcTemplate.query(sql, params, new BeanPropertyRowMapper<Account>(Account.class));
Any ideas what could be the issue?
So the problem is not within your SQL code, but problem exist in search method implementation
Existing Code
public List<Account> search(String uncleanedQuery, int offset) {
String query = uncleanedQuery.replaceAll("([-+.^:,])","");
String sql = "select * from account where upper(name) like upper(:query) order by name asc";
MapSqlParameterSource params = new MapSqlParameterSource().addValue("query", "%" + query + "%");
List<Account> accountsSearched = namedParameterJdbcTemplate.query(sql, params, new BeanPropertyRowMapper<Account>(Account.class));
Set<Account> accountSearchSet = new HashSet<Account>(accountsSearched);
List<Account> accounts = new ArrayList<Account>(accountSearchSet);
return accounts;
}
In the above code, we are fetching data correctly but assigning it to HashSet. HashSet does not respect ordering by name and generates random order for Account, due to which you are getting random order every time.
Solution 1:
There is no reason, you actually need Set. Using set just making your program slow. If you want to get DISTINCT data then modify SQL query.
public List<Account> search(String uncleanedQuery, int offset) {
String query = uncleanedQuery.replaceAll("([-+.^:,])","");
String sql = "select * from account where upper(name) like upper(:query) order by name asc";
MapSqlParameterSource params = new MapSqlParameterSource().addValue("query", "%" + query + "%");
List<Account> accountsSearched = namedParameterJdbcTemplate.query(sql, params, new BeanPropertyRowMapper<Account>(Account.class));
return accountsSearched;
}
Solution 2:
Still, you want to go with your approach then change code to use TreeSet and order based on the name
public List<Account> search(String uncleanedQuery, int offset) {
String query = uncleanedQuery.replaceAll("([-+.^:,])", "");
System.out.println("Search Query Called");
String sql = "select * from account where upper(name) like upper(:query) order by name";
MapSqlParameterSource params = new MapSqlParameterSource().addValue("query", "%" + query + "%");
List<Account> accountsSearched = namedParameterJdbcTemplate.query(sql, params,
new BeanPropertyRowMapper<Account>(Account.class));
Comparator<Account> comp = new Comparator<Account>() {
#Override
public int compare(Account a1, Account a2) {
return a1.getName().compareTo(a2.getName());
}
};
SortedSet<Account> accountSearchSet = new TreeSet<Account>(comp);
accountSearchSet.addAll(accountsSearched);
List<Account> accounts = new ArrayList<Account>(accountSearchSet);
return accounts;
}
New to Spring JDBC template but I'm wondering if I am able to pass a list of parameters and execute query once for each parameter in list. As I've seen many examples, the list of parameters being passed is for the execution of the query using all the parameters provided. Rather I am trying to execute query multiple times and for each time using new parameter in list.
For example:
Let's say I have a List of Ids - params (Strings)
List<String> params = new ArrayList<String>();
params.add("1234");
params.add("2345");
trying to do something like:
getJdbcTemplate().query(sql, params, new CustomResultSetExtractor());
which I know as per documentation is not allowed. I mean for one it has to be an array. I've seen simple examples where query is something like "select * from employee where id = ?" and they are passing new Object[]{"1234"} into method. And I'm trying to avoid the IN() condition. In my case each id will return multiple rows which is why I'm using ResultSetExtractor.
I know one option would be to iterate over list and include each id in list as a parameter, something like:
for(String id : params){
getJdbcTemplate().query(sql, new Object[]{id}, new CustomResultSetExtractor());
}
Just want to know if I can do this some other way. Sorry, I Should mention that I am trying to do a Select. Originally was hoping to return a List of custom objects for each resultset.
You do need to pass an array of params for the API, but you may also assume that your first param is an array. I believe this should work:
String sql = "select * from employee where id in (:ids)"; // or should there be '?'
getJdbcTemplate().query(sql, new Object[]{params}, new CustomResultSetExtractor());
Or you could explicitly specify, that the parameter is an array
getJdbcTemplate().query(sql, new Object[]{params}, new int[]{java.sql.Types.ARRAY}, new CustomResultSetExtractor());
You can use preparedStatement and do batch job:
eg. from http://docs.spring.io/spring/docs/current/spring-framework-reference/html/jdbc.html
public int[] batchUpdate(final List<Actor> actors) {
int[] updateCounts = jdbcTemplate.batchUpdate("update t_actor set first_name = ?, " +
"last_name = ? where id = ?",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, actors.get(i).getFirstName());
ps.setString(2, actors.get(i).getLastName());
ps.setLong(3, actors.get(i).getId().longValue());
}
public int getBatchSize() {
return actors.size();
}
});
return updateCounts;
}
I know you don't want to use the in clause, but I think its the best solution for your problem.
If you use a for in this way, I think it's not optimal.
for(String id : params){
getJdbcTemplate().query(sql, new Object[]{id}, new CustomResultSetExtractor());
}
I think it's a better solution to use the in clause. And then use a ResultSetExtractor to iterate over the result data. Your extractor can return a Map instead of a List, actually a Map of List.
Map<Integer, List<MyObject>>
Here there is a simple tutorial explaining its use
http://pure-essence.net/2011/03/16/how-to-execute-in-sql-in-spring-jdbctemplate/
I think this is the best solution:
public List<TestUser> findUserByIds(int[] ids) {
String[] s = new String[ids.length];
Arrays.fill(s, "?");
String sql = StringUtils.join(s, ',');
return jdbcTemplate.query(String.format("select * from users where id in (%s)", sql),
ArrayUtils.toObject(ids), new BeanPropertyRowMapper<>(TestUser.class));
}
this one maybe what you want. BeanPropertyRowMapper is just for example, it will be very slow when there's a lot of records. you should change it to another more efficient RowMapper.
NamedParameterJdbcTemplate namedParameterJdbcTemplate = new NamedParameterJdbcTemplate(jdbcTemplate.getDataSource());
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("STR_REF",(String)strRefkeys);
String sql = "SELECT TAGDESCRIPTION FROM xx, xx WHERE localeid='en_US' AND " +
"xx.TAGID=xx.RECORDID and TAGDESCRIPTION is not null AND xx.REFERENCEKEY in (:STR_REF)";
List<String> ifCurrent = namedParameterJdbcTemplate.queryForList(sql, params,String.class);
not getting any results in ifCurrent although run same query as SQL query and get results.
Am I passing any wrong params?
This is what is getting passed in strRefkeys
for(String refStr : strRefkeysLst) {
strRefkeysBuf.append("'");
strRefkeysBuf.append(refStr.toUpperCase());
strRefkeysBuf.append("',");
}
strRefkeys = strRefkeysBuf.toString();
if(strRefkeys.trim().length()>1){strRefkeys = strRefkeys.substring(0, strRefkeys.length()-1);}
You're passing a string parameter where a collection of values is expected. strRefKeys should be a set of accepted values for the REFERENCEKEY column.
i have table named gifts that contains field company_value_id and i want to make select for all company_value_id,count(company_value_id) so that the result will be list of object and each object will contain company_value_id,count(company_value_id)
i am using spring jpa with annotations as follows:
public interface GiftsRepository extends JpaRepository<Gifts, String> {
#Query("from Gifts g where g.companyGuid = :companyGuid")
List<Gifts> getGiftsByCompany(#Param("companyGuid") String companyGuid);
}
please advise, thanks.
i was able to accomplish it as follows:
#Query("select g.value.id,cr.value.name,count(g.value.id) from Gift g where g.user.id=:userId group by g.value")
List<Object[]> getUserGifts(
#Param("userId") String userId);
and in the service layer i extract the values as follows:
List<Object[]> results = giftsRepository
.getUserGifts(userId);
for (Object[] result : results) {
String id = (String) result[0];
String name = (String) result[1];
int count = ((Number) result[2]).intValue();
}
You need add a parameter to your function,just like this:
#Query("from Gifts g where g.companyGuid = :companyGuid")
List<Gifts> getGiftsByCompany(#Param("companyGuid") String companyGuid,Pageable pageable);
and the pageabel can be create like this:
Pageable pageable = new PageRequest(pageIndex, pageSize, Direction.ASC, sortColumn);