How to query data via Spring data JPA with user defined offset and limit (Range) - spring

Is it possible to fetch data in user defined ranges [int starting record -int last record]?
In my case user will define in query String in which range he wants to fetch data.
I have tried something like this
Pageable pageable = new PageRequest(0, 10);
Page<Project> list = projectRepository.findAll(spec, pageable);
Where spec is my defined specification but unfortunately this do not help.
May be I am doing something wrong here.
I have seen other spring jpa provided methods but nothing are of much help.
user can enter something like this localhost:8080/Section/employee? range{"columnName":name,"from":6,"to":20}
So this says to fetch employee data and it will fetch the first 15 records (sorted by columnName ) does not matter as of now.
If you can suggest me something better that would be great.if you think I have not provided enough information please let me know, I will provide required information.
Update :I do not want to use native or Create query statements (until I don't have any other option).
May be something like this:
Pageable pageable = new PageRequest(0, 10);
Page<Project> list = projectRepository.findAll(spec, new pageable(int startIndex,int endIndex){
// here my logic.
});
If you have better options, you can suggest me that as well.
Thanks.

Your approach didn't work, because new PageRequest(0, 10); doens't do what you think. As stated in docs, the input arguments are page and size, not limit and offset.
As far as I know (and somebody correct me if I'm wrong), there is no "out of the box" support for what you need in default SrpingData repositories. But you can create custom implementation of Pagable, that will take limit/offset parameters. Here is basic example - Spring data Pageable and LIMIT/OFFSET

We can do this with Pagination and by setting the database table column name, value & row counts as below:
#Transactional(readOnly=true)
public List<String> queryEmployeeDetails(String columnName,String columnData, int startRecord, int endRecord) {
Query query = sessionFactory.getCurrentSession().createQuery(" from Employee emp where emp.col= :"+columnName);
query.setParameter(columnName, columnData);
query.setFirstResult(startRecord);
query.setMaxResults(endRecord);
List<String> list = (List<String>)query.list();
return list;
}

If I am understanding your problem correctly, you want your repository to allow user to
Provide criteria for query (through Specification)
Provide column to sort
Provide the range of result to retrieve.
If my understanding is correctly, then:
In order to achieve 1., you can make use of JpaSpecificationExecutor from Spring Data JPA, which allow you to pass in Specificiation for query.
Both 2 and 3 is achievable in JpaSpecificationExecutor by use of Pagable. Pageable allow you to provide the starting index, number of record, and sorting columns for your query. You will need to implement your range-based Pageable. PageRequest is a good reference on what you can implement (or you can extend it I believe).

So i got this working as one of the answer suggested ,i implemented my own Pageable and overrided getPagesize(),getOffset(),getSort() thats it.(In my case i did not need more)
public Range(int startIndex, int endIndex, String sortBy) {
this.startIndex = startIndex;
this.endIndex = endIndex;
this.sortBy = sortBy;
}
#Override
public int getPageSize() {
if (endIndex == 0)
return 0;
return endIndex - startIndex;
}
#Override
public int getOffset() {
// TODO Auto-generated method stub
return startIndex;
}
#Override
public Sort getSort() {
// TODO Auto-generated method stub
if (sortBy != null && !sortBy.equalsIgnoreCase(""))
return new Sort(Direction.ASC, sortBy);
else
return new Sort(Direction.ASC, "id");
}
where startIndex ,endIndex are starting and last index of record.
to access it :
repository.findAll(spec,new Range(0,20,"id");

There is no offset parameter you can simply pass. However there is a very simple solution for this:
int pageNumber = Math.floor(offset / limit) + ( offset % limit );
PageRequest pReq = PageRequest.of(pageNumber, limit);
The client just have to keep track on the offset instead of page number. By this I mean your controller would receive the offset instead of the page number.
Hope this helps!

Related

Memory leak with Criteria API Pageable

I implemented pageable functionality into Criteria API query and I noticed increased memory usage during query execution. I also used spring-data-jpa method query to return same result, but there memory is cleaned up after every batch is processed. I tried detaching, flushing, clearing objects from EntityManager, but memory use would keep going up, occasionally it will drop but not as much as with method queries. My question is what could cause this memory use if objects are detached and how to deal with it?
Memory usage with Criteria API pageable:
Memory usage with method query:
Code
Since I'm also updating entities retrieved from DB, I use approach where I save ID of last processed entity, so when entity gets updated query doesen't skip next selected page. Below I provide code example that is not from real app I'm working on, but it just recreation of the issue I'm having.
Repository code:
#Override
public Slice<Player> getPlayers(int lastId, Pageable pageable) {
List<Predicate> predicates = new ArrayList<>();
CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder();
CriteriaQuery<Player> criteriaQuery = criteriaBuilder.createQuery(Player.class);
Root<Player> root = criteriaQuery.from(Player.class);
predicates.add(criteriaBuilder.greaterThan(root.get("id"), lastId));
criteriaQuery.where(criteriaBuilder.and(predicates.toArray(Predicate[]::new)));
criteriaQuery.orderBy(criteriaBuilder.asc(root.get("id")));
var query = entityManager.createQuery(criteriaQuery);
if (pageable.isPaged()) {
int pageSize = pageable.getPageSize();
int offset = pageable.getPageNumber() > 0 ? pageable.getPageNumber() * pageSize : 0;
// Fetch additional element and skip it based on the pageSize to know hasNext value.
query.setMaxResults(pageSize + 1);
query.setFirstResult(offset);
var resultList = query.getResultList();
boolean hasNext = pageable.isPaged() && resultList.size() > pageSize;
return new SliceImpl<>(hasNext ? resultList.subList(0, pageSize) : resultList, pageable, hasNext);
} else {
return new SliceImpl<>(query.getResultList(), pageable, false);
}
}
Iterating through pageables:
#Override
public Slice<Player> getAllPlayersPageable() {
int lastId = 0;
boolean hasNext = false;
Pageable pageable = PageRequest.of(0, 200);
do {
var players = playerCriteriaRepository.getPlayers(lastId, pageable);
if(!players.isEmpty()){
lastId = players.getContent().get(players.getContent().size() - 1).getId();
for(var player : players){
System.out.println(player.getFirstName());
entityManager.detach(player);
}
}
hasNext = players.hasNext();
} while (hasNext);
return null;
}
I think you are running into a query plan cache issue here that is related to the use of the JPA Criteria API and how numeric values are handled. Hibernate will render all numeric values as literals into an intermediary HQL query string which is then compiled. As you can imagine, every "scroll" to the next page will be a new query string so you gradually fill up the query plan cache.
One possible solution is to use a library like Blaze-Persistence which has a custom JPA Criteria API implementation and a Spring Data integration that will avoid these issues and at the same time improve the performance of your queries due to a better pagination implementation.
All your code would stay the same, you just have to include the integration and configure it as documented in the setup section.

Spring data JDBC query creation with pagination complains IncorrectResultSizeDataAccessException: Incorrect result size

I'm struggling to trying the pagination feature, as described in the reference document.
This is my table schema:
CREATE TABLE cities
(
id int PRIMARY KEY,
name varchar(255),
pref_id int
);
Repository:
public interface CityRepository extends CrudRepository<CityEntity, Integer> {
Page<CityEntity> findAll(Pageable pageable);
// get all cities in the prefecture
Page<CityEntity> findByPrefId(Integer prefId, Pageable pageable);
}
Test code:
Page<CityEntity> allCities = repository.findAll(PageRequest.of(0, 10));
Page<CityEntity> cities = repository.findByPrefId(1, PageRequest.of(0, 10));
findAll works well, but findByPrefId throws the following error:
Incorrect result size: expected 1, actual 10
org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 10
at org.springframework.dao.support.DataAccessUtils.nullableSingleResult(DataAccessUtils.java:100)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForObject(NamedParameterJdbcTemplate.java:237)
at org.springframework.data.jdbc.repository.query.AbstractJdbcQuery.lambda$singleObjectQuery$1(AbstractJdbcQuery.java:115)
at org.springframework.data.jdbc.repository.query.PartTreeJdbcQuery.execute(PartTreeJdbcQuery.java:98)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor$QueryMethodInvoker.invoke(QueryExecutorMethodInterceptor.java:195)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:130)
...
If I change the method signature into List<CityEntity> findByPrefId(Integer prefId, Pageable pageable), it works.
Am I missing something? I'm using the latest version of spring-data-jdbc (2.0.2.RELEASE).
I don't know about the technicality, but this is what I learned from experience.
In your case, if the total number of cities is lesser than the pageable.getPageSize(), then your repository will return a List<>.
But if total number of cities is bigger than the pageable.getPageSize() then your repository will return a Page<>.
Knowing that, this is what I did to work around it.
Long amount = repository.countByPrefId(prefId);
if(pagination.getPageSize()>amount ) {
List<CityEntity> list = repository.findByPrefId(prefId);
} else {
Page<CityEntity> pages = repository.findByPrefId(person, PageRequest.of(0, 10));
}
This also means that in your repository you'll have two differents methods, one with Pageable as a parameter and one with only PrefId as a parameter.
I believe the accepted answer is referring to Spring Data JPA which does work by returning pages based on a count query derived from the custom query OR manually set via countQuery, no reason for the if/else.
However this flat out does not work in Spring Data JDBC.
https://jira.spring.io/browse/DATAJDBC-554
Workaround provided in link but for reference:
interface FooRepository extends PagingAndSortingRepository<FooEntity, Long> {
List<FooEntity> findAllByBar(String bar, Pageable pageable);
Long countAllByBar(String bar);
}
And then combining those 2 queries like this:
List<FooEntity> fooList = repository.findAllByBar("...", pageable);
Long fooTotalCount = repository.countAllByBar("...");
Page<FooEntity> fooPage = PageableExecutionUtils.getPage(fooList, pageable, () -> fooTotalCount);

Using spring jdbc template to query for list of parameters

New to Spring JDBC template but I'm wondering if I am able to pass a list of parameters and execute query once for each parameter in list. As I've seen many examples, the list of parameters being passed is for the execution of the query using all the parameters provided. Rather I am trying to execute query multiple times and for each time using new parameter in list.
For example:
Let's say I have a List of Ids - params (Strings)
List<String> params = new ArrayList<String>();
params.add("1234");
params.add("2345");
trying to do something like:
getJdbcTemplate().query(sql, params, new CustomResultSetExtractor());
which I know as per documentation is not allowed. I mean for one it has to be an array. I've seen simple examples where query is something like "select * from employee where id = ?" and they are passing new Object[]{"1234"} into method. And I'm trying to avoid the IN() condition. In my case each id will return multiple rows which is why I'm using ResultSetExtractor.
I know one option would be to iterate over list and include each id in list as a parameter, something like:
for(String id : params){
getJdbcTemplate().query(sql, new Object[]{id}, new CustomResultSetExtractor());
}
Just want to know if I can do this some other way. Sorry, I Should mention that I am trying to do a Select. Originally was hoping to return a List of custom objects for each resultset.
You do need to pass an array of params for the API, but you may also assume that your first param is an array. I believe this should work:
String sql = "select * from employee where id in (:ids)"; // or should there be '?'
getJdbcTemplate().query(sql, new Object[]{params}, new CustomResultSetExtractor());
Or you could explicitly specify, that the parameter is an array
getJdbcTemplate().query(sql, new Object[]{params}, new int[]{java.sql.Types.ARRAY}, new CustomResultSetExtractor());
You can use preparedStatement and do batch job:
eg. from http://docs.spring.io/spring/docs/current/spring-framework-reference/html/jdbc.html
public int[] batchUpdate(final List<Actor> actors) {
int[] updateCounts = jdbcTemplate.batchUpdate("update t_actor set first_name = ?, " +
"last_name = ? where id = ?",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, actors.get(i).getFirstName());
ps.setString(2, actors.get(i).getLastName());
ps.setLong(3, actors.get(i).getId().longValue());
}
public int getBatchSize() {
return actors.size();
}
});
return updateCounts;
}
I know you don't want to use the in clause, but I think its the best solution for your problem.
If you use a for in this way, I think it's not optimal.
for(String id : params){
getJdbcTemplate().query(sql, new Object[]{id}, new CustomResultSetExtractor());
}
I think it's a better solution to use the in clause. And then use a ResultSetExtractor to iterate over the result data. Your extractor can return a Map instead of a List, actually a Map of List.
Map<Integer, List<MyObject>>
Here there is a simple tutorial explaining its use
http://pure-essence.net/2011/03/16/how-to-execute-in-sql-in-spring-jdbctemplate/
I think this is the best solution:
public List<TestUser> findUserByIds(int[] ids) {
String[] s = new String[ids.length];
Arrays.fill(s, "?");
String sql = StringUtils.join(s, ',');
return jdbcTemplate.query(String.format("select * from users where id in (%s)", sql),
ArrayUtils.toObject(ids), new BeanPropertyRowMapper<>(TestUser.class));
}
this one maybe what you want. BeanPropertyRowMapper is just for example, it will be very slow when there's a lot of records. you should change it to another more efficient RowMapper.

Fetch history of records using LINQ

I am using entity framework with repository pattern and unit of work objects..
I have an entity Request with properties "RequestId", "OldRequestId", which can be accessed using requestRepository object.
eg: requestRepostiory.GetAll(), requestRepository.GetFiltered(r=> r.Requestid =10)
If I pass a RequestId, it should retrieve me the specific record.
If the OldRequestId is not null in the retrieved record, it should bring the old request data as well.
It should go on until the OldRequestId is null.
Simple way would be something like this:
public static IEnumerable<Data> GetRecursive(int id)
{
while (true)
{
var tmp = GetFiltered(x => x.Requestid == id);
yield return tmp;
if (tmp.OldRequestId.HasValue)
id = tmp.OldRequestId.Value;
else
yield break;
}
}
Please note, that this code would run make multiple queries towards the database. Performance won't be the best, but it might work for your scenario.

IQueryable (non generic) : missing Count and Skip ? it works with IQueryable<T>

i have an extension method which a person was really helpful to give me... it does an orderby on IQueryable ... but i wanted one to do a normal IQueryable (non generic)
Here is the code, The count and Skip and i think Take are missing .
public static IQueryable GetPage(this IQueryable query,
int page, int pageSize, out int count)
{
int skip = (int)((page - 1) * pageSize);
count = query.Count(); //COUNT DOESN'T EXIST
return query.Skip(skip).Take((int)pageSize); // NEITHER SKIP
}
Here is the and it works perfectly no errors.
public static IQueryable<T> GetPage<T>(this IQueryable<T> query,
int page, int pageSize, out int count)
{
int skip = (int)((page - 1) * pageSize);
count = query.Count();
return query.Skip(skip).Take((int)pageSize);
}
Any ideas how i can get around this? I don't want to change my return types as it works perfectly and i have another extension method called ToDataTable and it also functions on a non generic IQueryable ..
Is there a work around?
Thanks in advance
EDIT
I call it like so on an existing IQueryable
IQueryable<Client> gen = null;
IQueryable nongen = null;
var test = gen.GetPage(); //COMPILES!
var test 1 = non.GetPage(); // Doesn't compile because GETPAGE
// for non generic is broken as it has
// invalid methods like COUNT and SKIP etc.
I tried removing the GetPage non generic version but then my non Generic Iqueryable doesn't pickup the extension due to the fact its not a Iqueryable but only an IQueryable
Well, quite simply those methods aren't available for IQueryable. If you look at the Queryable methods you'll see they're almost all based on IQueryable<T>.
If your data source will really be an IQueryable<T> at execution time, and you just don't know what T is, then you could find that out with reflection... or in C# 4, just use dynamic typing:
public static IQueryable GetPage(this IQueryable query,
int page, int pageSize, out int count)
{
int skip = (int)((page - 1) * pageSize);
dynamic dynamicQuery = query;
count = Queryable.Count(dynamicQuery);
return Queryable.Take(Queryable.Skip(dynamicQuery, skip), pageSize);
}
The dynamic bit of the C# compiler will take care of working out T for you at execution time.
In general though, I'd encourage you to just try to use the generic form everywhere instead - it's likely to be significantly simpler.
The question is old but when someone needs the IQueryable extension methods then he/she should return IQueriable when the return type is anonymous.

Resources