I am developing a page in a application using Spring, hibernate. In that page I have to display records by fetching from database in a datatable using jqgrid. The problem at here is the records that will be added to database in hundreds for every hour and total records in database is 300 000 and it's goes on increasing. Now what is the best solution for this problem(in terms of pagination and loading data)?
Here is solution for this.
store you data in session after retrieving from database and apply paging.
http://docs.spring.io/spring/docs/3.0.x/api/org/springframework/beans/support/PagedListHolder.html
I hop this will work.
yes i got you problem.
Twitter is also facing same problem to return tweets.
then solve this issue using max_id and since_id
here is documentation(flow) is there please refer it.
https://dev.twitter.com/docs/working-with-timelines
You can apply same algorithm in your paging it will solve issue.
is this solve you issue?
Related
as my question says, I would like to know what is my best choice for loading data in bigger systems in Laravel.
At the moment I use Laravel Eloquent to pull data from database and in my views I use dataTables JS library. It is effective for smaller systems or websites. However I seem to find myself in a position where systems that are bigger and more complex take very long time to load those data (sometimes more than 15 seconds).
I have found some solutions:
Eager loading relations helps with relations in row
Using DB instead of Eloquent
using Laravel pagination instead of dataTables pagination
loading data to dataTables from ajax source
However the pagination has some problems, especially with dataTables having the option of ordering/searching data.
My question is do you have any advice on how to load data in most effective way, so that it is as fast as it could be and the code as clean as possible. What do you do, when you need to load ginormous amount of data?
On of the best ways to optimize your queries is by using MySQL indexes. As per this documentation:
Without an index, MySQL must begin with the first row and then read
through the entire table to find the relevant rows. The larger the
table, the more this costs. If the table has an index for the columns
in question, MySQL can quickly determine the position to seek to in
the middle of the data file without having to look at all the data.
This is much faster than reading every row sequentially.
The most simple way to create an index is the following:
CREATE INDEX index_name
ON table_name (column1, column2, ...);
If you want the Laravel way of creating index, you can do it by using index() method, as per official documentation:
$table->index('column');
I'm using Laravel 5.2 and when working with pagination, I cannot limit the result. When I try with $query->take($x), the result is x records, but when I try with $query->take($x)->paginate(5), it give me all records with 5 records per page.
Could anyone work with laravel pagination could give a hint to resolve that problem?
Thank you so much!
It is not about just fetching the records, you will have to add a number of features in the grid e.g. may be export, searching, filtration etc. I have used Nayjest grid and would recommend you to use another one. I came across and issue and contacted the community and that's what is their reply:
At all, I would recommend to migrate to next major version of grids that is moved to separate repository: https://github.com/view-components/grids
Tip: If you have a lot of usage of nayjest/grids, you can use view-components/grids in parallel for your new reports.
so you should go for:
https://github.com/view-components/grids
I am using kendoUi Grid with 20k records, When i am changing page size 20 to 200 grid taking 40 to 50 sec to work. some times it taking min. paging is client side only.
For large datasets, it's better to use server paging, mainly because:
Faster loads
Avoid wasting memory
Avoid unnecessary database and network loads, database just gives you the number of records of the current page
You should consider enabling server paging at datasource level, and then read pagination values on backend side before performing the query to the database.
http://docs.telerik.com/kendo-ui/api/javascript/data/datasource#configuration-serverPaging
good question, i am also faced this type of issue.
Please use MVVM logic instance of MVC for bind the grid,
for more please find this below link.
http://demos.telerik.com/kendo-ui/grid/mvvm
I've got just over 10,000,000 records in the database of my component and I think getItems/getListQuery is trying to load every single one of them into memory. The search form on the site extremely slow or comes back saying php is out of memory.
phpMyAdmin seems to be able to handle displaying this data - why not Joomla?
The strange thing is that the items are then displayed correctly using the globally set list limit of 5 to a page.
I've just looked and Joomla's cache is disabled - is that screwing me up here?
Many thanks in advance!
I fixed it in the end by copying the getPagination, getTotal, getItems etc. from the library's (list.php) and into my model (to override them). Then in each method, I made sure the results where returned instead of sending them to the cache.
The getTotal function seems to count the number of rows instead of doing a seperate count(*). That's ok with a few thousand records but over 1/2 million is asking for trouble!
I need an advise, I am using Jqgrid as a grid tool, somehow I am finding some complication of showing very big data I am talking of 450 records with big data and 10 columns in there.
Is there any better grid you guys suggest to work with which gives me a better performance
I've always used dataTables and i always reccomend it because it's easy to use and configure. It is perfect and fast to display data ( i used it with tables with more than 100.000 rows with no problems, configuring server side procesiing correctly ).
The only thing you must now is that (as far as i know) it doesn't support colspan in the body of the table so, if your layouts require that, using datatables becomes impossible. (i usually found other way to show things rather than using colspan but for some this is a blocker)