I'm new to Django & web dev and looking for guidance on getting server-side processing working with my datatable.
Basically, I have an external .db SQLite file with 500k records and I would like to display the records on a bootstrap datatable.
In my views.py file, whenever they visit index.html, I make a query statement to fetch the records (not sure if that's inefficient but it's a small hobby website)
groups = cursor.execute("""SELECT * FROM PlayerGroups""")
return render(request, 'home/index.html', {'groups': groups}) # the issue since it returns 500k records which is too much for the client to handle.
I understand I need to put something like this in "index.html":
<script>
$(document).ready(function(){
$('#example').dataTable( {
"bProcessing": true,
"bServerSide": true,
"sAjaxSource": <confused about this part...>
});
});
</script>
I'm confused about making my own database API and linking the ajax source to my views/urls file. All of the guides I've looked online show server-side processing with models and stuff. I just have a external .db file I want to show on a datatable.
Any guidance or help would mean a lot.
No you don't want to query and display 500,000 records. That's too heavy for both frontend and backend. One solution is pagination.
Divide frontend into pages. Each page displays a fixed number, 10 for example, of records. When users request the first page, query the first 10 records in your db table. When users request the next page, query the next 10 records. And so on.
You can accomplish that using sqlite LIMIT and OFFSET clauses
SELECT
id, name
FROM
table
LIMIT 10 OFFSET offset;
where offset = page_number * 10.
Related
I have a slider component with a back and forward button and I want to get data in sort of pagination how can I do this?
I just want to do this without using Vue.js or livewire just laravel , JS and JQuery.
Laravel's Query Builder provides a paginate method that gives you a LengthAwarePaginator. Normally, you would render this in a blade view with $items->links(), but instead you can convert its result to JSON (https://laravel.com/docs/8.x/pagination#converting-results-to-json). That way you can use the results in JavaScript any way you'd like.
First, why using slider to create a pagination? there are better approach like lazy load, filter column, and simple pagination. lazy load is best for performance i guess.
If you insist to create a slider component, just wondering, how many data that you have? and how many data that you want to retrieve to screen? Imagine, you have more than 1.000.000 rows in your database, you sure wouldn't want to query for them all for real time updates to your screen. Why i tell you 'query them all'?
Let me tell you, if you do a pagination using laravel default pagination (using limit and offset), the flow behind this is like this:
DB will select all of your rows that you have
DB will limit your row depends on limit parameter
DB will scan row one-by-one to match the offset that you want
DB will retrieve the data that you want
With that flow, that's why pagination (limit & offset) is not good enough if you have a lot of data. The solution is change your pagination logic from limit & offset to query like this (case: auto incremental id):
SELECT
*
FROM
payments
WHERE
Id > 15
LIMIT 20
or for descending approach, you can query like this:
SELECT
*
FROM
payments
WHERE
Id < 50
ORDER BY Id DESC
LIMIT 20
Then in laravel, send a json that include a pagination data. So, in javascript you just have to align the pagination data with library or your own pagination logic.
Let's imagine we have a web-page with the content and one of the elements is table where we have couple of columns. It is done with Joomla, so basically I am working with the web-page constructor if I can call it like this and not with code. In the last column I have a link with query parameters, so something like this: link?qparam1=sth1&qparam2=sth. The values for these query parameters should be taken from the first and second column and inserted in this link. Otherwise I need manually to copy those values to each and every link which makes in very slow and inefficient especially when table values are changed, the link must be updated as well.
Is it possible to fetch the data from columns and include into the link?
Write a system plugin, Get data from a table, insert/update data to request
$app = JFactory::getApplication();
$app->input->set('qparam1','<value from table>');
it will update request data.
I have table in mysql with 20000 records.
How can I retrieve data faster?
I used datatable plugin for pagination and I also use codeigniter.
It took about one minute to split records to about 4000 page
please use pagination [ci Pagination, php pagination, jquery pagination ajax(faster than others) ] if you are using bootstrap datatable its not working because datatable fetch all the data in one hit after that it breaks data on view which is time consuming you need to break the data in small groups like 20-20 or 50-50 and you need to fetch data in every hit of pagination numbers.
if you are usign ci pagination it will reload on every page hit however if you are using jquery pagination it help you to fetch data in every number hit without refreshing the pages.
please go through the following link: https://phppot.com/php/ajax-pagination-with-php/
in order to implement ajax pagination.
Create view and load table headers
add id on tbody
start ajax pass number want to break = 50, and page = 1
send those value to controller and fetch data as per page.
in your modal you need to add limit and offset which will revised on every hit of pagination number.
I'm trying to implement a dashboard similar to facebook in cakephp (getting posts and post them to timeline and while you press see more it keeps retrieving posts from previous offsets) , but im still confused about the logic and tools , should i use the cakephp pagination class in my implementations.
$this->paginate();
it somehow should be called through ajax accourding to some performance wise
Any helps or suggestions where to start from ?
Thanks All
Don't use paginate
If you paginate something that you are prepending data to - you're going to get data overlapping such that you ask for page 2 - and get the end of, as far as the current user is concerned, the previous page.
Use a timestamp
The normal technique for an endless stream of data is to use a query like:
SELECT *
FROM foos
WHERE created >= $previousLastTimestamp
ORDER BY created DESC
LIMIT 20
Note that while I'm using created in this example - it can be any field that is pseudo unique.
When you first render the page, store the timestamp of the last entry in a javascript variable, then your "get more posts" logic should be:
Make an ajax (get) request, passing the last timestamp
Perform the above sql query (as a $this->Foo->find call)
in your js update the last timestamp so that you know where you are up to for the next time the user clicks "get more posts"
The reason to use a >= condition is that, unless the field you are testing against has unique values, it's possible for there to be multiple rows with the value you're testing for. If you have a naturally-unique field that you are sorting by (id) then you don't need to use greater-or-equal, you can simply use greater-than, and avoid needing to think about duplicate rows.
Here's a reference which explains in more detail why you should handle systems like this avoiding traditional pagination.
I'm working with Google charts, in this case the Google ColumnChart.
My project is to get the data from the database and to put it into a ColumnChart.
After the data has been loaded into the DataTable, the data can be filtered.
For example:
The user wants the data to be from 2012 only instead of from 2010 until 2013.
Now I've got two choices:
Do one database query to get all of the data and filter it by using a DataView. With this DataView, it's possible to show a piece of the DataTable. So, everytime the user changes the filter options, a new DataView will be made based on the filter options.
Execute a new database query every time the user changes the filter options. A new DataTable can be created with the results.
The query in choice one will return about 2000 rows, each with about 20 values, and there are about 50 users requesting the page at the same time. There are about 50 filter options available.
Which choice has the best performance?
If choice one is the best, what about memory usage?
In short, what's the best option?
If there are any other options available, please let me know.
Thanks in advance