I have a requirement where i am supposed to return millions of records from my database when an api endpoint is being called. As per requirement, pagination is being done on the frontend so returning paginated data is not an option. I have tried this;
$this->employee_records = DB::table('master_data')->select(DB::raw("employee_number, first_name, middle_name, last_name, ministry, grade, original_date_of_hire, provider_name, pfa_name, date_of_birth"))->orderBy('first_name')->cursor();
But the load time is still in minutes. How can i optimize this? Thanks
You can give try to ElasticSearch, it can get job done faster, if not fastest!
https://www.elastic.co/
Related
I am using AWS Amplify for a project. Within it, I'm using Datastore and GraphQL. The data is stored in DynamoDB. I store the statistics of each user in a table. Each time the user does something useful in the app, a new record is added in the table with the latest statistic. I want to show only the latest statistic in my interface. For this I'm using the following code.
var userStats = await DataStore.query(
UserStats,
(c) => c.userId('eq', userid),
{sort: (s) => s.altCreatedAt(SortDirection.DESC)},
{limit: 1}
);
This query is fetching results from the table for the correct userid, it is sorting it correctly, but the {limit: 1} part of it is not working. I get everything, not just the last record. Then, in subsequent code, I have to take the first item in the array. I feel that is inefficient. It would be better if the query working right and I get just the latest statistic. How can I solve this? Is there a better way to handle this problem?
Im trying to figure out the best way to paginate in laravel when i am working with big datasets, the method i am using now is overworking the memory. this is what i am doing: i query from multiple tables, map the results etc, then merge multiple collections, and only after that do i paginate it, i realized though that this is a really bad way of going about this, because i am querying way to much data for no reason, and as the data grows the slower it will become.
the question is what would be the correct way?
i thought maybe i would paginate and then work with the data, the issue is that for instance if i was trying to sort all the merged data by date i wouldn't be getting the proper results because one table may have less data then the other...
here is some of the code to help with clarifying the question
first i will query two tables orders table and transactions table
$transactions = Transaction::all();
$orders = Order::all();
then i will send this all to a action to map it all and format it the way i need to with laravel resources
return $history->execute(['transactions' => $transactions, 'orders' => $orders]);
what this action does is simply map all the transactions and return a resource, then it will map all the orders and return resources for them as well, after which it will merge it all and return the result.
then i take all of this and run it through a pagination action which uses the Illuminate\Pagination\LengthAwarePaginator class.
the results are exactly the way i want them its simply a memory overload.
i tried paginating first instead of doing
$transactions = Transaction::all();
$orders = Order::all();
i did
$transactions = Transaction::orderBy('created_at' 'desc' )->paginate($request->PerPage);
$orders = Order::orderBy('created_at' 'desc' )->paginate($request->PerPage);
and then run it through the action that returns resources.
there are a few issues doing this, firstly, i will no longer get back the pagination data so the app won't know the current page or how many records there are, secondly, if for example the orders table has 2 records, and the transactions table has 10 records, event though the dates of the orders table is after those of the transactions table they will still get queried first.
Use chunk method or paginate
Read documentation.
Parse doesn't return users emails in query response, I know what this is a preventive security measure in order to avoid data leaks, but I need get emails in aggregation query, how can I get it?
Can someone provide sample of code... I can't find needed info in docs.
Parse Server prevents sensitive information to be passed. If you are looking for a query that provides email information you'll need to use the masterkey.
new Parse.Query(Parse.User)
.find({useMasterKey: true});
I have something similar to twitter's feeds where we load it in real-time. How do i track how many people have seen a tweet? I am not talking about if you go to domain.com/post/32434 and that loads a status. I am talking about AJAX real-time query where one post is being loaded one after the other.
Will Google Analytics or Charbeat have anything that will help fulfill this need for me?
Why not managing your own counter in the database?
I don't really know your criteria, but let say you basically want to count how many times a given tweet was loaded.
Very quickly, I could think of this:
The table:
CREATE TABLE tweets_loads(ID_TWEET INT NOT NULL, LOADS BIGINT NOT NULL DEFAULT 0, PRIMARY KEY `ID_TWEET`) ENGINE=InnoDB;
The query at each ajax request:
INSERT INTO tweets_loads (ID_TWEET, LOADS) VALUES (myTweetId, 1)
ON DUPLICATE KEY UPDATE LOADS = LOADS + 1;
(Assuming mysql)
Run it from php and that's it... Up to you then to check competition between inserts though, but in theory mysql should handle it just well...
I am using an IList<Employee> where i get the records more then 5000 by using linq which could be better? empdetailsList has 5000
Example :
foreach(Employee emp in empdetailsList)
{
Employee employee=new Employee();
employee=Details.GetFeeDetails(emp.Emplid);
}
The above example takes a lot of time in order to iterate each empdetails where i need to get corresponding fees list.
suggest me anybody what to do?
Linq to SQL/Linq to Entities use a deferred execution pattern. As soon as you call For Each or anything else that indirectly calls GetEnumerator, that's when your query gets translated into SQL and performed against the database.
The trick is to make sure your query is completely and correctly defined before that happens. Use Where(...), and the other Linq filters to reduce as much as possible the amount of data the query will retrieve. These filters are built into a single query before the database is called.
Linq to SQL/Linq to Entities also both use Lazy Loading. This is where if you have related entities (like Sales Order --> has many Sales Order Lines --> has 1 Product), the query will not return them unless it knows it needs to. If you did something like this:
Dim orders = entities.SalesOrders
For Each o in orders
For Each ol in o.SalesOrderLines
Console.WriteLine(ol.Product.Name)
Next
Next
You will get awful performance, because at the time of calling GetEnumerator (the start of the For Each), the query engine doesn't know you need the related entities, so "saves time" by ignoring them. If you observe the database activity, you'll then see hundreds/thousands of database roundtrips as each related entity is then retrieved 1 at a time.
To avoid this problem, if you know you'll need related entities, use the Include() method in Entity Framework. If you've got it right, when you profile the database activity you should only see a single query being made, and every item being retrieved by that query should be used for something by your application.
If the call to Details.GetFeeDetails(emp.Emplid); involves another round-trip of some sort, then that's the issue. I would suggest altering your query in this case to return fee details with the original IList<Employee> query.