I have got a situation where I have list of records say 10,000, I am using datatable and I am using paging,(10 records per display). I wanted to put put that list in the session as:
facesContext........put("mylist", mylist);
And in the getters of the mylist, I have
public List<MyClass> getMyList() {
if(mylist== null){
mylist= (List<MyClass>) FacesContext......getSessionMap().get("mylist");
}
return mylist;
}
Now the problem is whene ever i click on paging button to go to second page, only the first records are displayed,
I know i am missing some thing, and I have few questions:
Is the way of putting the list in session correct.
Is this the way I should be calling the list in my case.
Thnaks in advance...
Something entirely different: I strongly recommend to not put those 10.000 records in the session scope. That is plain inefficient. If 100 users are visiting your datatable, those records would be duplicated in memory for every user. This makes no sense. Just leave them in the database and write SQL queries accordingly that it returns exactly the rows you want to display per request. That's the job the DB is designed for. If the datamodel is well designed (indexes on columns involved in WHERE and if necessary ORDER BY clauses), then it's certainly faster than hauling the entire table in Java's memory for each visitor.
You can find more insights and code examples in this article.
Related
I'm trying to dramatically cut down on pricey DB queries for an app I'm building, and thought I should perhaps just return IDs of a child collection (then find the related object from my React state), rather than returning the children themselves.
I suppose I'm asking, if I use 'pluck' to just return child IDs, is that more efficient than a general 'get', or would I be wasting my time with that?
Yes,pluck method is just fine if you are trying to retrieving a Single Column from tables.
If you use get() method it will retrieve all information about child model and that could lead to a little slower process for querying and get results.
So in my opinion, You are using great method for retrieving the result.
Laravel has also different methods for select queries. Here you can look Selects.
The good practice to perform DB select query in a application, is to select columns that are necessary. If id column is needed, then id column should be selected, instead of all columns. Otherwise, it will spend unnecessary memory to hold unused data. If your mind is clear, pluck and get are the same:
Model::pluck('id')
// which is the same as
Model::select('id')->get()->pluck('id');
// which is the same as
Model::get(['id'])->pluck('id');
I know i'm a little late to the party, but i was wondering this myself and i decided to research it. It proves that one method is faster than the other.
Using Model::select('id')->get() is faster than Model::get()->pluck('id').
This is because Illuminate\Support\Collection::pluck will iterate over each returned Model and extract only the selected column(s) using a PHP foreach loop, while the first method will make it cheaper in general as it is a database query instead.
So you know the background I'm coming from, I've been a professional programmer for over twelve years. My best language by far is C# but I've done C, C++, and most recently objectiveC. I've done a lot of work accessing data in databases but I haven't done as much UI work as most people (Except in IOS).
Recently I've begun using the Entity framework in C# for a job and I must say I wish I'd discovered it sooner. I wouldn't say it's the best thing since sliced bread but it's pretty damned close. After using it for a while it got me thinking about best practices and usage as compared to the old school method of using IDBConnections and IDBCommands for everything.
I was coding for a situation where I was going to be listing the contents of a table of users from a database in a bound data grid with the intention of giving the user the ability to do standard CRUD stuff. I started off by making an User class and a IUserManager interface with a corresponding implementation. Each user is assigned to a department and naturally there'd need to be a way to perform CRUD on departments too so I added a Department class, an IDepartmentManager interface and an implementation for that too. I set it up so that the grid bound on the results of the .GetAll() method on the IUserManager interface. Then I started filling in the guts.
I don't have the code in front of me any more but I basically used IDBConnection to tap into the datastore with an IDBCommand using a SQL query. Then I called command.ExecuteReader() and iterated the .Read() method on the IDataReader object. Using the ordinal for each column I pulled out the data, validated it and slipped it into a User class and added the class to a Dictionary that the method would then return. All the DB classes are of course IDisposable so wrapping them in a using takes care of cleaning up the mess.
Pretty standard stuff, I've done it a bazillion times.
That's when I realized that the departmentId I was pulling from the DB wasn't what I wanted to display in my grid. Telling someone 'this guy is in department 7' isn't as useful as saying 'this guy is in accounting'. So I first toyed with modding my query to get both the departmentId and name, and storing the name on the user object for display later. Then I decided to give the user a Department class instance that it would hang onto during it's lifetime that would be populated. That's when I converted the guts to linq.
public Dictionary<int, User> GetAll()
{
var result = new Dictionary<int, User>();
using (var datastore = new myEntities())
{
result = (from user in datastore.userInfoes
join department in datastore.userDepartmentInfoes on user.departmentID equals department.departmentID
select new User()
{
UserIndex = user.id,
FirstName = user.firstName,
LastName = user.lastName,
Department = new Department()
{
DepartmentId = user.departmentID.Value,
DepartmentName = department.departmentName,
},
Username = user.userName,
}
).ToDictionary(x => x.UserIndex, x => x);
}
return result;
}
That's where I started thinking (read: over-analysing probably)
The implementation I had would work just fine. It would even work pretty well for a small dataset. It'll even work fine for a largish dataset (say 10,000). Even if you counted every person in the company I currently work for five times over you'd have less than a thousand people.
But what if for a second I worked for a really big honking company that had 10 million employees? That would result in the departmentName strings being duplicated potentially millions of times.
That also got me thinking that unlike IOS's MVC implementation this particular situation wasn't going to query just enough users to fill the screen and then handle paging and stuff. As soon as the calling code refresh the data binding it was going to pull all 10 million users all at once and pass back the collection. That's going to be slow.
So that leaves me with the idea in my head that this method is both slow and inefficient with larger data sets. Not only that but the fact that there might be 2 million instances of 'Accounting' held with this data set it is going to be a major memory hog. We're also kind of defeating the purpose of a relational database here because of the Department class inside the User. In the DB you just have a departmentId int foreign key referencing an entry in another table. The link only occurs when you cross reference to the other table and even then there's really only one 'Accounting' string at any one time. In the above code you're going to have a whole lot of 'Accounting' strings floating around waiting to be cleaned up.
An MVC scenario would basically 'know' that it takes X number of entries to fill the grid's viewable area. It would only query X at a time starting from index Y and as the user navigated it would query and display additional records as needed. That's a heck of a lot better than querying all 10 million and letting them hang out somewhere whether they're displayed or not.
Like I said, I may very well be over-analysing this. I might also be incorrect in some of my assumptions with the way linq works. But in the interest of learning I figured I had to ask: What is the best way to do something like this? Is this sort of thing ok for small datasets? Would the whole thing be better off as an MCV implementation rather than pulling in the entire dataset to be displayed in the grid?
If you need the whole set of data in memory - you will have to load it anyway. I am sure you will not list 10kk users in a grid, right? The techniques that comes up is paging. Check this article from msdn with examples.
As for departments objects, does your UserInfo has a foreign key to the department? If so you should just have userInfo.Department available to you and no joins are needed.
If you bind the department data to the grid columns, why having the property of Department type? I assume your User class is something you bind to UI. Flatten it out into:
class User
{
Username
UserIndex
FirstName
LastName
DepartmentId
DepartmentName
}
What is the purpose of GetAll()? You return a dictionary and it feels like you need to enable lookups by id. Or do you use the result to enumerate the users?
For lookups, consider talking to the database to get you a single user data when needed. Implement caching if makes sense next.
For enumeration, do not return dictionary - that is all-in-memory object, return IEnumerable with yielded (paged?) results or even better IQueryable so that calling GetAll() doesn't execute the sql call right away, and the calling code can scope the call down by adding necessary filters
If making things work is only requirement, we can put all controlling login and DB handling logic even in the views & it will work. However this is not a right approach for reusable design.
Before I ask my real design question, below is my current understanding about separation of responsibilities in terms of model.
All Database related code, even db related logic, should go in models.
For a table, say 'my_tab', propel generate 4 classes, out of which only 2 classes 'MyTab.php' and 'MyTabPeer.php' should be edited.
MyTabPeer.php must only have data fetching.
Any logic, if required to fetch data, should go in 'MyTab.php'
This is simple and I hope it is correct, if not, please correct me.
Now, I have a special condition. I've 4 tables, say a, b, c, d. For that, propel generated 8 editable classes (excluding base*.php)
A.php APeer.php B.php BPeer.php
C.php CPeer.php D.php DPeer.php
One page of my application, shows Mailbox (say). Mailbox is not a table in database but it gets its data from complex join query between above 4 tables along with lot of calculation/conditions.
I generated that query, fetch data from it and displayed it. Mailbox is running as expected. However I did it in my controller (action class), which I know is not a right place for that.
My question is, where should I put that code? Possible options:
Controller, I think, is not a right place for DB logic/fetch.
I've 8 model classed however data do not belong to any one of them but as combination of them all.
A separate helper/lib, but I know I'll never reuse that code as its unique page of the site.
Anywhere else?
Please suggest if I'm wrong but I guess I should put it in models as it is fetching data. Since A is primary table, I probably should put code in A.php and APeer.php. If that is correct place, next question is, What should go in A.php & what should go in APeer.php? I've following operations to do:
Some logic to decide what columns, should I select.
As like mailbox, I can show received/sent message. Controller will tell what to show but there are some db logic to set conditions.
Then really fetch data from complex Join query.
Returned data will have all rows but I might need to merge few rows conditionally.
As per my understanding, Point 3 should go in APeer.php and rest in A.php. Is my understanding correct?
You should create separate model class i.e. Mailbox.
Method of this model should do the complex select and return data to your action in controller. This solution will not break MVC approach.
I am doing a ajax based search which suggests the values from the database tables as the user inputs fields in the search box. It does work but its really very slow, takes more than 10 seconds minimum for any suggestions to load. how do I optimize this to make it fast..
Like Index it/save it in cache/crawl?
Right now autoSearch() js function is fired which retrieves data from the related table to load the suggestions. how do I make this faster?
I dont know if this will make a difference but I am using grails/groovy for application development.
Thanks
Priyank
Have you added an index to any searched fields (or checked in the database to make sure the examined fields are indexed)?
The Grails GORM does not automatically create indices for text fields unless you add the appropriate static mapping to your domain class.
I am using an IList<Employee> where i get the records more then 5000 by using linq which could be better? empdetailsList has 5000
Example :
foreach(Employee emp in empdetailsList)
{
Employee employee=new Employee();
employee=Details.GetFeeDetails(emp.Emplid);
}
The above example takes a lot of time in order to iterate each empdetails where i need to get corresponding fees list.
suggest me anybody what to do?
Linq to SQL/Linq to Entities use a deferred execution pattern. As soon as you call For Each or anything else that indirectly calls GetEnumerator, that's when your query gets translated into SQL and performed against the database.
The trick is to make sure your query is completely and correctly defined before that happens. Use Where(...), and the other Linq filters to reduce as much as possible the amount of data the query will retrieve. These filters are built into a single query before the database is called.
Linq to SQL/Linq to Entities also both use Lazy Loading. This is where if you have related entities (like Sales Order --> has many Sales Order Lines --> has 1 Product), the query will not return them unless it knows it needs to. If you did something like this:
Dim orders = entities.SalesOrders
For Each o in orders
For Each ol in o.SalesOrderLines
Console.WriteLine(ol.Product.Name)
Next
Next
You will get awful performance, because at the time of calling GetEnumerator (the start of the For Each), the query engine doesn't know you need the related entities, so "saves time" by ignoring them. If you observe the database activity, you'll then see hundreds/thousands of database roundtrips as each related entity is then retrieved 1 at a time.
To avoid this problem, if you know you'll need related entities, use the Include() method in Entity Framework. If you've got it right, when you profile the database activity you should only see a single query being made, and every item being retrieved by that query should be used for something by your application.
If the call to Details.GetFeeDetails(emp.Emplid); involves another round-trip of some sort, then that's the issue. I would suggest altering your query in this case to return fee details with the original IList<Employee> query.