I have a main table with 4 lookup tables. All tables are bound to SQL tables accessed via stored procedures. Each lookup table has between 3 and 27 rows. The main table has about 22,000 rows (pet food & products).
My DataGridView displays rows of the main table with the lookup columns configured as ComboBoxes. I can insert, delete, edit the main table, ... all of these functions work fine. The initial loading of the form and inserting a new row each take about a second but this is not concerning. All other operations are relatively fast.
Here is a screen shot of the DataGridView:
The problem comes when I close the form...and only after having inserted one or more rows into the main table. The closing of the form can take up to a minute. In the closing of the form I now dispose of the binding sources for the 5 tables myself so that I can time them. Disposing of the binding source for the 4 lookup tables routinely takes 10-15 seconds per table. Closing the binding source for the main table takes no time at all. Again, this only happens after inserting a new row into the main table. I can edit main table rows, change lookup column values, and delete rows and closing the form in those use cases is instant.
I have tried running the program within VS, outside of VS running a debug EXE, and running outside of VS running a release version of the EXE, all with similar results.
What can I do to prevent this long delay disposing of the ComboBox binding sources? Is this typical and are there alternatives I should be considering?
After three days of pounding my head against the wall trying all kinds of unsuccessful resolutions, I rebuilt the application from scratch. That fixed the problem and I believe I discovered the cause. At first I just created the data set with all the table adapters, which was pretty fast, and then I added a single form and grid to mimic the condition I described above above. Testing confirmed no issues at all so I continued added more forms with the same ComboBox look-ups and it continues to work fine. I am pretty sure that there was something screwy in the data set definitions I had previously. Hopefully this helps someone in the future.
Related
My case is that a third party prepares a table in our schema domain on which we run different spring batch jobs that look for mutations (diff between the given third party table and our own tables). This table will contain about 200k records on average.
My question is simply: does generating a material view up front provide any benefits vs running the query at runtime?
Since the third party table will be populated on our command (basically it's a db boolean field that is set to 1, after which a scheduler picks it up to populate the table. Don't ask me why it's done this way), the query needs to run anyway.
Obviously from an application point of view, it seems more performant to query a flat material view. However, I'm not sure if there is any real performance benefit, since the material view needs to be built on db level.
Thanks.
The benefit of a materialized view here is if you are running the multiple times (more so if the query is expensive and / or there is a big drop in cardinality).
If you are only hitting the query once then you there isn't going to be a huge amount in it. You are running the same query either way and you have the overhead of inserting into the materialized view but you also have the benefit that you can tune this a lot easier than you could querying via JPA and could apply things like compression so less data is transferred back to the application but for 200k rows any difference is likely to be small.
All in all, unless you are running the same query multiple times then I wouldn't bother.
Update
One other thing to consider is coupling. Referencing a materialized view directly in JPA would allow you to update any logic without updating the application but the flip side of this is that logic is hidden outside the application which can make debugging a pain.
Also if you are just referencing a materialized view directly and not using any query rewrite or rollup features then am simple table created via CTAS would actually be better as you still have the precomputed data without the (small) overhead of maintaining the materialized view.
I am puzzled by Hibernate’s behavior when loading ResultSets with many columns via HQL: it seems like OracleStatement.getColumnIndex(String) is called over and over again, not just once for every column at the beginning of the load but once for every column when retrieving each and every row of data.
In case of a ResultSet with many columns this seems to take a significant amount of time (in our case about 43 % of the total time, see attached screenshot of a VisualVM profile). Our HQL loads a table with many columns, join fetching two other tables with to-many relations (both having lots of columns, too). Unfortunately we cannot restrict the columns to be loaded, because the task is to preload all objects into a Coherence cache on startup of the system, so the objects have to be complete.
As far as I can tell the problem arises because hydrating the mapped result objects of an HQL query from the ResultSet does use nullSafeGet() for each column which takes String arguments to identify the column and therefore has to call getColumnIndex().
(When loading the data from a ResultSet of an SQL query one can use getString(int), getTimestamp(int) etc. instead of String based versions to avoid this issue.)
We are still using an old version of Hibernate (3.6) but the source on github indicates that the same behavior is still present, as nullSafeGet() is still String based instead of taking an index (or object containing the index) which then could be precomputed once at the beginning of the load.
Is there something that I am missing?
Is there a reason for calling getColumnIndex() for each column of each row of data over and over again?
Is there a way around this which does not involve rewriting the query into SQL and using the index based accessors to build up the mapped objects manually?
The only similar issue I was able to find on the internet was this question which has no answer.
The query there had many columns, too.
Thanks for any help!
Thorsten
This problem is addressed in Hibernate 6, which switches from reading JDBC ResultSet by name to reading by position.
We have a TDBGrid that connected to TClientDataSet via TDataSetProvider in Delphi 7 with Oracle database.
It goes fine to show content of small tables, but the program hangs when you try to open a table with many rows (for ex 2 million rows) because TClientDataSet tries to load the whole table in memory.
I tried to set "FetchOnDemand" to True for our TClientDataSet and "poFetchDetailsOnDemand" to True in Options for TDataSetProvider, but it does not help to solve the problem. Any ides?
Update:
My solution is:
TClientDataSet.FetchOnDemand = T
TDataSetProvider.Options.poFetchDetailsOnDemand = T
TClientDataSet.PacketRecords = 500
I succeeded to solve the problem by setting the "PacketRecords" property for TCustomClientDataSet. This property indicates the number or type of records in a single data packet. PacketRecords is automatically set to -1, meaning that a single packet should contain all records in the dataset, but I changed it to 500 rows.
When working with RDBMS, and especially with large datasets, trying to access a whole table is exactly what you shouldn't do. That's a typical newbie mistake, or a borrowing from old file based small database engines.
When working with RDBMS, you should load the rows you're interested in only, display/modify/update/insert, and send back changes to the database. That means a SELECT with a proper WHERE clause and also an ORDER BY - remember row ordering is never assured when you issue a SELECT without an OREDER BY, a database engine is free to retrieve rows in the order it sees fit for a given query.
If you have to perform bulk changes, you need to do them in SQL and have them processed on the server, not load a whole table client side, modify it, and send changes row by row to the database.
Loading large datasets client side may fali for several reasons, lack of memory (especially 32 bit applications), memory fragmentation, etc. etc., you will flood the network probably with data you don't need, force the database to perform a full scan, maybe flloding the database cache as well, and so on.
Thereby client datasets are not designed to handle millions of billions of rows. They are designed to cache the rows you need client side, and then apply changes to the remote data. You need to change your application logic.
i have a crystal report that calls a function in Oracle. this function uses a select that selects a view as one of its' tables. this view uses multiple links to other databases(the view is a union of several queries). whenever this function runs, it appears as though the query for the view is run, and every link that creates the view seems to go into enq- DX contention.
is this related to this known issue?
http://surachartopun.com/2008/12/dbink-hangs-enq-dx-contention.html
the reason i ask is that my research seems to indicate that this problem should only happen when linking to different versiions of Oracle but all of the ones i am using are 10.2
Has this method ever worked? It does not sound like a good way to do things. I can just imagine the function causing an enormous amount of round trips as it processes each row.
Some other ways of doing this include putting the function as close to the biggest data table as possible and keeping the processing on that database. Only return what you need from the function.
You could create a table representing your desired data set from a query every morning and use Crystal to query that directly. Just refresh your data as required using stored procedures or advanced queues.
A materialized view on one of the database sources with a refresh would be another approach.
EDIT: yes, using database links can be resource intensive and cause the kind of problems you have seen. Has this ever worked or is there a new requirement you need a solution to.
If it used to work what changed?
If this is new I humbly suggest a different approach as I have suggested.
Let's say I have ten rows on a data grid and I have changed data of three rows.
I am trying to save data by Linq for those three rows, but I am not sure how it is possible to save them.
I can do it by looping whole rows by checking each row for any change.
Is there any smarter way to save multiple data rather than looping by code, such as For Next.
As long as the Data Context is kept alive while the changes are made in the grid, you can save all of the changes by calling SubmitChanges() using the Unit of Work pattern. This works fine in stateful (Winform/WPF) implementations. In web applications, the typical UI models only allow for editing a single row per page submission. In that case, the challenge becomes how to allow editing multiple records in a single page request. If you can do that, then you can batch up your updates and only call SubmitChanges once per page request.