I am trying to page a large set of data using IQueryable. Take method for a query that I am building dynamically. I was going to use the take method so I can keep taking until I have read the entire large dataset. Basically, I am reading the data in chucks to increase performance.
I am trying to allow a user via a few drop downs. Build a connectionstring to the database (database and table can change) and I want to be able to then take the table that is selected by the user and perform a select clause. Not knowing which table or elements are available during run time the select and from section will have to be dynamic.
I am not sure if this is possible with the Linq but does anyone have a idea on how I can solve this?
Related
My case is that a third party prepares a table in our schema domain on which we run different spring batch jobs that look for mutations (diff between the given third party table and our own tables). This table will contain about 200k records on average.
My question is simply: does generating a material view up front provide any benefits vs running the query at runtime?
Since the third party table will be populated on our command (basically it's a db boolean field that is set to 1, after which a scheduler picks it up to populate the table. Don't ask me why it's done this way), the query needs to run anyway.
Obviously from an application point of view, it seems more performant to query a flat material view. However, I'm not sure if there is any real performance benefit, since the material view needs to be built on db level.
Thanks.
The benefit of a materialized view here is if you are running the multiple times (more so if the query is expensive and / or there is a big drop in cardinality).
If you are only hitting the query once then you there isn't going to be a huge amount in it. You are running the same query either way and you have the overhead of inserting into the materialized view but you also have the benefit that you can tune this a lot easier than you could querying via JPA and could apply things like compression so less data is transferred back to the application but for 200k rows any difference is likely to be small.
All in all, unless you are running the same query multiple times then I wouldn't bother.
Update
One other thing to consider is coupling. Referencing a materialized view directly in JPA would allow you to update any logic without updating the application but the flip side of this is that logic is hidden outside the application which can make debugging a pain.
Also if you are just referencing a materialized view directly and not using any query rewrite or rollup features then am simple table created via CTAS would actually be better as you still have the precomputed data without the (small) overhead of maintaining the materialized view.
I am puzzled by Hibernate’s behavior when loading ResultSets with many columns via HQL: it seems like OracleStatement.getColumnIndex(String) is called over and over again, not just once for every column at the beginning of the load but once for every column when retrieving each and every row of data.
In case of a ResultSet with many columns this seems to take a significant amount of time (in our case about 43 % of the total time, see attached screenshot of a VisualVM profile). Our HQL loads a table with many columns, join fetching two other tables with to-many relations (both having lots of columns, too). Unfortunately we cannot restrict the columns to be loaded, because the task is to preload all objects into a Coherence cache on startup of the system, so the objects have to be complete.
As far as I can tell the problem arises because hydrating the mapped result objects of an HQL query from the ResultSet does use nullSafeGet() for each column which takes String arguments to identify the column and therefore has to call getColumnIndex().
(When loading the data from a ResultSet of an SQL query one can use getString(int), getTimestamp(int) etc. instead of String based versions to avoid this issue.)
We are still using an old version of Hibernate (3.6) but the source on github indicates that the same behavior is still present, as nullSafeGet() is still String based instead of taking an index (or object containing the index) which then could be precomputed once at the beginning of the load.
Is there something that I am missing?
Is there a reason for calling getColumnIndex() for each column of each row of data over and over again?
Is there a way around this which does not involve rewriting the query into SQL and using the index based accessors to build up the mapped objects manually?
The only similar issue I was able to find on the internet was this question which has no answer.
The query there had many columns, too.
Thanks for any help!
Thorsten
This problem is addressed in Hibernate 6, which switches from reading JDBC ResultSet by name to reading by position.
I'm learning about Oracle Views and I got the concept of views but little confused about performance.
I was watching video and there I listen that oracle view can increase the performance. Suppose I have created view like below.
CREATE VIEW SALES_MAN
AS
SELECT * FROM EMP WHERE JOB='SALESMAN';
Ok now I have executed query to get SALES_MAN detail.
SELECT * FROM SALES_MAN
Ok now confusion start.
I listened in video that once the above query SELECT * FROM SALES_MAN will be executed the DATA/RECORD will be placed into cache memory after hitting the oracle DB. and if I will execute same query( IN Current Session/Login ) Oracle Engine will not hit to Database and will give you record from CACHED-MEMORY is it right?
But I have read on many websites that View add nothing to SQL performance more
Another Reference that says view not help in performance. Here
So Views increase performance too or not?
A view is just a kind of virtual table defined by a query. It has no proper data and performance will depend on the underlying table(s) and the query definition.
But you also have Materialized View wich stores the results from the view query definition. Data is synchronized automatically, and you can add indexes to the view. In this case, you can achieve better performances. The cost to pay is
more space (the view contains data dupplicated),
no access to the up to date data from the underlying tables
You (and perhaps the creator of the un-cited video) are confusing two different concepts. The reason the data that was in the cache was used on the second execution was NOT because it was the second execution of the view. As long as data remains in the cache, it is available for ANY query that needs it. The very fact that the first execution of the view had to get the data from disk should be a strong clue. And what if your second use of the view wasn't until hours or days later when the data was no longer in the cache? The answer remains. A view does NOT improve performance.
I have a table with data relating to several moments in time that I have to keep updated. To save space and time, however, each row in my table refers to a given day and hourly and quarter-hourly data for that day are scattered throughout the several columns in that same row. When updating the data for a particular moment in time I, therefore, must choose the column that has to be be updated through some programming logic in my PL/SQL procedures and functions.
Is there a way to dynamically choose the column or columns involved in an update/merge operation without having to assemble the query string anew every time? Performance is a concern and the throughput must be high, so I can't do anything that would perform poorly.
Edit: I am aware of normalization issues. However I still would like to know a good way for choosing the columns to be updated/merged dynamically and programatically.
The only way to dynamically choose what column or columns to use for a DML statement is to use dynamic SQL. And the only way to use dynamic SQL is to generate a SQL statement that can then be prepared and executed. Of course, you can assemble the string in a more or less efficient manner, you can potentially parse the statement once and execute it multiple times, etc. in order to minimize the expense of using dynamic SQL. But using dynamic SQL that performs close to what you'd get with static SQL requires quite a bit more work.
I'd echo Ben's point-- it doesn't appear that you are saving time by structuring your table this way. You'll likely get much better performance by normalizing the table properly. I'm not sure what space you believe you are saving but I would tend to doubt that denormalizing your table structure is going to save you much if anything in terms of space.
One way to do what is required is to create a package with all possible updates (which aren't that many, as I'll only update one field at a given time) and then choosing which query to use depending on my internal logic. This would, however, lead to a big if/else or switch/case-like statement. Is there a way to achieve similar results with better performance?
Which one enhace the performance in an Access:
Having stored procedures
writing our query in VB code
or writing our query in Datasource of our control property
logicaly the first one should enhance the performance but here I have read another thing.What do you mean?
could you please explain me technicaly or show me a reference?
If you're talking about an MDB file, then you must be using "stored procedure" to refer to saved queries. Saved queries can execute more quickly than in-line SQL, because the query plan is stored with the query; when you execute inline SQL, the SQL has to be compiled into a query plan.
If you write a query into the DataSource property of a control, Access creates a hidden saved query, so the performance should be similar.
More generally, are you asking this question because you or your users have observed poor performance? If so, where is the performance poor? That information might help us suggest a good opportunity for improving the performance.
For example, suppose your form is an order form, and you have text boxes showing the customer's name and address. The record source of the form is the [Orders] table. The name text box has as its control source
SELECT [CustomerName] FROM [Customers] WHERE [CustomerID] = [CustomerIdTextBox]
Similarly, the control source of the address text box:
SELECT [Address] FROM [Customers] WHERE [CustomerID] = [CustomerIdTextBox]
In this case, you'll improve your performance markedly by changing the record source:
SELECT [Orders].*, [Customers].[CustomerName], [Customers].[Address]
FROM [Orders] INNER JOIN [Customers] ON [Orders].[CustomerID] = [Customers].[ID]
Then change the control source for the name text box to [CustomerName] and for the address text box to [Address].
Why is this faster? In the first design, you had two queries (or, rather, a query for each item you're pulling from the [Customers] table). In the new design, you have only one query.
If, on the other hand, you are asking about performance because of a theoretical idea that one approach might be faster than another, then you should use the approach that is simplest to maintain. If you can't observe the difference in performance, there's no benefit to using a more performant solution.
A stored procedure still needs to be executed. In Access, it has to be executed on your own computer, with a regular DBMS it still needs to be executed on the server. Which approach is more efficient depends on the problem you are solving.