I have a query that selects a single column and I am executing the query in batches using setFirstResults & setMaxResults methods of SQLQuery.
Snippet:
SQLQuery query = <query object with query projecting a single column>;
int maxResults = 50;
int batchSize = 50;
for (int i = 0; ; i++) {
query.setFirstResult(batchSize*i);
query.setMaxResults(maxResults);
List resultSet = query.list();
if(resultSet.isEmpty())
break;
//process result set
}
I set to true the showSQL parameter in hibernate config to see the query string that hibernate produces. For the first batch, i.e. when i=0 below is the query that hibernate generates:
select * from (/* query selecting single column here */) where rownum <= ?;
which makes sense since its the first batch and we want results from first row and rownum is used to restrict the number of results to maxResults.
Now for the second and subsequent batch reads, the query hibernate generates is:
select * from ( select row_.*, rownum rownum_ from (/*query selecting single column here */) row_ where rownum <= ?) where rownum_ > ?;
and you can clearly see, the above query is selecting two columns, one being the row number itself.
So when my query only selects one column, hibernate's version of the query is selecting two.
Is this known issue? Can I do something different or am I doing something wrong?
I don't want to cast the result set into two different types before using/processing it.
Related
we're facing performance issues with EclipseLink 2.7.7 when accessing Oracle 12.1 tables with paging. Investigation showed that Oracle does not use its indexes with EclipseLink paging.
I've extracted the sql sent to the database and was able to reproduce the issue using a database tool (DataGrip).
Example:
-- #1: without paging
SELECT col1 AS a1, col2 AS a2, col3 AS a3, ...
FROM <TABLE>
WHERE colN > to_timestamp('2021-12-08', 'yyyy-mm-dd'))
ORDER BY col1 DESC;
Explain plan shows that the index on colN is used. Fine.
When the same query is executed with paging, the original query is wrapped in two subselects:
-- #2 with EclipseLink paging
SELECT * FROM (
SELECT a.*, ROWNUM rnum FROM (
SELECT col1 AS a1, col2 AS a2, col3 AS a3, ...
FROM <TABLE>
WHERE colN > to_timestamp('2021-12-08', 'yyyy-mm-dd'))
ORDER BY col1 DESC
) a WHERE ROWNUM <= 100
) WHERE rnum > 0;
For this query, the explain plan shows that the index on colN is not used.
As a result, querying a table with millions of rows takes 50-90 seconds (depending on the hardware).
Side note: on my test database, this query returns 0 records since colN values are before 2021-12-08.
Oracle 12c introduced the OFFSET/FETCH syntax:
-- #3
SELECT col1 AS a1, col2 AS a2, col3 AS a3, ...
FROM <TABLE>
WHERE colN > to_timestamp('2021-12-08', 'yyyy-mm-dd'))
ORDER BY col1 DESC
OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY;
Using this syntax, indexes are at least sometimes used as expected. When they are used, execution time is below 1s which is acceptable.
However, I could not figure out how to convince EclipseLink to use this syntax.
If ORDER BY col1 DESC is removed from the original paged query (#2), the index is used the query returns fast enough. However, it will not return the desired records, so that does not help.
How can I implement performant paged queries using EclipseLink and Oracle 12?
How can I force oracle to use the index on colN when using paging and order by?
The OraclePlatform printSQLSelectStatement method is responsible for building the query used, nesting the queries to use rownum for the query you've seen. To use a new form, you would extend one of the OraclePlatform classes you are using (maybe Oracle12Platform) and override that method to append the syntax you want instead. Something like:
#Override
public void printSQLSelectStatement(DatabaseCall call, ExpressionSQLPrinter printer, SQLSelectStatement statement) {
int max = 0;
int firstRow = 0;
ReadQuery query = statement.getQuery();
if (query != null) {
max = query.getMaxRows();
firstRow = query.getFirstResult();
}
if (!(this.shouldUseRownumFiltering()) || (!(max > 0) && !(firstRow > 0))) {
super.printSQLSelectStatement(call, printer, statement);
return;
}
call.setFields(statement.printSQL(printer));
printer.printString("OFFSET ");
printer.printParameter(DatabaseCall.MAXROW_FIELD);
printer.printString(" ROWS FETCH NEXT ");
printer.printParameter(DatabaseCall.FIRSTRESULT_FIELD);
printer.printString(" ROWS ONLY");
call.setIgnoreFirstRowSetting(true);
call.setIgnoreMaxResultsSetting(true);
}
You would then specify your custom OraclePlatform class using a persistent property:
<property name="eclipselink.target-database" value="my.package.MyOracle12Platform"/>
If something like that works for you, please submit it as an enhancement request - though you might want to work some way to use the old behaviour into it, as the performance differences you've experienced might depend on the query/data involved.
Thanks to #Chris, I came up with the following Oracle12Platform. This solution currently ignores "Bug #453208 - Pessimistic locking with query row limits does not work on Oracle DB". See OraclePlatform.printSQLSelectStatement for details):
public class Oracle12Platform extends org.eclipse.persistence.platform.database.Oracle12Platform {
/**
* the oracle 12c `OFFSET x ROWS FETCH NEXT y ROWS ONLY` requires `maxRows` to return the row count
*/
#Override
public int computeMaxRowsForSQL(final int firstResultIndex, final int maxResults) {
return maxResults - max(firstResultIndex, 0);
}
#Override
public void printSQLSelectStatement(final DatabaseCall call, final ExpressionSQLPrinter printer, final SQLSelectStatement statement) {
int max = 0;
int firstRow = 0;
final ReadQuery query = statement.getQuery();
if (query != null) {
max = query.getMaxRows();
firstRow = query.getFirstResult();
}
if (!(this.shouldUseRownumFiltering()) || (!(max > 0) && !(firstRow > 0))) {
super.printSQLSelectStatement(call, printer, statement);
} else {
statement.setUseUniqueFieldAliases(true);
call.setFields(statement.printSQL(printer));
if (firstRow > 0) {
printer.printString(" OFFSET ");
printer.printParameter(DatabaseCall.FIRSTRESULT_FIELD);
printer.printString(" ROWS");
call.setIgnoreFirstRowSetting(true);
}
if (max > 0) {
printer.printString(" FETCH NEXT ");
printer.printParameter(DatabaseCall.MAXROW_FIELD); //see #computeMaxRowsForSQL
printer.printString(" ROWS ONLY");
call.setIgnoreMaxResultsSetting(true);
}
}
}
}
I had to override computeMaxRowsForSQL in order to get the row count instead of "lastRowNum" when calling printer.printParameter(DatabaseCall.MAXROW_FIELD);
I also try to deal with missing firstRow xor maxResults
I have a complex SQL query and I want to implement it through Active Records. This query has several AND / OR clauses grouped together with another criteria. I have gone through various articles where they said that we can use group_start() and group_end() but I am wondering if a group can be started inside another group too? The resulting serial numbers needs to be excluded from the result set to be produced by the outer query. Actually I tried using Join here, but it didn't work. Any working idea regarding joins here will be appreciable too.
As you can see in the query below, I have used double round brackets to represent multiple groups inside a group.
The resulting serial numbers needs to be excluded from the results of outer query too. Please tell me what will be its Codeigniter Active Record equivalent code.
select * from table2 WHERE NOT table2.serial IN (select columnname from table where ((col < val and val < col) or (col < val and val < col) or(val=col and val=col)) AND incol=intval AND intcol=intval)
Here, col is the column name, val is a value of DATE type, intval is
an Integer value
Try this syntax
$this->db->select("*")->from("table");
$this->db->group_start();
$this->db->group_start();
$this->db->where("val <",'col');
$this->db->where("val <",'col');
$this->db->group_end();
$this->db->or_group_start();
$this->db->or_where("val <",'col');
$this->db->where("val <",'col');
$this->db->group_end();
$this->db->or_group_start();
$this->db->or_where("val ",'col');
$this->db->where("val ",'col');
$this->db->group_end();
$this->db->group_end();
$this->db->where("incol ",'intval');
$this->db->where("incol ",'intval');
$this->db->get();
$last_query = $this->db->last_query();
$this->db->select('*')->from('table2');
$this->db->where_not_in('serial',$last_query);
$this->db->get();
echo $this->db->last_query();
The query string produced by the above is
SELECT * FROM `table2` WHERE
`serial` NOT IN(
SELECT columnname FROM `table` WHERE
(
(`val` < 'col' AND `val` < 'col') OR
(`val` < 'col' AND `val` < 'col') OR
(`val` = 'col' AND `val` = 'col')
) AND `incol` = 'intval' AND `incol` = 'intval'
);
i need help with the following dax statement.
Situation:
I have 2 tables. One table contains sell data with articleIDs, dateIDs and sell prices, another table contains stock movements data with articleIDs, dateIDs and purchase prices. According to the dateID i want to write the purchase prices into the first table using a calculated column because i need the prices for every row.
Example:
Table1 t1
t1.articleID = 123; t1.dateID = 20160905; t1.sellPrice = 62,55; t1.purchasePrice = My DAX Statement
Table2 t2
t2.articleID = 123; t2.dateID = 20160905; t2.purchasePrice = 37,07
t2.articleID = 123; t2.dateID = 20160905; t2.purchasePrice = 37,07
t2.articleID = 123; t2.dateID = 20160906; t2.purchasePrice = 37,07
t2.articleID = 456; t2.dateID = 20160905; t2.purchasePrice = 12,15
My DAX Statement:
= CALCULATE (
VALUES (t2[purchasePrice]);
TOPN (
1;
FILTER(FILTER(t2; t2[articleID] = t1[articleID]); t2[dateID] <= t1[dateID]); t2[dateID]; DESC
)
)
With my DAX Statement i get the following error:
A table of multiple values was supplied where a single value was expected.
It is normal that i have more than one row matching in the table 2.
Actually I just want the price of any of them on the corresponding dateID, even if they are tied. So i used the TOPN function with the value 1 and sorted by date but the error still remains. Is there a way to fix my DAX Statement to achieve this?
Create a calculated column in T1 and use this expression:
purchasePrice =
CALCULATE (
MAX ( T2[purchasePrice] ),
FILTER ( T2, T1[ArticleID] = T2[articleID] && T1[DateID] = T2[dateID] )
)
Note I use comma to separate passed arguments to the functions but I see in your expression you used semicolon. Change it to match your system list separator.
It is not tested but should work. Let me know if it works for you.
For multiple data insertion we have an efficient way: RecordSortedList
RecordSortedList rsl;
MyTable myTable;
;
rsl = new RecordSortedList(myTable.tableid);
rsl.sortOrder(fieldname2id(myTable.tableId,'RecId'));
myTable.field1 = 'Value1';
rsl.ins(myTable);
myTable.field1 = 'Value2';
rsl.ins(myTable);
rsl.insertDatabase();
Is the same possible for multiple records retrieval from db in one go? Something like
int i =1;
while(i<10000)
{
//enter records from db into a buffer in db
i++
}
//now bring the buffer from db in a single trip
//and do the data manipulation in AX
My intention is to optimize the db trip to the least.
Please Suggest.
Yes, it's called RecordLinkList - http://msdn.microsoft.com/en-us/library/aa643250(v=ax.50).aspx
A recordLinkList is a double linked list that can hold records of
different types at the same time. It is not keyed or sorted.
The recordLinkList is particularly useful for passing records from
different tables as a parameter instead of retrieving the same records
again.
There is no limit to the size of a recordSortedList; it is the
responsibility of the programmer to control its size and, therefore,
memory consumption.
You can also add different types of records.
static void RecordLinkList(Args _args)
{
RecordLinkList rll = new RecordLinkList();
SalesTable salesTable;
CustTable custTable;
InventTrans inventTrans;
Address address;
boolean iterate;
;
select firstonly salesTable;
select firstonly custTable;
select firstonly inventTrans;
select firstonly address;
rll.ins(salesTable);
rll.ins(custTable);
rll.ins(inventTrans);
rll.ins(address);
iterate = rll.first();
while (iterate)
{
switch (rll.fileId()) // FileId == TableId
{
case tablenum(SalesTable):
salesTable = rll.peek();
info(strfmt("SalesTable");
break;
case tablenum(CustTable):
custTable = rll.peek();
info("CustTable");
break;
case tablenum(InventTrans):
inventTrans = rll.peek();
info("InventTrans");
break;
default:
error(strfmt("Table %1 (%2) not expected", tableid2name(rll.fileId()), rll.fileId()));
}
iterate = rll.next();
}
info("Done");
}
The insertDatabase method as stated (use the RecordInsertList class instead of RecordSortedList, if you do not need the sorted order):
inserts multiple records on a single trip to the database.
However this is mostly from the programmers perspective. The operation from the SQL goes like this:
INSERT INTO MyTable ( Column1, Column2 )
VALUES ( Value1, Value2 ),
( Value1, Value2 ), ...
There are limits to the number of records inserted this way, so the AX kernel may split the list to make several calls to the SQL server.
The other way from DB to AX is easy:
while select myTable where ...
Which is translated to SQL as:
SELECT T1.Column1, T1.Column2 FROM MyTable T1 WHERE...
This transports the data from the table to AX as efficient as possible.
You may choose to use a QueryRun object instead, but the call to SQL stays the same.
If you do simple updates on the table, consider using update_recordset as this may move the updates to the SQL server and eliminating the round-trip.
I'm having a small issue with sorting the data returned from a query, with the aim of getting the oldest updated value in dataset so that I can update only that record. Here's what I'm doing:
WHERE ROWNUM = 1 AND TABLE1.ID != V_IGNOREID
AND TABLE1.LASTREADTIME = (SELECT MIN(TABLE1.LASTREADTIME) FROM TABLE1)
ORDER BY TABLE1.LASTREADTIME DESC;
It makes no difference as to whether the ORDER BY statement is included or not. If I only use the ROWNUM and equality checks, I get data, but it alternates between only two rows, which is why I'm trying to use the LASTREADTIME data (so that I can modify more than these two rows). Anybody have any thoughts on this, or any suggestions as to how I can use the MIN function effectively?
Cheers
select * from (
-- your original select without rownum and with order by
)
WHERE ROWNUM = 1
EDIT some explanation
I think the order by clause is applied on the resultset after the where clause. So if the rownum = 1 is in the same select statement with the order by, then it will be applied first and the order by will order only 1 row, which will be the first row of the unordered resultset.