Optimizing database trips for data retrieval - dynamics-ax-2009

For multiple data insertion we have an efficient way: RecordSortedList
RecordSortedList rsl;
MyTable myTable;
;
rsl = new RecordSortedList(myTable.tableid);
rsl.sortOrder(fieldname2id(myTable.tableId,'RecId'));
myTable.field1 = 'Value1';
rsl.ins(myTable);
myTable.field1 = 'Value2';
rsl.ins(myTable);
rsl.insertDatabase();
Is the same possible for multiple records retrieval from db in one go? Something like
int i =1;
while(i<10000)
{
//enter records from db into a buffer in db
i++
}
//now bring the buffer from db in a single trip
//and do the data manipulation in AX
My intention is to optimize the db trip to the least.
Please Suggest.

Yes, it's called RecordLinkList - http://msdn.microsoft.com/en-us/library/aa643250(v=ax.50).aspx
A recordLinkList is a double linked list that can hold records of
different types at the same time. It is not keyed or sorted.
The recordLinkList is particularly useful for passing records from
different tables as a parameter instead of retrieving the same records
again.
There is no limit to the size of a recordSortedList; it is the
responsibility of the programmer to control its size and, therefore,
memory consumption.
You can also add different types of records.
static void RecordLinkList(Args _args)
{
RecordLinkList rll = new RecordLinkList();
SalesTable salesTable;
CustTable custTable;
InventTrans inventTrans;
Address address;
boolean iterate;
;
select firstonly salesTable;
select firstonly custTable;
select firstonly inventTrans;
select firstonly address;
rll.ins(salesTable);
rll.ins(custTable);
rll.ins(inventTrans);
rll.ins(address);
iterate = rll.first();
while (iterate)
{
switch (rll.fileId()) // FileId == TableId
{
case tablenum(SalesTable):
salesTable = rll.peek();
info(strfmt("SalesTable");
break;
case tablenum(CustTable):
custTable = rll.peek();
info("CustTable");
break;
case tablenum(InventTrans):
inventTrans = rll.peek();
info("InventTrans");
break;
default:
error(strfmt("Table %1 (%2) not expected", tableid2name(rll.fileId()), rll.fileId()));
}
iterate = rll.next();
}
info("Done");
}

The insertDatabase method as stated (use the RecordInsertList class instead of RecordSortedList, if you do not need the sorted order):
inserts multiple records on a single trip to the database.
However this is mostly from the programmers perspective. The operation from the SQL goes like this:
INSERT INTO MyTable ( Column1, Column2 )
VALUES ( Value1, Value2 ),
( Value1, Value2 ), ...
There are limits to the number of records inserted this way, so the AX kernel may split the list to make several calls to the SQL server.
The other way from DB to AX is easy:
while select myTable where ...
Which is translated to SQL as:
SELECT T1.Column1, T1.Column2 FROM MyTable T1 WHERE...
This transports the data from the table to AX as efficient as possible.
You may choose to use a QueryRun object instead, but the call to SQL stays the same.
If you do simple updates on the table, consider using update_recordset as this may move the updates to the SQL server and eliminating the round-trip.

Related

EclipseLink with Oracle: "limit by rownum" does not use index

we're facing performance issues with EclipseLink 2.7.7 when accessing Oracle 12.1 tables with paging. Investigation showed that Oracle does not use its indexes with EclipseLink paging.
I've extracted the sql sent to the database and was able to reproduce the issue using a database tool (DataGrip).
Example:
-- #1: without paging
SELECT col1 AS a1, col2 AS a2, col3 AS a3, ...
FROM <TABLE>
WHERE colN > to_timestamp('2021-12-08', 'yyyy-mm-dd'))
ORDER BY col1 DESC;
Explain plan shows that the index on colN is used. Fine.
When the same query is executed with paging, the original query is wrapped in two subselects:
-- #2 with EclipseLink paging
SELECT * FROM (
SELECT a.*, ROWNUM rnum FROM (
SELECT col1 AS a1, col2 AS a2, col3 AS a3, ...
FROM <TABLE>
WHERE colN > to_timestamp('2021-12-08', 'yyyy-mm-dd'))
ORDER BY col1 DESC
) a WHERE ROWNUM <= 100
) WHERE rnum > 0;
For this query, the explain plan shows that the index on colN is not used.
As a result, querying a table with millions of rows takes 50-90 seconds (depending on the hardware).
Side note: on my test database, this query returns 0 records since colN values are before 2021-12-08.
Oracle 12c introduced the OFFSET/FETCH syntax:
-- #3
SELECT col1 AS a1, col2 AS a2, col3 AS a3, ...
FROM <TABLE>
WHERE colN > to_timestamp('2021-12-08', 'yyyy-mm-dd'))
ORDER BY col1 DESC
OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY;
Using this syntax, indexes are at least sometimes used as expected. When they are used, execution time is below 1s which is acceptable.
However, I could not figure out how to convince EclipseLink to use this syntax.
If ORDER BY col1 DESC is removed from the original paged query (#2), the index is used the query returns fast enough. However, it will not return the desired records, so that does not help.
How can I implement performant paged queries using EclipseLink and Oracle 12?
How can I force oracle to use the index on colN when using paging and order by?
The OraclePlatform printSQLSelectStatement method is responsible for building the query used, nesting the queries to use rownum for the query you've seen. To use a new form, you would extend one of the OraclePlatform classes you are using (maybe Oracle12Platform) and override that method to append the syntax you want instead. Something like:
#Override
public void printSQLSelectStatement(DatabaseCall call, ExpressionSQLPrinter printer, SQLSelectStatement statement) {
int max = 0;
int firstRow = 0;
ReadQuery query = statement.getQuery();
if (query != null) {
max = query.getMaxRows();
firstRow = query.getFirstResult();
}
if (!(this.shouldUseRownumFiltering()) || (!(max > 0) && !(firstRow > 0))) {
super.printSQLSelectStatement(call, printer, statement);
return;
}
call.setFields(statement.printSQL(printer));
printer.printString("OFFSET ");
printer.printParameter(DatabaseCall.MAXROW_FIELD);
printer.printString(" ROWS FETCH NEXT ");
printer.printParameter(DatabaseCall.FIRSTRESULT_FIELD);
printer.printString(" ROWS ONLY");
call.setIgnoreFirstRowSetting(true);
call.setIgnoreMaxResultsSetting(true);
}
You would then specify your custom OraclePlatform class using a persistent property:
<property name="eclipselink.target-database" value="my.package.MyOracle12Platform"/>
If something like that works for you, please submit it as an enhancement request - though you might want to work some way to use the old behaviour into it, as the performance differences you've experienced might depend on the query/data involved.
Thanks to #Chris, I came up with the following Oracle12Platform. This solution currently ignores "Bug #453208 - Pessimistic locking with query row limits does not work on Oracle DB". See OraclePlatform.printSQLSelectStatement for details):
public class Oracle12Platform extends org.eclipse.persistence.platform.database.Oracle12Platform {
/**
* the oracle 12c `OFFSET x ROWS FETCH NEXT y ROWS ONLY` requires `maxRows` to return the row count
*/
#Override
public int computeMaxRowsForSQL(final int firstResultIndex, final int maxResults) {
return maxResults - max(firstResultIndex, 0);
}
#Override
public void printSQLSelectStatement(final DatabaseCall call, final ExpressionSQLPrinter printer, final SQLSelectStatement statement) {
int max = 0;
int firstRow = 0;
final ReadQuery query = statement.getQuery();
if (query != null) {
max = query.getMaxRows();
firstRow = query.getFirstResult();
}
if (!(this.shouldUseRownumFiltering()) || (!(max > 0) && !(firstRow > 0))) {
super.printSQLSelectStatement(call, printer, statement);
} else {
statement.setUseUniqueFieldAliases(true);
call.setFields(statement.printSQL(printer));
if (firstRow > 0) {
printer.printString(" OFFSET ");
printer.printParameter(DatabaseCall.FIRSTRESULT_FIELD);
printer.printString(" ROWS");
call.setIgnoreFirstRowSetting(true);
}
if (max > 0) {
printer.printString(" FETCH NEXT ");
printer.printParameter(DatabaseCall.MAXROW_FIELD); //see #computeMaxRowsForSQL
printer.printString(" ROWS ONLY");
call.setIgnoreMaxResultsSetting(true);
}
}
}
}
I had to override computeMaxRowsForSQL in order to get the row count instead of "lastRowNum" when calling printer.printParameter(DatabaseCall.MAXROW_FIELD);
I also try to deal with missing firstRow xor maxResults

How to count for a substring over two tables in codeigniter

I have two tables
table1
id|name|next_field
table2
id|id_of_table1|whatelse
I do a msql query to get all entries of table1 and the number of entries in table2 who has table2.id_of_table1 = table1.id
This is my query - it works fine.
$select =array('table1.*', 'COUNT(table2.id) AS `my_count_result`',);
$this->db->select($select);
if($id!=false){ $this->db->where('id',$id); }
$this->db->from('table1 as t1');
$this->db->join('table2 as t2', 't1.id = t2.id_of_table1');
return $this->db->get()->result_array();
Now I have another field which has coma-separated data
table1.next_field = info1, info2, next,...
Now I want to check in the same way like the first query how often for example "info2" is as a part inside the table1.next_field
Is it possible, how?
After all i decide to change my database structure to make the work and handle much easier.

Hibernate returning two columns instead one when using setFirstResults & setMaxResults method with Oracle

I have a query that selects a single column and I am executing the query in batches using setFirstResults & setMaxResults methods of SQLQuery.
Snippet:
SQLQuery query = <query object with query projecting a single column>;
int maxResults = 50;
int batchSize = 50;
for (int i = 0; ; i++) {
query.setFirstResult(batchSize*i);
query.setMaxResults(maxResults);
List resultSet = query.list();
if(resultSet.isEmpty())
break;
//process result set
}
I set to true the showSQL parameter in hibernate config to see the query string that hibernate produces. For the first batch, i.e. when i=0 below is the query that hibernate generates:
select * from (/* query selecting single column here */) where rownum <= ?;
which makes sense since its the first batch and we want results from first row and rownum is used to restrict the number of results to maxResults.
Now for the second and subsequent batch reads, the query hibernate generates is:
select * from ( select row_.*, rownum rownum_ from (/*query selecting single column here */) row_ where rownum <= ?) where rownum_ > ?;
and you can clearly see, the above query is selecting two columns, one being the row number itself.
So when my query only selects one column, hibernate's version of the query is selecting two.
Is this known issue? Can I do something different or am I doing something wrong?
I don't want to cast the result set into two different types before using/processing it.

Oracle Table Variables

Using Oracle PL/SQL is there a simple equivalent of the following set of T-SQL statements? It seems that everything I am finding is either hopelessly outdated or populates a table data type with no explanation on how to use the result other than writing values to stdout.
declare #tempSites table (siteid int)
insert into #tempSites select siteid from site where state = 'TX'
if 10 > (select COUNT(*) from #tempSites)
begin
insert into #tempSites select siteid from site where state = 'OK'
end
select * from #tempSites ts inner join site on site.siteId = ts.siteId
As #AlexPoole points out in his comment, this is a fairly contrived example.
What I am attempting to do is get all sites that meet a certain set of criteria, and if there are not enough matches, then I am looking to use a different set of criteria.
Oracle doesn't have local temporary tables, and global temporary tables don't look appropriate here.
You could use a common table expression (subquery factoring):
with tempSites (siteId) as (
select siteid
from site
where state = 'TX'
union all
select siteid
from site
where state = 'OK'
and (select count(*) from site where state = 'TX') < 10
)
select s.*
from tempSites ts
join site s on s.siteid = ts.siteid;
That isn't quite the same thing, but gets all the TX IDs, and only includes the OK ones if the count of TX ones - which has to be repeated - is less than 10. The CTE is then joined back to the original table, which all seems a bit wasteful; you're hitting the same table three times.
You could use a subquery directly in a filter instead:
select *
from site
where state = 'TX'
or (state = 'OK'
and (select count(*) from site where state = 'TX') < 10);
but again the TX sites have to be retrieved (or at least counted) a second time.
You can do this with a single hit of the table using an inline view (or CTE if you prefer) with an analytic count - which add the count of TX rows to the columns in the actual table, so you'd probably want to exclude that dummy column from the final result set (but using * is bad practice anyway):
select * -- but list columns, excluding tx_count
from (
select s.*,
count(case when state = 'TX' then state end) over (partition by null) as tx_count
from site s
where s.state in ('TX', 'OK')
)
where state = 'TX'
or (state = 'OK' and tx_count < 10);
From your description of your research it sounds like what you've been looking at involved PL/SQL code populating a collection, which you could still do, but it's probably overkill unless your real situation is much more complicated.

SQL Select query Performance improvement

I've a bunch of queries like this :
-- get all data that exists in source but not yet in destination
SELECT
*
INTO #temp
FROM source T010T
WHERE NOT EXISTS
(
SELECT TOP 1 1 FROM destination P510T
WHERE WH_CD = T010T.WH_CD
AND POS_NO = T010T.POS_NO
AND SLIP_NO = T010T.TRAN_NO
AND OPE_DATE = T010T.SL_REC_DATE
)
-- process the data
....
-- insert data into destination
Insert into destination select * From #temp
I'm wondering will this way of approach will affect the performance? Because I haven't got the real data to test and this is running locally so I'm kind of scared that when put into reality, those queries will be a pain in the a##!
Is there any better alternatives?
p/s : columns on both table used in the comparison are all primary keys primarykey(wh_cd,pos_no,slip_no,ope_date) ...
Try with Left Join instead:
SELECT
*
INTO #temp
FROM source T010T
LEFT JOIN destination P510T
ON WH_CD = T010T.WH_CD
AND POS_NO = T010T.POS_NO
AND SLIP_NO = T010T.TRAN_NO
AND OPE_DATE = T010T.SL_REC_DATE
WHERE P510T.WH_CD IS NULL

Resources