I have the below Hibernate NamedQuery that runs into issues when one of the "in" expressions has 1000 or more items. When I have 1000 or more items in the ma.deviceId in (:devices), I get java.sql.SQLException: ORA-01795: maximum number of expressions in a list is 1000
The only "in" expression that I need to take care of is the "and ma.deviceId in (:devices)". Anyone have any ideas on how to rewrite this NamedQuery? I'm using JPA and Hibernate.
#NamedQuery(name = "Messages.findMessages", query = " SELECT ma from Messages ma JOIN FETCH ma.messagePayLoadXml mx LEFT OUTER JOIN FETCH ma.messageProperties mp " +
" WHERE ma.customerId = :customerId and ma.time >= :startTime and ma.time <= :endTime " +
" and ma.deviceId in (:devices) and mx.messageType = 'XML' and mx.alerts in " +
" ( select mtfm.messageType from MessageTypeFeatureMap mtfm where mtfm.feature in (:featureType) ) " +
" and ma.messageKey = mx.messageKey and ( mp.deleted = 0 or mp.deleted is null ) " +
" order by ma.time desc " )
There are 2 ways.
1) Store your list in intermediary table and do
... IN (SELECT ... FROM intermediaryTable)
2) Break your list into sublists each upto 1000 elements and write your query as
(... IN (subList1) OR ... IN (subList2) ...)
For our application we have done a simple fix for this situation...
If number of values are greater than 999
QUERY = SELECT + FROM + WHERE + COND1 +
( FIELD IN ( 999 values ) OR FIELD IN ( 999 values )...) + ORDER
Else
QUERY = SELECT + FROM + WHERE + COND1 + IN (...) + ORDER
Related
I have a table with this structure:
CNMA_CO_PLATFORM_MESSAGE|AUDI_TI_CREATION|FIELD4|OTHER FIELDS
test-jj#2774#20210422112434957#00026129|22/04/21 11:24:34,957000000|11|..
test-jj2#2774#20210422112434957#00026129|22/04/21 11:24:34,957000000|12|..
test-jj3#2774#20210422112434957#00026129|22/04/21 11:24:34,957000000|13|..
This combination is the PRIMARY_KEY of the table:
CNMA_CO_PLATFORM_MESSAGE|AUDI_TI_CREATION
Well, I have an JdbcPagingItemReader defined like this (Pagesize is 1):
#StepScope
#Bean
public JdbcPagingItemReader<PendingNotificationDTO> pendingNotificationReader(
#Value("#{stepExecution}") StepExecution stepExecution){
final JdbcPagingItemReader<PendingNotificationDTO> reader = new JdbcPagingItemReader<>();
reader.setDataSource(daoDataSource);
reader.setName("pendingNotificationReader");
//Creamos la Query
final OraclePagingQueryProvider oraclePagingQueryProvider = new OraclePagingQueryProvider();
oraclePagingQueryProvider.setSelectClause("SELECT " +
" cegct.AUDI_TI_CREATION, "+
" CNMA_CO_PLATFORM_MESSAGE, " +
" OTHERFIELDS... ");
oraclePagingQueryProvider.setFromClause("FROM TABLE1 cegct " +
" JOIN TABLE1 notip ON cegct.field1 = notip.field1 " +
" AND notip.field2 = :frSur ");
oraclePagingQueryProvider.setWhereClause("WHERE "
+ " cegct.field3 = 0 "
+ " AND cegct.field4 in (:notifStatusList) ");
//Indicamos conjunto de campos no repetibles para poder paginar
Map<String, Order> sortKeys = new HashMap<>();
sortKeys.put("CNMA_CO_PLATFORM_MESSAGE", Order.DESCENDING);
sortKeys.put("AUDI_TI_CREATION", Order.DESCENDING);
oraclePagingQueryProvider.setSortKeys(sortKeys );
reader.setQueryProvider(oraclePagingQueryProvider);
String frSur = stepExecution.getJobExecution().getExecutionContext().getString(Constants.FM_ROLE_SUR_ZK);
String notifStatus = stepExecution.getJobExecution().getExecutionContext().getString(Constants.STATUS_REPORTS);
Map<String, Object> parameters = new HashMap<>();
parameters.put("frSur", frSur);
parameters.put("notifStatusList", Arrays.asList(StringUtils.split(notifStatus, ",")));
reader.setParameterValues(parameters );
Integer initLoaded = stepExecution.getJobExecution().getExecutionContext().getInt(Constants.RECOVER_PENDING_NOT_COMMIT);
reader.setPageSize(initLoaded);
reader.setRowMapper(new BeanPropertyRowMapper<PendingNotificationDTO>(PendingNotificationDTO.class));
return reader;
}
(I hide some irrelevant fields and table names)
Well, I run a test and my 3 records are valid to the select, these are selected one to one by the page size. Anyway, the first chunk-reader generated select my "test-jj3#..." record, my second chunk-reader select "test-jj2#.." and my third chunk-reader doesn't select doesn't recover any record (It should recover last 'test-jj#...' element.
These are the generated sqls (I hide some sensible no relevant fields)
First chunk, Select 1 register
SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION
CNMA_CO_PLATFORM_MESSAGE, [otherfields]
FROM [FROM]
WHERE [where]
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC
) WHERE ROWNUM <= 1;
Second chunk, Select 1 register (Here, the rownum filter by the sortkeys)
SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION
CNMA_CO_PLATFORM_MESSAGE, [otherfields]
FROM [FROM]
WHERE [where]
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC
) WHERE
ROWNUM <= 1 AND (
(CNMA_CO_PLATFORM_MESSAGE < 'test-jj3#2774#20210422112434957#00026129')
OR
(CNMA_CO_PLATFORM_MESSAGE = 'test-jj3#2774#20210422112434957#00026129' AND AUDI_TI_CREATION < TO_DATE('2021-04-22 11:24:34', 'YYYY-MM-DD HH24:MI:SS'))
);
Third chunk, select 0 registers
SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION
CNMA_CO_PLATFORM_MESSAGE, [otherfields]
FROM [FROM]
WHERE [where]
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC
) WHERE
ROWNUM <= 1 AND (
(CNMA_CO_PLATFORM_MESSAGE < 'test-jj2#2774#20210422112434957#00026129')
OR
(CNMA_CO_PLATFORM_MESSAGE = 'test-jj2#2774#20210422112434957#00026129' AND AUDI_TI_CREATION < TO_DATE('2021-04-22 11:24:34', 'YYYY-MM-DD HH24:MI:SS'))
);
Sorry for my english, I hope you can understand my problem.
Logs for the Prepared SQL Statement
Executing prepared SQL statement [SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION,
CNMA_CO_PLATFORM_MESSAGE,
OTHERFIELDS...
FROM TABLE1 cegct
JOIN TABLE2 notip ON cegct.field1 = notip.field1
AND notip.field2 = ?
WHERE cegct.field3 = 0
AND cegct.field4 in (?, ?, ?)
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC) WHERE ROWNUM <= 1]
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 1, parameter value [1], value class [java.lang.String], SQL type unknown
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 2, parameter value [11], value class [java.lang.String], SQL type unknown
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 3, parameter value [12], value class [java.lang.String], SQL type unknown
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 4, parameter value [13], value class [java.lang.String], SQL type unknown
A bind variable is a single value; therefore when you use:
AND cegct.field4 in (:notifStatusList)
Then :notifStatusList is a single string and is NOT a list of values and you effectively doing the same as:
AND cegct.field4 = :notifStatusList
If the bind variable :notifStatusList is a single value then it will work; however, when you try to pass in multiple values then it will not match those multiple values but will try to match field4 to the entire delimited list (which fails and will filter out all the rows).
If you want to pass a delimited string then use:
AND ',' || :notifStatusList || ',' LIKE '%,' || cegct.field4 || ',%'
Alternatively, pass the values as an array (rather than a delimited string) into an Oracle collection and then test to see if it is in that collection.
I have a linq query
from c in db.Custommer
join m in db.Membership on c.ID equals m.CustomerID
select (c.LastName + ", " + c.FirstName + " " + c.MiddleName);
The MiddleName could be NULL, how do I replace that null with a space or ignore it?
If I leave it this way, the query does not return any records for customers who don't have middle names.
You can do as such:
from c in db.Custommer
join m in db.Membership on c.ID equals m.CustomerID
select (c.LastName + ", " + c.FirstName + " " + (c.MiddleName ?? "");
This should do the trick :)
I'm trying to run a query with NOT IN clause like:
SELECT * FROM table WHERE column NOT IN (?,?,...) (>1000 items) and I'm getting ORA-01795: maximum number of expressions in a list is 1000 error.
In my script I'm doing something like:
my $lparam = join ', ' => ('?') x #ids;
$lquery = "SELECT * FROM table WHERE column NOT IN ($lparam)";
$lcsr = $zdb->prepare($lquery);
$lcsr->execute( #ids );
I want to split the NOT IN clause to something like where (A not in (a,b,c) AND A not in (d,e,f)) ... How can we achieve this?
Here you go, adding triples and counting them.
my $count = 0;
$lquery = "SELECT * FROM table WHERE (A ";
while (#ids -$count > 3) {
$lquery .= "NOT in (?, ?, ?) AND A ";
$count += 3;
}
my $lparam = join ', ' => ('?') x (#ids - $count);
$lquery .= "NOT IN ($lparam))";
You can go with the IN list using a combination of the column as follows:
SELECT * FROM table WHERE (column,1) NOT IN ((?,1),(?,1),...) (>1000 items)
Here, 1 is used as the second column. And you can give more than 1000 values in the IN clause list. It is a workaround of skipping the limit.
I'd like to show the number of records in the history table grouped by name of service
service(code,name)
history(id, code,....)
Please note that there is no relationship between the two table history and service, history stores the activity independently from the other tables
I have tested this sql query and it returns the expected result:
select s.name, count(*) from history c
join service s
on c.code=s.code
where c.state='INITIALE'
group by s.name
Actually, I'd like to write it in jpql, I did alike
Query query =entityManager.createQuery(" select s.name, count(*) from ServiceEntity s join"
+ " HistoryEntity c "
+ " where c.code=s.code and c.state='INITIALE'"
+ " group by c.name order by c.name"
);
I got this error : Path expected for join!....
Invalid path: 'c.code'....right-hand operand of a binary operator was null....unexpected end of subtree
Try this
Query query = entityManager.createQuery("select s.name, count(s) from ServiceEntity s, HistoryEntity c "
+ " where c.code = s.code and c.state = 'INITIALE'"
+ " group by s.name order by s.name"
);
qry1 = "Select * from ser_complaint_master a,ser_complaint_status b,company_master c
where a.complaint_no=b.complaint_no
and a.allocation_code=c.co_code
and c.co_br_code='" + Session["BRCODE"] + "'
and a.Complaint_Date>='" + Frdat + "' and a.Complaint_Date<='" + Todat + "'
and a.status in ('Completed')
and a.complaint_type in('" + cmptype + "')";
How to use ORDER BY in select query if more than one tables are involved.
Add order by a.complaint_no to the end of the query.