JDBC Batch Insert with Returning Clause - oracle

Is there any way to get the values of affected rows using returning clause in JAVA while using JDBC Batch Insert statement? I am able to get the required values of a single row affected.But not for all Batch Inserts?
Code :
try {
String query = "INSERT INTO temp ( "
+ "org_node_id, org_node_category_id, org_node_name, "
+ "customer_id, created_by, created_date_time, "
+ "updated_date_time, activation_Status )"
+ " VALUES (seq_org_node_id.nextval, 11527, 'Abcd', 9756, 1, sysdate, sysdate, 'AC')"
+" returning org_node_id, org_node_name INTO ?, ?";
con = DBUtils.getOASConnection();
OraclePreparedStatement ps = (OraclePreparedStatement) con.prepareStatement(query);
ps.registerReturnParameter(1, Types.INTEGER);
ps.registerReturnParameter(2, Types.VARCHAR);
ps.execute();
ResultSet rs = ps.getReturnResultSet();
rs.next();
System.out.println("Org ID : "+ rs.getInt(1));
System.out.println("Org Name : "+ rs.getString(2));
} catch (SQLException e) {
e.printStackTrace();
}

Batching INSERT .. RETURNING statements isn't supported by ojdbc, but bulk insertion can work using PL/SQL's FORALL command.
Given a table...
CREATE TABLE x (
i INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
j VARCHAR2(50),
k DATE DEFAULT SYSDATE
);
...and types...
CREATE TYPE t_i AS TABLE OF NUMBER(38);
/
CREATE TYPE t_j AS TABLE OF VARCHAR2(50);
/
CREATE TYPE t_k AS TABLE OF DATE;
/
...you can work around this limitation by running a bulk insert, and bulk collecting the results (as I've shown also in this blog post) like this:
try (Connection con = DriverManager.getConnection(url, props);
CallableStatement c = con.prepareCall(
"DECLARE "
+ " v_j t_j := ?; "
+ "BEGIN "
+ " FORALL j IN 1 .. v_j.COUNT "
+ " INSERT INTO x (j) VALUES (v_j(j)) "
+ " RETURNING i, j, k "
+ " BULK COLLECT INTO ?, ?, ?; "
+ "END;")) {
// Bind input and output arrays
c.setArray(1, ((OracleConnection) con).createARRAY(
"T_J", new String[] { "a", "b", "c" })
);
c.registerOutParameter(2, Types.ARRAY, "T_I");
c.registerOutParameter(3, Types.ARRAY, "T_J");
c.registerOutParameter(4, Types.ARRAY, "T_K");
// Execute, fetch, and display output arrays
c.execute();
Object[] i = (Object[]) c.getArray(2).getArray();
Object[] j = (Object[]) c.getArray(3).getArray();
Object[] k = (Object[]) c.getArray(4).getArray();
System.out.println(Arrays.asList(i));
System.out.println(Arrays.asList(j));
System.out.println(Arrays.asList(k));
}
The results are:
[1, 2, 3]
[a, b, c]
[2018-05-02 10:40:34.0, 2018-05-02 10:40:34.0, 2018-05-02 10:40:34.0]

Related

JdbcPagingItemReader Spring batch skipping last element

I have a table with this structure:
CNMA_CO_PLATFORM_MESSAGE|AUDI_TI_CREATION|FIELD4|OTHER FIELDS
test-jj#2774#20210422112434957#00026129|22/04/21 11:24:34,957000000|11|..
test-jj2#2774#20210422112434957#00026129|22/04/21 11:24:34,957000000|12|..
test-jj3#2774#20210422112434957#00026129|22/04/21 11:24:34,957000000|13|..
This combination is the PRIMARY_KEY of the table:
CNMA_CO_PLATFORM_MESSAGE|AUDI_TI_CREATION
Well, I have an JdbcPagingItemReader defined like this (Pagesize is 1):
#StepScope
#Bean
public JdbcPagingItemReader<PendingNotificationDTO> pendingNotificationReader(
#Value("#{stepExecution}") StepExecution stepExecution){
final JdbcPagingItemReader<PendingNotificationDTO> reader = new JdbcPagingItemReader<>();
reader.setDataSource(daoDataSource);
reader.setName("pendingNotificationReader");
//Creamos la Query
final OraclePagingQueryProvider oraclePagingQueryProvider = new OraclePagingQueryProvider();
oraclePagingQueryProvider.setSelectClause("SELECT " +
" cegct.AUDI_TI_CREATION, "+
" CNMA_CO_PLATFORM_MESSAGE, " +
" OTHERFIELDS... ");
oraclePagingQueryProvider.setFromClause("FROM TABLE1 cegct " +
" JOIN TABLE1 notip ON cegct.field1 = notip.field1 " +
" AND notip.field2 = :frSur ");
oraclePagingQueryProvider.setWhereClause("WHERE "
+ " cegct.field3 = 0 "
+ " AND cegct.field4 in (:notifStatusList) ");
//Indicamos conjunto de campos no repetibles para poder paginar
Map<String, Order> sortKeys = new HashMap<>();
sortKeys.put("CNMA_CO_PLATFORM_MESSAGE", Order.DESCENDING);
sortKeys.put("AUDI_TI_CREATION", Order.DESCENDING);
oraclePagingQueryProvider.setSortKeys(sortKeys );
reader.setQueryProvider(oraclePagingQueryProvider);
String frSur = stepExecution.getJobExecution().getExecutionContext().getString(Constants.FM_ROLE_SUR_ZK);
String notifStatus = stepExecution.getJobExecution().getExecutionContext().getString(Constants.STATUS_REPORTS);
Map<String, Object> parameters = new HashMap<>();
parameters.put("frSur", frSur);
parameters.put("notifStatusList", Arrays.asList(StringUtils.split(notifStatus, ",")));
reader.setParameterValues(parameters );
Integer initLoaded = stepExecution.getJobExecution().getExecutionContext().getInt(Constants.RECOVER_PENDING_NOT_COMMIT);
reader.setPageSize(initLoaded);
reader.setRowMapper(new BeanPropertyRowMapper<PendingNotificationDTO>(PendingNotificationDTO.class));
return reader;
}
(I hide some irrelevant fields and table names)
Well, I run a test and my 3 records are valid to the select, these are selected one to one by the page size. Anyway, the first chunk-reader generated select my "test-jj3#..." record, my second chunk-reader select "test-jj2#.." and my third chunk-reader doesn't select doesn't recover any record (It should recover last 'test-jj#...' element.
These are the generated sqls (I hide some sensible no relevant fields)
First chunk, Select 1 register
SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION
CNMA_CO_PLATFORM_MESSAGE, [otherfields]
FROM [FROM]
WHERE [where]
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC
) WHERE ROWNUM <= 1;
Second chunk, Select 1 register (Here, the rownum filter by the sortkeys)
SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION
CNMA_CO_PLATFORM_MESSAGE, [otherfields]
FROM [FROM]
WHERE [where]
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC
) WHERE
ROWNUM <= 1 AND (
(CNMA_CO_PLATFORM_MESSAGE < 'test-jj3#2774#20210422112434957#00026129')
OR
(CNMA_CO_PLATFORM_MESSAGE = 'test-jj3#2774#20210422112434957#00026129' AND AUDI_TI_CREATION < TO_DATE('2021-04-22 11:24:34', 'YYYY-MM-DD HH24:MI:SS'))
);
Third chunk, select 0 registers
SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION
CNMA_CO_PLATFORM_MESSAGE, [otherfields]
FROM [FROM]
WHERE [where]
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC
) WHERE
ROWNUM <= 1 AND (
(CNMA_CO_PLATFORM_MESSAGE < 'test-jj2#2774#20210422112434957#00026129')
OR
(CNMA_CO_PLATFORM_MESSAGE = 'test-jj2#2774#20210422112434957#00026129' AND AUDI_TI_CREATION < TO_DATE('2021-04-22 11:24:34', 'YYYY-MM-DD HH24:MI:SS'))
);
Sorry for my english, I hope you can understand my problem.
Logs for the Prepared SQL Statement
Executing prepared SQL statement [SELECT * FROM (
SELECT
cegct.AUDI_TI_CREATION,
CNMA_CO_PLATFORM_MESSAGE,
OTHERFIELDS...
FROM TABLE1 cegct
JOIN TABLE2 notip ON cegct.field1 = notip.field1
AND notip.field2 = ?
WHERE cegct.field3 = 0
AND cegct.field4 in (?, ?, ?)
ORDER BY CNMA_CO_PLATFORM_MESSAGE DESC, AUDI_TI_CREATION DESC) WHERE ROWNUM <= 1]
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 1, parameter value [1], value class [java.lang.String], SQL type unknown
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 2, parameter value [11], value class [java.lang.String], SQL type unknown
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 3, parameter value [12], value class [java.lang.String], SQL type unknown
20221116 12:52:43.560 TRACE org.springframework.jdbc.core.StatementCreatorUtils [[ # ]] - Setting SQL statement parameter value: column index 4, parameter value [13], value class [java.lang.String], SQL type unknown
A bind variable is a single value; therefore when you use:
AND cegct.field4 in (:notifStatusList)
Then :notifStatusList is a single string and is NOT a list of values and you effectively doing the same as:
AND cegct.field4 = :notifStatusList
If the bind variable :notifStatusList is a single value then it will work; however, when you try to pass in multiple values then it will not match those multiple values but will try to match field4 to the entire delimited list (which fails and will filter out all the rows).
If you want to pass a delimited string then use:
AND ',' || :notifStatusList || ',' LIKE '%,' || cegct.field4 || ',%'
Alternatively, pass the values as an array (rather than a delimited string) into an Oracle collection and then test to see if it is in that collection.

how to import huge tsv file into h2 in memory database with spring boot

I have a huge tsv files and I need to import them into my h2 in memory database.
I can read it with Scanner and import it line by line but it takes for hours !
is there any faster way to import tsv file into h2 in memory database ?
Use insert into select convert for direct importing from file into your h2 table.
How to read CSV file into H2 database :
public static void main (String [] args) throws Exception {
Connection conn = null;
Statement stmt = null;
Class.forName("org.h2.Driver");
conn = DriverManager.getConnection("jdbc:h2:~/test", "", "");
stmt = conn.createStatement();
stmt.execute("drop table if exists csvdata");
stmt.execute("create table csvdata (id int primary key, name varchar(100), age int)");
stmt.execute("insert into csvdata ( id, name, age ) select convert( \"id\",int ), \"name\", convert( \"age\", int) from CSVREAD( 'c:\\tmp\\sample.csv', 'id,name,age', null ) ");
ResultSet rs = stmt.executeQuery("select * from csvdata");
while (rs.next()) {
System.out.println("id " + rs.getInt("id") + " name " + rs.getString("name") + " age " + rs.getInt("age") );
}
stmt.close();
}
Or
SELECT * FROM CSVREAD('test.csv');
-- Read a file containing the columns ID, NAME with
SELECT * FROM CSVREAD('test2.csv', 'ID|NAME', 'charset=UTF-8 fieldSeparator=|');
SELECT * FROM CSVREAD('data/test.csv', null, 'rowSeparator=;');
-- Read a tab-separated file
SELECT * FROM CSVREAD('data/test.tsv', null, 'rowSeparator=' || CHAR(9));
SELECT "Last Name" FROM CSVREAD('address.csv');
SELECT "Last Name" FROM CSVREAD('classpath:/org/acme/data/address.csv');
h2 csvread function
NOTE: You can specify file's field separator for these commands.

SpringJDBC Gives ORA-00933: SQL command not properly ended But Query Runs OK in DB Client

The following Oracle query runs OK in my DB Client, PL/SQL Developer, and returns 1 result.
On running it via NamedParameterJdbcTemplate (SpringJDBC) in my Java app, I get
java.sql.SQLSyntaxErrorException: ORA-00933: SQL command not properly ended
There can't be any space issues or anything obvious, because this exact query completes in PL/SQL.
private static final String SELECT1 =
" SELECT COUNT(*) "
" FROM table1 t1, table2 t2 " +
" WHERE t1.received_date > TRUNC(sysdate - 1) " +
" AND t1.received_date < TRUNC(sysdate) " +
" AND t1.type IN ('TYPE1', 'TYPE2') " +
" AND t2.received_num = t1.received_num; ";
public int getSelect1() {
HashMap<String,Object> paramMap = new HashMap<String,Object>();
return jdbcTemplate.queryForObject(SELECT1, paramMap, Integer.class);
}
I think you don't require the semi colon with the sql string.
private static final String SELECT1 =
" SELECT COUNT(*) " +
" FROM table1 t1, table2 t2 " +
" WHERE t1.received_date > TRUNC(sysdate - 1) " +
" AND t1.received_date < TRUNC(sysdate) " +
" AND t1.type IN ('TYPE1', 'TYPE2') " +
" AND t2.received_num = t1.received_num ";

JPA and Hibernate NamedQuery Error: maximum number of expressions in a list is 1000

I have the below Hibernate NamedQuery that runs into issues when one of the "in" expressions has 1000 or more items. When I have 1000 or more items in the ma.deviceId in (:devices), I get java.sql.SQLException: ORA-01795: maximum number of expressions in a list is 1000
The only "in" expression that I need to take care of is the "and ma.deviceId in (:devices)". Anyone have any ideas on how to rewrite this NamedQuery? I'm using JPA and Hibernate.
#NamedQuery(name = "Messages.findMessages", query = " SELECT ma from Messages ma JOIN FETCH ma.messagePayLoadXml mx LEFT OUTER JOIN FETCH ma.messageProperties mp " +
" WHERE ma.customerId = :customerId and ma.time >= :startTime and ma.time <= :endTime " +
" and ma.deviceId in (:devices) and mx.messageType = 'XML' and mx.alerts in " +
" ( select mtfm.messageType from MessageTypeFeatureMap mtfm where mtfm.feature in (:featureType) ) " +
" and ma.messageKey = mx.messageKey and ( mp.deleted = 0 or mp.deleted is null ) " +
" order by ma.time desc " )
There are 2 ways.
1) Store your list in intermediary table and do
... IN (SELECT ... FROM intermediaryTable)
2) Break your list into sublists each upto 1000 elements and write your query as
(... IN (subList1) OR ... IN (subList2) ...)
For our application we have done a simple fix for this situation...
If number of values are greater than 999
QUERY = SELECT + FROM + WHERE + COND1 +
( FIELD IN ( 999 values ) OR FIELD IN ( 999 values )...) + ORDER
Else
QUERY = SELECT + FROM + WHERE + COND1 + IN (...) + ORDER

linq to sql: issue with linq query

I have two tables
EmpInf
EmpId,
EmpName,
Salary,
DepartNumber.
Dept
DeptNo,
Deptname,
I also have a listview1 and dropdownlist1 which is bound to EmpInf.EmpName
While passing a particular query
FilterControl.DataClasses1DataContext obj = new DataClasses1DataContext();
protected void DropDownList1_SelectedIndexChanged1(object sender, EventArgs e)
{
var a = from r in obj.EmpInfs join s in obj.Dept1s on r.DeptNumber equals s.DeptNo where r.EmpName == "'" + DropDownList1.SelectedValue + "'" select s;
ListView1.DataSource = a;
ListView1.DataBind();
}
Whenever I select a particular name from a dropdownlist, it returns No data was returned. What particular code am I missing or is there any other error?
Put a breakpoint in SelectedIndexChanged1 and look at the value of SelectedValue to make sure DropDownList1.SelectedValue has the Employee name. You can also try DropDownList1.SelectedText.
string selected = DropDownList1.SelectedValue.ToString();
// e = employee | d = department
var query =
from e in obj.EmpInfs
join d in obj.Dept1s on e.DeptNumber equals d.DeptNo
where e.EmpName == "'" + DropDownList1.SelectedValue + "'"
select d;
Change this line:
where e.EmpName == "'" + DropDownList1.SelectedValue + "'"
to this one:
where e.EmpName == selected
OK, my last attempt here... Do this before databinding:
ListView1.DataSource = query.ToList();
Most likely the problem here is with your addition of the single quotes on the search criteria in your Where clause:
var a = from r in obj.EmpInfs
join s in obj.Dept1s on r.DeptNumber equals s.DeptNo
where r.EmpName == "'" + DropDownList1.SelectedValue + "'"
select s;
Assuming DropDownList1.SelectedValue is "Smith" then your generated sql will be along the lines of:
SELECT *
FROM <tables>
WHERE EmpName = ''Smith''
Notice the double single quotes. To double check this, put a breakpoint after your query is generated and then call .ToString() on it to get the equivalent TSQL. To fix this, remove the "'" from your LINQ query as that will be automatically added for string parameters:
var a = from r in obj.EmpInfs
join s in obj.Dept1s on r.DeptNumber equals s.DeptNo
where r.EmpName == DropDownList1.SelectedValue
select s;

Resources