I am trying to update certain rows in my oracle DB using id which is of RAW(255).
Sample ids 0BF3957A016E4EBCB68809E6C2EA8B80, 1199B9F29F0A46F486C052669854C2F8...
#Autowired
private NamedParameterJdbcTemplate jdbcTempalte;
private static final String UPDATE_SUB_STATUS = "update SUBSCRIPTIONS set status = :status, modified_date = systimestamp where id in (:ids)";
public void saveSubscriptionsStatus(List<String> ids, String status) {
MapSqlParameterSource paramSource = new MapSqlParameterSource();
List<String> idsHexToRaw = new ArrayList<>();
String temp = new String();
for (String id : ids) {
temp = "hextoraw('" + id + "')";
idsHexToRaw.add(temp);
}
paramSource.addValue("ids", idsHexToRaw);
paramSource.addValue("status", status);
jdbcTempalte.update(*UPDATE_SUB_STATUS*, paramSource);
}
This above block of code is executing without any error but the updates are not reflected to the db, while if I skip using hextoraw() and just pass the list of ids it works fine and also updates the data in table. see below code
public void saveSubscriptionsStatus(List<String> ids, String status) {
MapSqlParameterSource paramSource = new MapSqlParameterSource();]
paramSource.addValue("ids", ids);
paramSource.addValue("status", status);
jdbcTempalte.update(UPDATE_SUB_STATUS, paramSource);
}
this code works fine and updates the table, but since i am not using hextoraw() it scans the full table for updation which I don't want since i have created indexes. So using hextoraw() will use index for scanning the table but it is not updating the values which is kind of weird.
Got a solution myself by trying all the different combinations :
#Autowired
private NamedParameterJdbcTemplate jdbcTempalte;
public void saveSubscriptionsStatus(List<String> ids, String status) {
String UPDATE_SUB_STATUS = "update SUBSCRIPTIONS set status = :status, modified_date = systimestamp where id in (";
MapSqlParameterSource paramSource = new MapSqlParameterSource();
String subQuery = "";
for (int i = 0; i < ids.size(); i++) {
String temp = "id" + i;
paramSource.addValue(temp, ids.get(i));
subQuery = subQuery + "hextoraw(:" + temp + "), ";
}
subQuery = subQuery.substring(0, subQuery.length() - 2);
UPDATE_SUB_STATUS = UPDATE_SUB_STATUS + subQuery + ")";
paramSource.addValue("status", status);
jdbcTempalte.update(UPDATE_SUB_STATUS, paramSource);
}
What this do is create a query with all the ids to hextoraw as id0, id1, id2...... and also added this values in the MapSqlParameterSource instance and then this worked fine and it also used the index for updating my table.
After running my new function the query look like : update
SUBSCRIPTIONS set status = :status, modified_date = systimestamp
where id in (hextoraw(:id0), hextoraw(:id1), hextoraw(:id2)...)
MapSqlParameterSource instance looks like : {("id0", "randomUUID"),
("id1", "randomUUID"), ("id2", "randomUUID").....}
Instead of doing string manipulation, Convert the list to List of ByteArray
List<byte[]> productGuidByteList = stringList.stream().map(item -> GuidHelper.asBytes(item)).collect(Collectors.toList());
parameters.addValue("productGuidSearch", productGuidByteList);
public static byte[] asBytes(UUID uuid) {
ByteBuffer bb = ByteBuffer.wrap(new byte[16]);
bb.putLong(uuid.getMostSignificantBits());
bb.putLong(uuid.getLeastSignificantBits());
return bb.array();
}
Related
I'm fairly new to Oracle but I have used the Bulk insert on a couple other applications. Most seem to go faster using it but I've had a couple where it slows down the application. This is my second one where it slowed it down significantly so I'm wondering if I have something setup incorrectly or maybe I need to set it up differently. In this case I have a console application that processed ~1,900 records. Inserting them individually it will take ~2.5 hours and when I switched over to the Bulk insert it jumped to 5 hours.
The article I based this off of is http://www.oracle.com/technetwork/issue-archive/2009/09-sep/o59odpnet-085168.html
Here is what I'm doing, I'm retrieving some records from the DB, do calculations, and then write the results out to a text file. After the calculations are done I have to write those results back to a different table in the DB so we can look back at what those calculations later on if needed.
When I make the calculation I add the results to a List. Once I'm done writing out the file I look at that List and if there are any records I do the bulk insert.
With the bulk insert I have a setting in the App.config to set the number of records I want to insert. In this case I'm using 250 records. I assumed it would be better to limit my in memory arrays to say 250 records versus the 1,900. I loop through that list to the count in the App.config and create an array for each column. Those arrays are then passed as parameters to Oracle.
App.config
<add key="UpdateBatchCount" value="250" />
Class
class EligibleHours
{
public string EmployeeID { get; set; }
public decimal Hours { get; set; }
public string HoursSource { get; set; }
}
Data Manager
public static void SaveEligibleHours(List<EligibleHours> listHours)
{
//set the number of records to update batch on from config file Subtract one because of 0 based index
int batchCount = int.Parse(ConfigurationManager.AppSettings["UpdateBatchCount"]);
//create the arrays to add values to
string[] arrEmployeeId = new string[batchCount];
decimal[] arrHours = new decimal[batchCount];
string[] arrHoursSource = new string[batchCount];
int i = 0;
foreach (var item in listHours)
{
//Create an array of employee numbers that will be used for a batch update.
//update after every X amount of records, update. Add 1 to i to compensated for 0 based indexing.
if (i + 1 <= batchCount)
{
arrEmployeeId[i] = item.EmployeeID;
arrHours[i] = item.Hours;
arrHoursSource[i] = item.HoursSource;
i++;
}
else
{
UpdateDbWithEligibleHours(arrEmployeeId, arrHours, arrHoursSource);
//reset counter and array
i = 0;
arrEmployeeId = new string[batchCount];
arrHours = new decimal[batchCount];
arrHoursSource = new string[batchCount];
}
}
//process last array
if (arrEmployeeId.Length > 0)
{
UpdateDbWithEligibleHours(arrEmployeeId, arrHours, arrHoursSource);
}
}
private static void UpdateDbWithEligibleHours(string[] arrEmployeeId, decimal[] arrHours, string[] arrHoursSource)
{
StringBuilder sbQuery = new StringBuilder();
sbQuery.Append("insert into ELIGIBLE_HOURS ");
sbQuery.Append("(EMP_ID, HOURS_SOURCE, TOT_ELIG_HRS, REPORT_DATE) ");
sbQuery.Append("values ");
sbQuery.Append("(:1, :2, :3, SYSDATE) ");
string connectionString = ConfigurationManager.ConnectionStrings["Server_Connection"].ToString();
using (OracleConnection dbConn = new OracleConnection(connectionString))
{
dbConn.Open();
//create Oracle parameters and pass arrays of data
OracleParameter p_employee_id = new OracleParameter();
p_employee_id.OracleDbType = OracleDbType.Char;
p_employee_id.Value = arrEmployeeId;
OracleParameter p_hoursSource = new OracleParameter();
p_hoursSource.OracleDbType = OracleDbType.Char;
p_hoursSource.Value = arrHoursSource;
OracleParameter p_hours = new OracleParameter();
p_hours.OracleDbType = OracleDbType.Decimal;
p_hours.Value = arrHours;
OracleCommand objCmd = dbConn.CreateCommand();
objCmd.CommandText = sbQuery.ToString();
objCmd.ArrayBindCount = arrEmployeeId.Length;
objCmd.Parameters.Add(p_employee_id);
objCmd.Parameters.Add(p_hoursSource);
objCmd.Parameters.Add(p_hours);
objCmd.ExecuteNonQuery();
}
}
I'm using spring framework and oracle DB for the Web solution system.
The problem is when I call the web page related on oracle sequence,
sometimes more than two rows are inserted on the DB table.
That rows has not duplicated values but increased values from sequence.
Also I already checked the java code,
but I didn't use the loop or for sentences or call twice insert sentences.
Is that occurred often?
and how can I solve the problem?
Do I have to add the code for checking value or make the oracle trigger on the table?
This is the code.
public void insertDefaultLParameter(HttpServletRequest request, String workflowId) throws Exception{
String newLParaId = mapper.getNewLParaId();
HashMap<String, String> condition = new HashMap<String, String>();
condition.put("newLParaId", newLParaId);
condition.put("paraValue", "2013-05");
condition.put("workflowId", workflowId);
mapper.insertLParameter(condition);
mapper.insertLParameterMapping(condition);
}
"getNewLParaId()" called the sequence "MT_L_PARA_MAPPING_SEQ" like the below sql.
SELECT 'LPARA' || LPAD(MT_L_PARA_MAPPING_SEQ.nextval,10,0) FROM DUAL
After getting the value, the value is inserted into two tables through "insertLParameter" and "insertLParameterMapping" mapping id.
And the below is the code which call "insertDefaultLParameter" class.
See the bottom of the code.
You can find "rfmService.insertDefaultLParameter(request, praWfId);".
#RequestMapping(value="/insertWorkflowInfo", method=RequestMethod.POST, headers="Accept=application/json")
public ModelAndView insertWorkflowDetail(Locale locae, Model model, HttpServletRequest request) throws Exception{
HttpSession session = request.getSession(false);
User user = (User)session.getAttribute("user");
String userId = user.getUserId();
String workflowNm = "";
String alsPpsCd = "";
String workflowDesc = "";
String pid = "";
String plevel = "";
if(request.getParameter("workflowNm") != null || !request.getParameter("workflowNm").equals("")) workflowNm = request.getParameter("workflowNm");
if(request.getParameter("alsPpsCd") != null || !request.getParameter("alsPpsCd").equals("")) alsPpsCd = request.getParameter("alsPpsCd");
if(request.getParameter("workflowDesc") != null || !request.getParameter("workflowDesc").equals("")) workflowDesc = request.getParameter("workflowDesc");
if(request.getParameter("pid") != null || !request.getParameter("pid").equals("")) pid = request.getParameter("pid");
if(request.getParameter("plevel") != null || !request.getParameter("plevel").equals("")) plevel = request.getParameter("plevel");
//0.
//HashMap<String,String> treeMgtInfo = new HashMap<String,String>();
//treeMgtInfo.put("treeId",pid);
//
HashMap<String,String> input = new HashMap<String,String>();
//2.
String praWfId = wfService.selectPraWfid();
input.put("praWfId", praWfId);
input.put("wfNm",workflowNm);
input.put("wfDesc", SecurityUtil.removeXSS(workflowDesc));
input.put("alsPpsCd",alsPpsCd);
input.put("usrId",userId);
input.put("rgUsrId",userId);
// 1.
wfService.insertWorkflowDetail(input);
input.put("treeNm",workflowNm);
//
int treeLevCd = new Integer(plevel) +1 ;
input.put("treeLevCd",""+treeLevCd);
input.put("treeBjCd",Constant.TREE_BJ_CD_WORKFLOW);
input.put("treeCd",Constant.TREE_CD_WORKFLOW);
//
input.put("treeLrkRufId",praWfId);
input.put("upTreeId",pid);
//
int sceXrsSeqVl = treeService.selectSceXrsSeqId(input);
input.put("sceXrsSeqVl",""+sceXrsSeqVl);
String treeId = treeService.selectTreeIdInfo();
input.put("treeId",treeId);
// 2.
rfmService.insertDefaultLParameter(request, praWfId);
// 3.
treeService.insertTreeMgtInfoWithId(input);
ModelAndView modelAndView=new ModelAndView("defaultViews");
modelAndView.addObject("treeId",treeId);
modelAndView.addObject("praWfId",praWfId);
return modelAndView;
}
This is all of code related on the sequence.
I´m trying to use dapper with Oracle (ODP.NET) and I would like to use the "QueryMultiple" functionality.
Passing this string to the QueryMultiple method:
var query = "Select CUST_ID CustId from Customer_info WHERE CUST_ID=:custId;" +
"Select CUST_ID CustId from BCR WHERE CUST_ID=:custId";
I´m getting a ORA-00911: invalid character error
Is there any way to do this or it´s not possible?
Tks
The OP has probably long since solved the issue by now, but as of the time of writing, this question has only one answer and it doesn't really solve the problem of using Dapper's QueryMultiple() method with Oracle. As #Kamolas81 correctly states, by using the syntax from the official examples, one will indeed get the ORA-00933: SQL command not properly ended error message. I spent a while searching for some sort of documentation about how to do QueryMultiple() with Oracle, but I was surprised that there wasn't really one place that had an answer. I would have thought this to be a fairly common task. I thought that I'd post an answer here to save me :) someone some time in the future just in case anybody happens to have this same problem.
Dapper seems to just pass the SQL command straight along to ADO.NET and whatever db provider is executing the command. In the syntax from the examples, where each command is separated by a line break, SQL server will interpret that as multiple queries to run against the database and it will run each of the queries and return the results into separate outputs. I'm not an ADO.NET expert, so I might be messing up the terminology, but the end effect is that Dapper gets the multiple query outputs and then works its magic.
Oracle, though, doesn't recognize the multiple queries; it thinks that the SQL command is malformed and returns the ORA-00933 message. The solution is to use cursors and return the output in a DynamicParameters collection. For example, whereas the SQL Server version would look like this:
var sql =
#"
select * from Customers where CustomerId = #id
select * from Orders where CustomerId = #id
select * from Returns where CustomerId = #id";
the Oracle version of the query would need to look like this:
var sql = "BEGIN OPEN :rslt1 FOR SELECT * FROM customers WHERE customerid = :id; " +
"OPEN :rslt2 FOR SELECT * FROM orders WHERE customerid = :id; " +
"OPEN :rslt3 FOR SELECT * FROM returns Where customerid = :id; " +
"END;";
For queries run against SQL Server, Dapper can handle it from there. However, because we're returning the result sets into cursor parameters, we'll need to use an IDynamicParameters collection to specify parameters for the command. To add an extra wrinkle, the normal DynamicParameters.Add() method in Dapper uses a System.Data.DbType for the optional dbType parameter, but the cursor parameters for the query need to be of type Oracle.ManagedDataAccess.Client.OracleDbType.RefCursor. To solve this, I used the solution which #Daniel Smith proposed in this answer and created a custom implementation of the IDynamicParameters interface:
using Dapper;
using Oracle.ManagedDataAccess.Client;
using System.Data;
public class OracleDynamicParameters : SqlMapper.IDynamicParameters
{
private readonly DynamicParameters dynamicParameters = new DynamicParameters();
private readonly List<OracleParameter> oracleParameters = new List<OracleParameter>();
public void Add(string name, OracleDbType oracleDbType, ParameterDirection direction, object value = null, int? size = null)
{
OracleParameter oracleParameter;
if (size.HasValue)
{
oracleParameter = new OracleParameter(name, oracleDbType, size.Value, value, direction);
}
else
{
oracleParameter = new OracleParameter(name, oracleDbType, value, direction);
}
oracleParameters.Add(oracleParameter);
}
public void Add(string name, OracleDbType oracleDbType, ParameterDirection direction)
{
var oracleParameter = new OracleParameter(name, oracleDbType, direction);
oracleParameters.Add(oracleParameter);
}
public void AddParameters(IDbCommand command, SqlMapper.Identity identity)
{
((SqlMapper.IDynamicParameters)dynamicParameters).AddParameters(command, identity);
var oracleCommand = command as OracleCommand;
if (oracleCommand != null)
{
oracleCommand.Parameters.AddRange(oracleParameters.ToArray());
}
}
}
So all of the code together goes something like this:
using Dapper;
using Oracle.ManagedDataAccess.Client;
using System.Data;
int selectedId = 1;
var sql = "BEGIN OPEN :rslt1 FOR SELECT * FROM customers WHERE customerid = :id; " +
"OPEN :rslt2 FOR SELECT * FROM orders WHERE customerid = :id; " +
"OPEN :rslt3 FOR SELECT * FROM returns Where customerid = :id; " +
"END;";
OracleDynamicParameters dynParams = new OracleDynamicParameters();
dynParams.Add(":rslt1", OracleDbType.RefCursor, ParameterDirection.Output);
dynParams.Add(":rslt2", OracleDbType.RefCursor, ParameterDirection.Output);
dynParams.Add(":rslt3", OracleDbType.RefCursor, ParameterDirection.Output);
dynParams.Add(":id", OracleDbType.Int32, ParameterDirection.Input, selectedId);
using (IDbConnection dbConn = new OracleConnection("<conn string here>"))
{
dbConn.Open();
var multi = dbConn.QueryMultiple(sql, param: dynParams);
var customer = multi.Read<Customer>().Single();
var orders = multi.Read<Order>().ToList();
var returns = multi.Read<Return>().ToList();
...
dbConn.Close();
}
Building on greyseal96's helpful answer, I created this implementation of IDynamicParameters:
public class OracleDynamicParameters : SqlMapper.IDynamicParameters
{
private readonly DynamicParameters dynamicParameters;
private readonly List<OracleParameter> oracleParameters = new List<OracleParameter>();
public OracleDynamicParameters(params string[] refCursorNames) {
dynamicParameters = new DynamicParameters();
AddRefCursorParameters(refCursorNames);
}
public OracleDynamicParameters(object template, params string[] refCursorNames) {
dynamicParameters = new DynamicParameters(template);
AddRefCursorParameters(refCursorNames);
}
private void AddRefCursorParameters(params string[] refCursorNames)
{
foreach (string refCursorName in refCursorNames)
{
var oracleParameter = new OracleParameter(refCursorName, OracleDbType.RefCursor, ParameterDirection.Output);
oracleParameters.Add(oracleParameter);
}
}
public void AddParameters(IDbCommand command, SqlMapper.Identity identity)
{
((SqlMapper.IDynamicParameters)dynamicParameters).AddParameters(command, identity);
var oracleCommand = command as OracleCommand;
if (oracleCommand != null)
{
oracleCommand.Parameters.AddRange(oracleParameters.ToArray());
}
}
}
Assuming the same query, it can be used so:
var queryParams = new { id };
string[] refCursorNames = { "rslt1", "rslt2", "rslt3" };
var dynParams = new OracleDynamicParameters(queryParams, refCursorNames);
...
var multi = dbConn.QueryMultiple(sql, param: dynParams);
I suspect this is two or three separate things:
Your first query should not have a semi-colon
There is no new-line character between the queries
The usage notes imply that the bind character is # not : (no idea if this depends on the RDBMS being used).
If you look at the Dapper Google Code page the example given for QueryMultiple() is:
var sql =
#"
select * from Customers where CustomerId = #id
select * from Orders where CustomerId = #id
select * from Returns where CustomerId = #id";
using (var multi = connection.QueryMultiple(sql, new {id=selectedId}))
{
var customer = multi.Read<Customer>().Single();
var orders = multi.Read<Order>().ToList();
var returns = multi.Read<Return>().ToList();
...
}
Remove the semi-colon; add a new line and if you still have issues change the bind character.
I have a very large list of strings containing GUIDs and a 100,000+ record table in a database with an Entity Framework model in my app.
What' is the most efficient approach to finding all records in a particular dataset where the GUID List is not present?
The following is performing very slowly:
var list= new List<string> { "1", "2", "3" };
return (from t1 in db.Items
where (!list.Contains(t1.GUID))
When I have a lot of parameters (several hundreds or more) to a query I use bulk insert to temporary table and then join it from main table.
My code looks something like this:
private static DataTable FillDataTable(IEnumerable<int> keys) {
var dataTable = new DataTable("Stage");
dataTable.Locale = CultureInfo.CurrentCulture;
dataTable.Columns.Add("Key", typeof(int));
foreach (var key in keys) {
var row = dataTable.NewRow();
row[0] = key;
dataTable.Rows.Add(row);
}
return dataTable;
}
private static void CreateStageTable(SqlConnection connection, string tableName, DataTable dataTable) {
var sql = new StringBuilder();
sql.AppendLine("CREATE TABLE {StageTableName} ( ");
sql.AppendLine(" Key INT NOT NULL ");
sql.AppendLine(") ");
sql.Replace("{StageTableName}", SqlUtilities.QuoteName(tableName));
using (var command = connection.CreateCommand()) {
command.CommandText = sql.ToString();
command.CommandType = CommandType.Text;
command.ExecuteNonQuery();
}
using (var bulkcopy = new SqlBulkCopy(connection)) {
bulkcopy.DestinationTableName = tableName;
bulkcopy.WriteToServer(dataTable);
}
}
public void DoQuery(IEnumerable<int> keys) {
var dataTable = FillDataTable(keys);
using (var connection = new SqlConnection(_connectionString)) {
connection.Open();
CreateStageTable(connection, "#Stage", dataTable);
string sql = "SELECT x " +
"FROM tbl " +
" LEFT JOIN {StageTableName} AS Stage " +
" ON x.Key = Stage.Key "
"WHERE Stage.Key IS NULL";
...
}
}
Don't use a List, use a HashSet<string>, this will give you O(1) lookup ups instead of O(n).
I use Spring Jdbc Template that way:
public List<User> getUsersForGrid(int rows, int page, String sidx,
String sord) {
int fromRecord = 0;
int toRecord = 0;
toRecord = page * rows;
fromRecord = (page - 1) * rows;
StringBuilder sqlB = new StringBuilder();
sqlB.append("SELECT user_id, username ");
sqlB.append("FROM users ");
sqlB.append("WHERE :fromRecord <= rownum AND rownum <= :toRecord ");
sqlB.append("ORDER BY %s %s ");
String sql = String.format(sqlB.toString(), sidx, sord);
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("fromRecord", fromRecord);
params.addValue("toRecord", toRecord);
List<Map<String, Object>> rsRows = this.jdbcTemplate.queryForList(sql
.toString(),params);
List<User> users = new ArrayList<User>();
for (Map<String, Object> row : rsRows) {
BigDecimal id = (BigDecimal) row.get("user_id");
String username = (String) row.get("username");
User user = new User(id.intValue(), username);
users.add(user);
}
return users;
}
and get java.sql.SQLException: Invalid column type
sidx is column nate("user_id" for example) sord is asc/desc
When pass no params(execute only
sql.append("SELECT user_id, username ");
sql.append("FROM users ");
) everything is OK.
Update: Works with:
sqlB.append("WHERE ? <= rownum AND rownum <= ? ");
and
this.jdbcTemplate.queryForList(sql.toString(),new Object[]{fromRecord, toRecord});
Seems like problem with Spring MapSqlParameterSource and named parameters. I use Spring 3.1.3
DB is Oracle 11.2
describe users;
Name Null Type
------------------------------ -------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
USER_ID NOT NULL NUMBER
USERNAME NOT NULL VARCHAR2(40)
PASSWORD NOT NULL VARCHAR2(20)
ENABLED NOT NULL NUMBER
I think the problem is with your order by clause,
you are trying to dynamically change your order by clause.
Just try
StringBuilder sql = new StringBuilder();
sql.append("SELECT user_id, username ");
sql.append("FROM users ");
sql.append("WHERE :fromRecord <= rownum AND rownum <= :toRecord ");
sql.append("ORDER BY user_id asc ");
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("fromRecord", fromRecord);
params.addValue("toRecord", toRecord);
If the above works, then instead of using the MapSqlParameterSource for changing the order by clause use something like
StringBuilder sql = new StringBuilder();
sql.append("SELECT user_id, username ");
sql.append("FROM users ");
sql.append("WHERE :fromRecord <= rownum AND rownum <= :toRecord ");
sql.append("ORDER BY %s %s ");
//Format the sql string accordingly
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("fromRecord", fromRecord, Types.INTEGER);
params.addValue("toRecord", toRecord, Types.INTEGER);
Hope it helps.
try this
List<Map> rows = getJdbcTemplate().queryForList(sql);
for (Map row : rows) {
BigDecimal id = (BigDecimal) row.get("user_id");
String username = (String) row.get("username");
User user = new User(id.intValue(), username);
users.add(user);
}
ok try
MapSqlParameterSource namedParameters = new MapSqlParameterSource();
namedParameters.addValue("fromRecord", fromRecord);
namedParameters.addValue("toRecord", toRecord);
namedParameters.addValue("sidx", sidx);
namedParameters.addValue("sord", sord);
return this.getNamedParameterJdbcTemplate().query(query,
namedParameters, new UserElementMapper());
public class UserMapper implements RowMapper<User> {
public EmailElement mapRow(ResultSet rs, int rowNum) throws SQLException {
User user = new User();
emailElement.setID(rs.getInt("user_id"));
emailElement.setUsernameo(rs.getString("username"));
return user;
}
}