I created jsp web apps to perform 4 millions data insert. The insertion process using loop from another table, so i did select query -> loop -> insert to other table, but everytime i run this, the page load slowly an ended with timeout. So not all data was successfully inserted
I had tried to use bul data insert, it didnt hep me.
btw this is the sample code :
pstatement = connection.prepareStatement(insertquery);
pstatement.setString(1, request.getParameter("promo"));
while (rset.next()) {
pstatement.setString(1, rset.getString(1));
pstatement.setString(2, request.getParameter("promo"));
pstatement.addBatch();
out.print(rset.getString(1) + " Added<br>");
if (++countbatch % batchSize == 0) {
pstatement.executeBatch();
}
}
pstatement.executeBatch();
Another try using select trick, it also didnt help
String insertquery = "INSERT INTO datapin (msisdn,nama_promo) SELECT msisdnlist.msisdn AS msisdn, ? AS nama_promo FROM msisdnlist ";
pstatement = connection.prepareStatement(insertquery);
pstatement.setString(1, request.getParameter("promo"));
pstatement.executeQuery();
Anyone have better idea ?? Thanks :)
Is it required that the jsp page waits for the 4M inserts to complete?
I think waiting for a long lasting operation is always tricky in a web app.
You might start a thread in the request handler that will do the inserts async.
The request handler could return a token.
Then this token could be used by a second JSP page to periodically check the status of the thread.
Related
This question already has answers here:
Unable to edit db entries using EFCore, EntityState.Modified: "Database operation expected to affect 1 row(s) but actually affected 0 row(s)."
(17 answers)
Closed 2 years ago.
I am using Sql Server on Linux with EC Core (2.1.0-preview1-final)
I am trying to update some data coming from a web Api service (PUT) request. The data (ticket) is passed correctly and gets deserialised into an object (ticket).
When I try to update, I use the following code:
public Ticket UpdateTicket(Ticket ticket)
{
using (var ctx = new SupportTicketContext(_connectionString))
{
ctx.Entry(ticket).State = EntityState.Modified;
ctx.SaveChanges(); // <== **BLOWS UP HERE**
var result = ctx.Tickets
.First(t => t.TicketId == ticket.TicketId);
return result;
}
}
The code throws the following error:
Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data
may have been modified or deleted since entities were loaded
I am able to Insert and fetch from the database, no problem.
If I restart Visual Studio, the error occurs usually on the second time I try to update ANY ticket (i.e. any other ticketId - it seems to be on the second and subsequent requests).
The updates are unpredictably successful! Sometimes I can update another ticket and it goes through (even on the 3rd or susequent request)
I have tried a number of modifications to the code, including
ctx.Attach(ticket);
but this does not help.
How do I get it to update the database successfully?
Any ideas on how to debug this? Logging seems to be difficult to set up.
Any ideas greatly appreciated.
There were two errors. First, I tried to call the Update() method without specifying a DbSet:
_applicationDbContext.Update(user);
Correct way:
_applicationDbContext.Users.Update(user);
The second error was trying to update a model that without first pulling it from the database:
public bool UpdateUser(IdentityUser model)
{
_applicationDbContext.Users.Update(model);
_applicationDbContext.SaveChanges();
return true;
}
Nope.
I first needed to retrieve it from the database, then update it:
public bool UpdateUser(IdentityUser model)
{
var user = _applicationDbContext.Users.FirstOrDefault(u => u.Id == model.Id);
user.PhoneNumberConfirmed = model.PhoneNumberConfirmed;
_applicationDbContext.Users.Update(user);
_applicationDbContext.SaveChanges();
return true;
}
Had such problem too, but the reason was a trigger I was using.
Changed the trigger type from "INSTEAD OF INSERT" to "AFTER INSERT" and instead of modifying and inserting the data I was allowing my command to insert the data to the table and then updated it using the trigger.
I have a background job that fetches data from google adwords. Now my issue is when I fetch the data using a background worker.
When the response its empty what it the best thing to do is there any way to re run again or what is the best approach ?
public function handle()
{
$googleService = new GoogleAds;
$data = $googleService->report()
->from('CRITERIA_PERFORMANCE_REPORT')
->during('20170101', '20170210')
->select('CampaignId, Id, Criteria, IsNegative, Clicks, Ctr, Cost, Labels')
->getObject();
if(!isset($data->result) || empty($data->result)){
//what to do when no data back ?
}
$this->transform->response($data);
}
You can throw an exception, then it will go back to your queue, and the worker will try to execute it again.
When you launch your worker, there is a --tries parameter that indicates how many time it will try to execute before it goes to the table failed_jobs.
You can check the reference in the official documentation.
i'm just starting out with Dexie, and I seem to be coming unstuck.
I have a small database (less than 1000 rows), and i'm trying to delete each row one-by-one once I know that the row has been sent to a remote API.
I can also successfully save to the table (which is defined by an ID and a column storing a serialised object)
here's my code:
if (online) {
//we query the db and send each event
database.open()
let allEvents = database.events.toCollection()
let total = allEvents.count(function (count) {
console.log(count + ' events in total')
//a simple test to ensure we're seeing the right number of records
})
allEvents.each(function(thisEvent){
//push to remote API
console.log('deleting ' + thisEvent.id)
database.events.delete(thisEvent.id) //<= this doesn't seem to be working
})
}
All of this with the exception of the final delete statement.
Any ideas on how I should fix this? the important thing for me is to delete on a per-row basis.
thanks in advance!
I was experiencing the same problem, and the answer from Eugenia Pais wasn't working for me. So after some tests, I saw the trouble was with the type of the variable: I was using a string, but a number is needed, so this is how I solved it:
function removeRow (primaryKey) {
primaryKey = parseInt(primaryKey);
databaseName.tableName.where('primaryKey').equals(primaryKey).delete().then(function(deleteCount) {
console.log ("Deleted " + deleteCount + " rows");
}).catch(function(error) {
console.error ("Error: " + error);
});
So be aware you are using a number as argument.
The correct way to delete each row should be selecting the specific row and delete it:
database.tableName.where(indexId).equals(indexValue).delete();
The data type of the key is not a problem you could verify it in my example here: example
db.example.where('key').equals('one').delete();
Maybe you are trying to delete by a property that not is an index.
I'm using EF 5 with Oracle database.
I'm doing a select count in a table with a specific parameter. When I'm using EF, the query returns the value 31, as expected, But the result takes about 10 seconds to be returned.
using (var serv = new Aperam.SIP.PXP.Negocio.Modelos.SIP_PA())
{
var teste = (from ens in serv.PA_ENSAIOS_UM
where ens.COD_IDENT_UNMET == "FBLDY3840"
select ens).Count();
}
If I execute the simple query bellow the result is the same (31), but the result is showed in 500 milisecond.
SELECT
count(*)
FROM
PA_ENSAIOS_UM
WHERE
COD_IDENT_UNMET 'FBLDY3840'
There are a way to improve the performance when I'm using EF?
Note: There are 13.000.000 lines in this table.
Here are some things you can try:
Capture the query that is being generated and see if it is the same as the one you are using. Details can be found here, but essentially, you will instantiate your DbContext (let's call it "_context") and then set the Database.Log property to be the logging method. It's fine if this method doesn't actually do anything--you can just set a breakpoint in there and see what's going on.
So, as an example: define a logging function (I have a static class called "Logging" which uses nLog to write to files)
public static void LogQuery(string queryData)
{
if (string.IsNullOrWhiteSpace(queryData))
return;
var message = string.Format("{0}{1}",
queryData.Trim().Contains(Environment.NewLine) ?
Environment.NewLine : "", queryData);
_sqlLogger.Info(message);
_genLogger.Trace($"EntityFW query (len {message.Length} chars)");
}
Then when you create your context point to LogQuery:
_context.Database.Log = Logging.LogQuery;
When you do your tests, remember that often the first run is the slowest because the server has to actually do the work, but on the subsequent runs, it often uses cached data. Try running your tests 2-3 times back to back and see if they don't start to run in the same time.
I don't know if it generates the same query or not, but try this other form (which should be functionally equivalent, but may provide better time)
var teste = serv.PA_ENSAIOS_UM.Count(ens=>ens.COD_IDENT_UNMET == "FBLDY3840");
I'm wondering if the version you have pulls data from the DB and THEN counts it. If so, this other syntax may leave all the work to be done at the server, where it belongs. Not sure, though, esp. since I haven't ever used EF with Oracle and I don't know if it behaves the same as SQL or not.
I have noticed that crystal report runs the Linq query once again when the page index is changed, means when we load second page from first page?
So just wanted to know if we can get which page is loaded so that we can keep values in session.
Just a hint is required as I am not getting the desired results from Google.
Update:
I am sorry in a hurry I just clicked on a wrong tag.
So the problem is like:
This is my code below which I use fr running my crystal report:
var rpt = new Result();
List<class> lst1 = new DALMethod().Get();
rpt.SetDataSource(lst1);
CRReportViewer.ReportSource = rpt;
When I switch from page one to two or more, this method in DAL is called again taking the same time it took first time to load, so I just want to have the data in session when query runs first time, and next time when I get the page index, then I will show data from session.
Is there a way around by which I can get the page index in this c# code?
I had found the solution, hope this might help someone else:
I was using a generic list as a data source:
As soon as we get to know the page loads for the first time, I mean not a postback, we can initialize a list to be maintained in session.
After showing the report we can add the data source (which is a list type).
On Report page shift data will be taken from session.
if (!IsPostBack)
{
//clear session and create new session
Session["ReportGenericList"] = null;
}
List<class> datasourceLst=null;
if (Session["ReportGenericList"] != null)
{
datasourceLst= (List<class>)Session["ReportGenericList"];
}
else
{
datasourceLst = //call methods to fill datasource
Session["ReportGenericList"] = datasourceLst;
}