I just want to load some files into a Azure DB.
I am using the "Microsoft SQL Server" DB Type for the connection.
The problem is that when I insert like more than 10.000 rows, I have sometimes (90% of the time) an error:
Exception in component tMSSqlOutput_5
java.sql.BatchUpdateException: I/O Error: Connection reset
at net.sourceforge.jtds.jdbc.JtdsStatement.executeBatch(JtdsStatement.java:1091)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tFileInputDelimited_5Process(extractGC_child2.java:28852)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tFileList_6Process(extractGC_child2.java:32386)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tFileList_5Process(extractGC_child2.java:31540)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tMSSqlRow_1Process(extractGC_child2.java:30657)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tLoop_2Process(extractGC_child2.java:30440)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tFileList_4Process(extractGC_child2.java:29664)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tJava_3Process(extractGC_child2.java:34020)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tMSSqlInput_1Process(extractGC_child2.java:33593)
at dev_storch.extractgc_child2_0_1.extractGC_child2.tFTPConnection_2Process(extractGC_child2.java:33154)
[FATAL]: dev_storch.extractgc_child2_0_1.extractGC_child2 - tMSSqlOutput_5 I/O Error: Connection reset
[FATAL]: dev_storch.extractgc_child2_0_1.extractGC_child2 - tMSSqlRow_7 Invalid state, the Connection object is closed.
But when the volume of data inserted is lower, I don't receive any error.
My configuration looks like this:
tMSSQLConnection. Then I have some components to load files from a folder and load it inside a table.
The error comes at the tMSSQLOutput.
The following of the job are logs filling.
I tried to change the Batch size, to not use a DBConnection, but doesn't work.
I tried with a Generic JDBC component and it seems to work everytime. But I don't want to use the generic JDBC components because on the ouptut components, we can not choose the colume DB type (but maybe someone know how is it possible):
MSSQL:
Generic JDBC:
Thank you in advance...
here one solution maybe your :
Be aware that the batch size must be lower than or equal to the limit of parameter markers authorized by the JDBC driver (generally 2000) divided by the number of columns.
Related
I have a spring boot application web-socket server up and running as a WEBRTC signaling server.
This server has to log some data to the database, based on the messages rotated from/to the server, via different client sockets.
While trying a conference call from client side using more than 2 peers, and running the signaling as debugger with breakpoints, the scenario was successfully completed and the database was updated as expected and the conference call took place.
But, if I run the server without debug and breakpoints, I get an sql error
Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: HikariProxyPreparedStatement#1623454983 wrapping com.mysql.cj.jdbc.ClientPreparedStatement:
delete from my_table where my_table_id='601cbe2b-6af2-4e82-a69c-52b92d30686c'; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: HikariProxyPreparedStatement#1623454983 wrapping com.mysql.cj.jdbc.ClientPreparedStatement:
delete from my_table where my_table_id='601cbe2b-6af2-4e82-a69c-52b92d30686c'
I am using Spring JPA and calling the save function on every message received by the web-socket after populating the entity data and all its nested lists and relational entity object, in order to keep data of the call flow.
conferenceRepository.save(conference);
I think the error is because of queries running concurrently on the database, in random order, while the data is still not there in the database to act upon.
As in debug mode, I am taking time to move from one breakpoint to another, and this is assuring the data persistence.
But I am not totally sure of the problem.
Is there an optimal way to apply concurrent database calls and updates and make sure the data is preserved and persisted properly in the database for concurrent web-socket related messages?
I am using the Spring Data JDBC (version 1.1.6) method CrudRepository.findAllById to load entities from a database with a large number of IDs. The underlying database connection uses a Postgres database. The invocation of the method raises a PSQLException:
2020-05-28 05:58:35,260 WARN com.zaxxer.hikari.pool.ProxyConnection [task-2] HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#1224f39f marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:358)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:159)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:109)
...
Caused by: java.io.IOException: Tried to send an out-of-range integer as a 2-byte value: 137525
at org.postgresql.core.PGStream.sendInteger2(PGStream.java:275)
at org.postgresql.core.v3.QueryExecutorImpl.sendParse(QueryExecutorImpl.java:1553)
at org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1876)
at org.postgresql.core.v3.QueryExecutorImpl.sendQuery(QueryExecutorImpl.java:1439)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
The exception seem to be ultimately caused by a 16-bit limit on the number of values in the SELECT IN (?) clause, that is generated by Sprint Data JDBC upon using findAllById.
Am I supposed to partition the list of IDs myself? Shouldn't the CrudRepository.findAllById handle this correctly in a fashion compatible with the underlying database dialect?
Am I supposed to partition the list of IDs myself?
Yes, assuming this kind of query makes sense in the first place.
Spring Data JDBC currently creates a straight forward select ... where id in (..) query which in turn is limited by the capabilities of the underlying database/JDBC driver.
With the apparent limit being ~216 for Postgres there doesn't seem to be much need for special handling in Spring Data JDBC since looking for so many ids in a single select seems rare enough to justify some manual coding.
I am trying to update from 4.0 to 4.5.1 but the process always fails at UpdateMeasuresDebtToMinutes. I am using MySQL 5.5.27 as a database with InnoDB as table engine.
Basically the problem looks like this problem
After the writeTimeout exceeds (600 seconds) there is an exception in the log
Caused by: java.io.EOFException: Can not read response from server. Expected to read 81 bytes, read 15 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3166) ~[mysql-connector-java-5.1.27.jar:na]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3676) ~[mysql-connector-java-5.1.27.jar:na]
Adding the indexes as proposed in the linked issue did not help.
Investigating further I noticed several things:
the migration step reads data from a table and wants to write back to the same table (project_measures)
project_measures contains more than 770000 rows
the process always hangs after 249 rows
the hanging happens in org.sonar.server.migrations.MassUpdate when calling update.addBatch() which after the BatchSession.MAX_BATCH_SIZE (250) forces an execute and a commit
is there a way to configure the DB connection to allow this to proceed?
First of all, could you try to revert your db to 4.0 and try again ?
Then, could you please give us the JDBC url (sonar.jdbc.url) you're using ?
Thanks
As I need that sonar server to run I finally implemented a workaround.
It seems I cannot write to the database at all, as long as a big result set is still open (I tried with a second table but the same issue as before).
Therefore I changed all migrations that need to read and write the project_measurestable (org.sonar.server.db.migrations.v43.TechnicalDebtMeasuresMigration, org.sonar.server.db.migrations.v43.RequirementMeasuresMigration, org.sonar.server.db.migrations.v44.MeasureDataMigration) to load the changed data into a memory structure and after closing the read resultset write it back.
This is as hacky as it sounds and will not work for larger datasets where you would need to this with paging through the data or storing everything into a secondary datastore.
Furthermore I found that later on (in 546_inverse_rule_key_index.rb) an index needs to be created on the rules table which is larger than the max key length on mysql (2 varchar(255) columns with UTF-8 is more than 1000bytes .. ) so I had to limit the key length on that too ..
As I said, it is a workaround and therefore I will not accept it as an answer ..
I have a JEE application searching a large Oracle databse for data. The application uses JDBC to query the database.
The issue I am having is that the results page is unable to be displayed. I get the following error:
The connection to the server was reset while the page was loading.
This happens after 60 seconds. When I run the sql query manually using a SQL client, the results return in 3 seconds.
I have checked the logs and there are no exceptions that I can see.
Do any of you know the best way to find what is causing the connection to be reset? If I break my search date range into 2, and search both ranges individually, both return results. So it seems that it's the larger result set causing the issue.
Any help is welcome.
You are probably right about the larger result set. Often when running a query from a SQL client, you'll get the first set of records right away. If you page down to force pull of all records, then it bogs down. Perhaps your hitting the same issue with JDBC client where it takes more than 60 sec to get all the rows. I've not done JDBC in a while, but can you get it to stream the result set?
Regards,
Roger
All views are mine ...
How to prevent the connection to the oracle server being gets lost if it is kept for some ideal time
If you use the newest JDBC spec 4.0 there is a isValid() method available for a connection that allows you to check if the connection is usable, if not then get a new (reconnect) connection and execute your SQL.
One possible way that I knew to save the database connection from being getting lost is to send a dummy query after the threshhold time, By Threash hold I mean the time after which the connection to the database is expected to become idle or get lost.
Some thing like
Ping_time_to_DB=60
if(Current_time - Last_Ping_time > Ping_time_to_DB)
{
--send a dummy query like
select 1 from dual;
}