Recently, a JHipster 7.7.0 application started generating server side/database errors for ALL tables. The application was deployed in production, and users entered data without problem. Suppose there is an entity called 'Product', and users have added several entries:
| ID | Name |
|------|-------------|
| 1001 | Corn flakes |
| 1002 | Baked beans |
| 1003 | Carrots |
| 1004 | Apples |
| 1005 | Watermelon |
First thing I notice is the ID values start counting up from 1001. (This is new... in JHipster 6 ID values start at 1.)
After I made some improvements to the application code, I redeployed into production. A user attempts to add a new product, and the following error is generated in the server logs:
2022-03-24 12:54:43.775 ERROR 11277 --- [ XNIO-1 task-1] o.h.e.jdbc.batch.internal.BatchingBatch : HHH000315: Exception executing batch [java.sql.BatchUpdateException: (conn=33) Duplicate entry '1001' for key 'PRIMARY'], SQL: insert into product (name, id) values (?, ?)
2022-03-24 12:54:43.776 WARN 11277 --- [ XNIO-1 task-1] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 1062, SQLState: 23000
2022-03-24 12:54:43.776 ERROR 11277 --- [ XNIO-1 task-1] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=33) Duplicate entry '1001' for key 'PRIMARY'
2022-03-24 12:54:43.779 ERROR 11277 --- [ XNIO-1 task-1] o.z.problem.spring.common.AdviceTraits : Internal Server Error
org.springframework.dao.DataIntegrityViolationException: could not execute batch; SQL [insert into product (name, id) values (?, ?)]; constraint [PRIMARY]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:276)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:566)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at com.mycompany.app.web.rest.ProductResource$$EnhancerBySpringCGLIB$$84c14d6d.createProduct(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
Clearly, the ID counter restarted at 1001, found an entry (Corn flakes), and the database produced the error.
When the user attempts to save again, an almost identical error occurs but this time the ID value has incremented to 1002, which now conflicts with Baked beans, and the database throws another 'Duplicate entry' exception.
Repeated saves fail until the user gets past 1005 (Watermelon), and the save succeeds with 1006. (Subsequent saves into this table continue without error.)
Similar errors occur on all entity tables, and can only be resolved by repeatedly 'saving' until the ID has incremented past the last ID value already in that table.
This is a problem.
Investigation
I've been generating JHipster applications since version 6, and generally without issue.
I looked at an old JHipster 6 project and compared entities with those in new JHipster 7 projects, and noticed that there has been a shift in strategy to how an ID value is generated.
In JHipster 6 (and older, I presume) ID values are defined for each entity as follows:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
Coupled with Liquibase changelog file entries for each entity (e.g. ..._added_entity_Product.xml):
<column name="id" type="bigint" autoIncrement="${autoIncrement}">
<constraints primaryKey="true" nullable="false"/>
</column>
And the liquibase 00000000000000_initial_schema.xml file, contains (near the top):
<property name="autoIncrement" value="true"/>
In this case, the database tables created define the ID field as an AUTO_INCREMENT field, meaning the ID values and increments are managed by the database:
CREATE TABLE `product` {
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL
} ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
However, in JHipster 7, things have changed. For each entity, the ID property is defined as:
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "sequenceGenerator")
#SequenceGenerator(name = "sequenceGenerator")
#Column(name = "id")
private Long id;
Coupled with Liquibase changelog file entries (e.g. ..._added_entity_Product.xml):
<column name="id" type="bigint">
<constraints primaryKey="true" nullable="false"/>
</column>
And the liquibase 00000000000000_initial_schema.xml file, contains (near the top):
<changeSet id="00000000000000" author="jhipster">
<createSequence sequenceName="sequence_generator" startValue="1050" incrementBy="50"/>
</changeSet>
In this configuration, a sequence generator appears to be managing ID values and increments. Database tables no longer define the ID field as an AUTO_INCREMENT:
CREATE TABLE `product` {
`id` bigint(20) NOT NULL,
`name` varchar(255) DEFAULT NULL
} ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
Temporary Fix
To get things working, I performed the following changes to my JHipster 7 project:
Modified each domain POJO entity's ID property annotation:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
Modified each entity's liquibase changelog XML file (..._added_entity_Product, etc):
<column name="id" type="bigint" autoIncrement="${autoIncrement}">
<constraints primaryKey="true" nullable="false"/>
</column>
Modified the liquibase 00000000000000_initial_schema.xml file, by adding the following line near the top:
<property name="autoIncrement" value="true"/>
Now, my database is generated such that the each entity table ID values are defined as:
`id` bigint(20) NOT NULL AUTO_INCREMENT
This appears to be working for me, but I am not happy about this.
Questions
Ideally, I'd like to NOT change the JHipster 7 original configuration, and better understand how this ID sequence generation is managed. (I'm not sure tinkering with changelog files is the right way of fixing this, and have more faith in the JHipster team's database expertise than my own.)
How I can take control of this sequence generator, and instruct it pick up from whichever value is appropriate for each entity, particularly after a redeploy/update? Is it possible?
Can I instruct it to find the next available value in the database automatically without throwing errors and failing?
Why have the JHipster team made this change? Is there an advantage? Can someone point me to a technical resource so I can learn about this advantage?
Can anyone point me in the right direction, so I can learn and improve my understanding?
NOTE: In case it is not clear, the project suffering from this database issue was JHipster 7 from inception. It was NOT originally in JHipster 6 and upgraded to 7. The only reason I mention JHipster 6 is because previous projects using JHipster 6 did not experience this problem, so I was investigating what changed between versions of JHipster, particularly regarding handling of ID values.
Environment: JHipster 7.7.0 (Angular, monolithic), MariaDB 10.4, OpenJDK 16.0.2_7, OS Windows 10 Pro and openSUSE 15.2, Firefox 98.0.2 and Chrome 99.0.4844.84.
Sequences are better for performance and can be pre-allocated per batch. See https://vladmihalcea.com/mariadb-10-3-database-sequences/
This strategy was already used for other databases, as MariaDB introduced sequences in 10.3, the JHipster team decided to apply same strategy over all databases and as it was a breaking change they made it on a major release.
Related
I have a scenario in which I am performing some Database(PostgreSQL) operations.
Some predefined entries are inserted in tables using the shell script in which I am giving assoc_id which is a NOT NULL constraint and primary key as well. For the same operation, the user can input values using UI for which assoc_id are auto-generated using hibernate sequence.
Now the plot is let's say pre-defined entries through shell script are given below as:
INSERT INTO config.profile_backend_point_asso(assoc_id,enabled,end_point_id,profile_id) VALUES
(1,true,1,1),
(2,true,2,1),
(3,true,3,1),
(4,true,4,1),
(5,true,5,1),
(6,true,6,1),
(7,true,7,1),
(8,true,8,1),
(9,true,9,1),
(10,true,10,1);
Now through UI user insert some entries and there may be a scenario where hibernate generates assoc_id as 11 just after the last insertion through the shell script.
Now let's say some more pre-defined data needs to be inserted through the shell script in such case we might be unaware of the last sequence generation of hibernate for the same table and new entries inserted through the script are as below:-
INSERT INTO config.profile_backend_point_asso(assoc_id,enabled,end_point_id,profile_id) VALUES
(11,true,11,1),
(12,true,12,1),
(13,false,13,1),
(14,true,14,1),
(15,true,15,1);
In such situation, the data through script will not get inserted as assoc_id 11 is already present which was inserted using UI.
Now how can I sync both the script and UI entries so that this situation may not occur?
NOTE: UI code is backed by Java code for auto-generation of assoc_id as below:
#Id
#SequenceGenerator(name = "hibernate_sequence", sequenceName =
"hibernate_sequence" , allocationSize=50)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator =
"hibernate_sequence")
#Column(name = "assoc_id")
private long assocId;
And if the operation is performed through UI then java code gives the below exception:
org.springframework.dao.DataIntegrityViolationException: A different
object with the same identifier value was already associated with the
session : [com.cavisson.model.ProfileBackendPointAssociation#50];
nested exception is javax.persistence.EntityExistsException: A
different object with the same identifier value was already associated
with the session :
[com.cavisson.model.ProfileBackendPointAssociation#50]
at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:410)
Also, hibernate_sequence is generating ids for all the tables in Database and this is just one such example.
Kindly help me with the workaround or ideas that I can implement to resolve this issue.
When using a Sequence, as commented, this is a DB issue. You should not provide an ID in insertion since it will be provided on DB. No assoc_id should be used on insert statement.
#Column(name="open")
Using sqlserver dialect with hibernate.
[SchemaUpdate] Unsuccessful: create table auth_session (id numeric(19,0) identity not null, active tinyint null, creation_date datetime not null, last_modified datetime not null, maxidle int null, maxlive int null, open tinyint null, sessionid varchar(255) not null, user_id numeric(19,0) not null, primary key (id), unique (sessionid))
[SchemaUpdate] Incorrect syntax near the keyword 'open'.
I would have expected hibernate to use quoted identifier when creating the table.
Any ideas on how to handle this... other than renaming the field?
With Hibernate as JPA 1.0 provider, you can escape a reserved keyword by enclosing it within backticks:
#Column(name="`open`")
This is the syntax inherited from Hiberate Core:
5.4. SQL quoted identifiers
You can force Hibernate to quote an
identifier in the generated SQL by
enclosing the table or column name in
backticks in the mapping document.
Hibernate will use the correct
quotation style for the SQL Dialect.
This is usually double quotes, but the
SQL Server uses brackets and MySQL
uses backticks.
<class name="LineItem" table="`Line Item`">
<id name="id" column="`Item Id`"/><generator class="assigned"/></id>
<property name="itemNumber" column="`Item #`"/>
...
</class>
In JPA 2.0, the syntax is standardized and becomes:
#Column(name="\"open\"")
References
Hibernate reference guide
5.4. SQL quoted identifiers
JPA 2.0 specification
2.13 Naming of Database Objects
Related questions
Hibernate, MySQL and table named “Repeat” - strange behaviour
Automatic reserved word escaping for Hibernate tables and columns
Had the same problem, but with a tablename called Transaction. If you set
hibernate.globally_quoted_identifiers=true
Then all database identifiers will be quoted.
Found my answer here
Special character in table name hibernate giving error
And found all available settings here
https://docs.jboss.org/hibernate/orm/5.2/userguide/html_single/appendices/Configurations.html
Could not find better docs for this though.
In my case the setting was in my Spring properties file. As mentioned in the comments, it could also be in other, hibernate related, configuration files.
Manually escaping the reserved keywords
If you are using JPA, you can escape with double quotes:
#Column(name = "\"open\"")
If you're using Hibernate native API, then you can escape them using backticks:
#Column(name = "`open`")
Automatically escaping reserved keywords
If you want to automatically escape reserved keywords, you can set to true the Hibernate-specific hibernate.globally_quoted_identifiers configuration property:
<property
name="hibernate.globally_quoted_identifiers"
value="true"
/>
Yaml format
spring:
jpa:
properties:
hibernate:
globally_quoted_identifiers: true
If you use as shown below it should work
#Column(name="[order]")
private int order;
#Column(name="\"open\"")
This will work for sure, Same problem happened with me, when I was learning hibernate.
There is also another option: hibernate.auto_quote_keyword
which
Specifies whether to automatically quote any names that are deemed keywords.
<property name="hibernate.auto_quote_keyword" value="true" />
Yaml
spring:
jpa:
properties:
hibernate:
auto_quote_keyword: true
No - change the column name.
This is database-specific, and you just can't create such a column. After all hibernate finally sends DDL to the database. If you can't create a valid DDL with this column name, this means hibernate can't as well. I don't think quoting would solve the issue even if you are writing the DDL.
Even if you somehow succeed to escape the name - change it. It will work with this database, but won't work with another.
Some JPA implementations (e.g the one I use, DataNucleus) automatically quote the identifier for you, so you never get this.
I am trying to use Liquibase to transform one database into another. I have two databases and would like to run a SQL script on one to make it the same as the other one. I am using Liquibase version 3.4.1 and two H2 databases. In order to generate the SQL script I first generate the diff xml using the diffChangeLog command which works fine. (Actually there is one problem: it generates columns which are of type VARCHAR with autoIncrement set to true, but that can be manually removed). I then use the command updateSQL to generate the SQL file from the diff xml. The SQL file contains the line:
ALTER TABLE PUBLIC.USERS ADD ENABLED BOOLEAN(1) NOT NULL;
Originally the db contains a table named USERS with two columns: USER and PASSWORD with one row of data (user1, password). The SQL script fails at that point as it tries to create a new column whose entry for the first row will be set to NULL but the column is supposed to be NOT NULL. The error is:
Error: NULL not allowed for column "ENABLED";
SQL statement:
ALTER TABLE PUBLIC.USERS ADD ENABLED BOOLEAN(1) NOT NULL [23502-187]
SQLState: 23502
ErrorCode: 23502
Error occured in: ALTER TABLE PUBLIC.USERS ADD ENABLED BOOLEAN(1) NOT NULL
I could try to delete the NOT NULL condition manually, but this doesn't work in all cases as described below. When I do remove the NOT NULL, this is what happens:
Another related error occurs when the following line is executed (right after the one above):
ALTER TABLE PUBLIC.USERS ADD USERNAME VARCHAR(50) NOT NULL;
Note that it is trying to add the column "USERNAME", not "USER", which the original database already has.
It gives the same error, and when I remove NOT NULL, another error comes up:
Error: Column "USERNAME" must not be nullable;
SQL statement:ALTER TABLE PUBLIC.USERS ADD CONSTRAINT CONSTRAINT_4 PRIMARY KEY (USERNAME) [90023-187]
SQLState: 90023
ErrorCode: 90023
Error occured in: ALTER TABLE PUBLIC.USERS ADD CONSTRAINT CONSTRAINT_4 PRIMARY KEY (USERNAME)
Since the column is a primary key it must not be nullable.
Is there a way to fix this easily? Is there an option in Liquibase which would get rid of these errors?
Add the column first, without the NOT NULL setting. Then update the column for the existing rows. Finally, apply the NOT NULL constraint.
<addNotNullConstraint catalogName="cat"
columnDataType="int"
columnName="id"
defaultNullValue="A String"
schemaName="public"
tableName="person"/>
I do not recommend using the defaultValue setting as it could lock up the table if you have a high rowcount. Very bad for maintenance windows in production.
When you're adding a NOT NULL column the trick is to give it a default value. In your Liquibase XML you'd want something like this, for the column definition:
<column
name="ENABLED"
type="BOOLEAN(1)"
defaultValue="0"
>
That should generate a NOT NULL WITH DEFAULT 0 clause, as appropriate for the platform.
It depends on your database. Mysql/Mariadb can be simple like this:
<changeSet author="tibi" id="201803062100-1">
<addColumn tableName="jhi_user">
<column name="status" type="varchar(20)" value="ENABLED" />
<constraints nullable="false" />
</addColumn>
</changeSet>
H2 does not allow this so you need this:
<changeSet author="tibi" id="201803062100-1">
<addColumn tableName="jhi_user">
<column name="status" type="varchar(20)" value="ENABLED" />
</addColumn>
<addNotNullConstraint columnDataType="varchar(20)" columnName="status" tableName="jhi_user" />
</changeSet>
or for your case:
<changeSet author="tibi" id="201803062100-1">
<addColumn tableName="jhi_user">
<column name="status" type="boolean" value="0" />
</addColumn>
<addNotNullConstraint columnDataType="boolean" columnName="status" tableName="jhi_user" />
</changeSet>
I'm trying to run in development mode (with H2) a setup currently used in production with a PostgreSQL database and I get an error. It would be best if I could reuse the production SQL without any change to it.
Using this setup:
# H2 Database
spring.datasource.datasource-class-name=org.h2.jdbcx.JdbcDataSource
spring.datasource.url=jdbc:h2:mem:userregistry;DB_CLOSE_DELAY=-1;MODE=PostgreSQL
This query:
CREATE TABLE IF NOT EXISTS users.user_userrole (
user_signum VARCHAR(20) NOT NULL,
role VARCHAR(255) NOT NULL,
CONSTRAINT user_userrole_pk PRIMARY KEY (user_signum, role),
CONSTRAINT user_fk FOREIGN KEY (user_signum) REFERENCES users.user (signum) MATCH SIMPLE,
CONSTRAINT role_fk FOREIGN KEY (role) REFERENCES users.userrole
(role_name) MATCH SIMPLE
);
Raises this exception:
org.h2.jdbc.JdbcSQLException: Syntax error in SQL statement "<<SQL OMITTED FOR BREVITY>>";
expected "INDEX, ON, NOT, DEFERRABLE, NOCHECK, CHECK, ,, )"; [42001-185]
Note that I am already using Mode=PostgreSQL. Any ideas?
Thanks
The H2 database does not support MATCH SIMPLE (same as Oracle, MS SQL Server, Apache Derby).
This is my second post about hibernate. Since I started using hibernate it has given more problems than the problems it has solved. It almost makes me feel like I should have stuck to plain old JDBC. Anyway,
Here is one of the problems I'm trying to battle.
My Sequence generator in the .hbm files looks as follows.
<id name="id" type="long" column="ID">
<generator class="sequence">
<param name="sequence">ADVENTURES_ID_SEQ</param>
<param name="allocationSize">1</param>
<param name="initialValue">17599</param>
</generator>
</id>
Please note: initial value is 17599. This is because LAST_NUMBER in the oracle sequence is: 17599
CREATED 25-APR-12
LAST_DDL_TIME 25-APR-12
SEQUENCE_OWNER ADVENTURE_ADMIN
SEQUENCE_NAME ADVENTURES_ID_SEQ
MIN_VALUE 1
MAX_VALUE 9999999999999999999999999999
INCREMENT_BY 1
CYCLE_FLAG N
ORDER_FLAG N
CACHE_SIZE 20
LAST_NUMBER 17599
When I run the code I see the next sequence generated as 200, 201 in the Hibernate Debug statement.
DEBUG SQL - select ADVENTURES_ID_SEQ.nextval from dual
DEBUG SequenceGenerator - Sequence identifier generated: 201
I expected the nextval should have been 17600. Seems like the oracle sequence is not getting used at all.
What is wrong in my configuration and how to fix it. Any help is greatly appreciated.
Thanks
Self Answer (Workaround): I'm still seeing the issue but for now I have a workaround. Since it worked for me I'm selecting the question as complete.
I'm letting Oracle generate the nextId using Trigger:
create or replace
TRIGGER ADVENTURE_ADMIN.ADVENTURES_ID_TRIG
BEFORE INSERT ON ADVENTURE_ADMIN.ADVENTURES FOR EACH ROW
WHEN (new.ID IS NULL)
BEGIN
SELECT ADVENTURES_ID_SEQ.NEXTVAL INTO :new.ID FROM DUAL;
END;
Then I'm letting hibernate use the Oracle Trigger generated ID. There are two ways of doing this.
First is by using Hibernate select. Disadvantage of this approach is you will need another column in the table with unique constraint which is used by the hibernate as a key to fetch the row. It doesn't really work for me as I have tables with primary key as the only unique key.
Second is by using TriggerAssignedIdentityGenerator created by Jean-Pol Landrain, Martin Zeltner. Source could be found here. This has enabled me to get around the problem of finding another unique key for the table.
Following is how I'm using it:
<id name="id" type="long" column="ID">
<generator class="org.hibernate.id.TriggerAssignedIdentityGenerator" />
</id>
Please Note: Hibernate version you are using matters. I'm using 3.5.4. hibernate 3.6 didn't work for me. JDBC and Oracle Driver version also matters. You can refer to the documentation in the source file for the same.