I have the following spring data source setup:
datasource:
name: postgres-datasource
url: ${POSTGRES_URL:jdbc:postgresql://localhost:5432/mydb}?reWriteBatchedInserts=true&prepareThreshold=0
username: ${POSTGRES_USER:mydb}
password: ${POSTGRES_PASS:12345}
driver-class: org.postgresql.Driver
hikari:
minimumIdle: 2
maximum-pool-size: 30
max-lifetime: 500000
idleTimeout: 120000
auto-commit: false
data-source-properties:
cachePrepStmts: true
useServerPrepStmts: true
prepStmtCacheSize: 500
jpa:
database-platform: org.hibernate.dialect.PostgreSQLDialect
properties:
hibernate:
dialect: org.hibernate.dialect.PostgreSQLDialect
# generate_statistics: true
order_inserts: true
order_updates: true
jdbc:
lob:
non_contextual_creation: true
batch_size: 50
Notice that auto-commit is false.
Since, I need to use both jooq and JPA and also have multiple schemas in my db, I have configured the following DataSourceConnectionProvider
public class SchemaSettingDataSourceConnectionProvider extends DataSourceConnectionProvider {
public SchemaSettingDataSourceConnectionProvider(TransactionAwareDataSourceProxy dataSource) {
super(dataSource);
}
public Connection acquire() {
try {
String tenant = TenantContext.getTenantId();
log.debug("Setting schema to {}", tenant);
Connection connection = dataSource().getConnection();
Statement statement = connection.createStatement();
statement.executeUpdate("SET SCHEMA '" + tenant + "'");
statement.close();
return connection;
} catch (SQLException var2) {
throw new DataAccessException("Error getting connection from data source " + dataSource(), var2);
}
}
I have the #EnableTransactionManagement on the spring boot config. With this setup in place, the connection is not committing after transaction is over.
#Transactional(propagation = Propagation.REQUIRES_NEW)
public FlowRecord insert(FlowName name, String createdBy) {
return dslContext.insertInto(FLOW, FLOW.NAME, FLOW.STATUS)
.values(name.name(), FlowStatus.CREATED.name())
.returning(FLOW.ID)
.fetch()
.get(0);
}
This does not commit. So, I tried adding the following code to my SchemaSettingDataSourceConnectionProvider class
#Override
public void release(Connection connection) {
connection.commit();
super.release(connection);
}
However, now the issue is that even when a transaction should get rolled back, for e.g due to a runtime exception, it still commits all the time.
Is there some configuration I am missing
UPDATE
Following the answer below, I provided a DataSourceTransactionManager bean and it worked for JOOQ.
public DataSourceTransactionManager jooqTransactionManager(DataSource dataSource) {
// DSTM is a PlatformTransactionManager
return new DataSourceTransactionManager(dataSource);
}
However, now my regular JPA calls are all failing with
Caused by: javax.persistence.TransactionRequiredException: Executing an update/delete query
at org.hibernate.internal.AbstractSharedSessionContract.checkTransactionNeededForUpdateOperation(AbstractSharedSessionContract.java:445)
at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1692)
So, I provided a JpaTransactionManager bean. Now this causes JOOQ Auto Configuration to throw multiple DataSourceTransactionManager beans present exception. After much trial and error, the one that worked for me was this:
private final TransactionAwareDataSourceProxy dataSource;
public DslConfig(DataSource dataSource) {
// A transaction aware datasource is needed, otherwise the spring #Transactional is ignored and updates do not work.
this.dataSource = new TransactionAwareDataSourceProxy(dataSource);
}
#Bean("transactionManager")
public PlatformTransactionManager transactionManager() {
// Needed for jpa transactions to work
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setDataSource(dataSource);
return transactionManager;
}
Notice that I am using a JpaTransactionManager but setting the datasource to be TransactionAwareDataSourceProxy. Further testing required, but looks like both JPA and JOOQ transactions are working now.
One thing to watch out for is to make sure you're using the correct #Transactional annotation. In my project there were two: one from a Jakarta package and one from a Spring package - make sure you're using the Spring annotation.
I don't know if your Spring config is correct.
We use Java config so it's hard to compare.
One obvious difference is that we have an explicit TransactionManager defined, is that something you maybe need to do?
Related
Hi All currently we are have deployed our springboot code to pcf which is running on aws.
we are using aws database - where we have cup service and VCAP_SERVICES which hold the parameter of db.
Below our configuration to get datasource
#Bean
public DataSource dataSource() {
if (dataSource == null) {
dataSource = connectionFactory().dataSource();
configureDataSource(dataSource);
}
return dataSource;
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(dataSource());
}
private void configureDataSource(DataSource dataSource) {
org.apache.tomcat.jdbc.pool.DataSource tomcatDataSource = asTomcatDatasource(dataSource);
tomcatDataSource.setTestOnBorrow(true);
tomcatDataSource.setValidationQuery("SELECT 1");
tomcatDataSource.setValidationInterval(30000);
tomcatDataSource.setTestWhileIdle(true);
tomcatDataSource.setTimeBetweenEvictionRunsMillis(60000);
tomcatDataSource.setRemoveAbandoned(true);
tomcatDataSource.setRemoveAbandonedTimeout(60);
tomcatDataSource.setMaxActive(Environment.getAsInt("MAX_ACTIVE_DB_CONNECTIONS", tomcatDataSource.getMaxActive()));
}
private org.apache.tomcat.jdbc.pool.DataSource asTomcatDatasource(DataSource dataSource) {
Objects.requireNonNull(dataSource, "There is no DataSource configured");
DataSource targetDataSource = ((DelegatingDataSource)dataSource).getTargetDataSource();
return (org.apache.tomcat.jdbc.pool.DataSource) targetDataSource;
}
Now when we have read replicas created , what configuration do i need to modify so our spring boot application uses the read replicas?
is Just #Transactional(readOnly = true) on the get call is enough - that it will be automatically taken care? or do i need to add some more configuration
#Repository
public class PostgresSomeRepository implements SomeRepository {
#Autowired
public PostgresSomeRepository(JdbcTemplate jdbcTemplate, RowMapper<Consent> rowMapper) {
this.jdbcTemplate = jdbcTemplate;
this.rowMapper = rowMapper;
}
#Override
#Transactional(readOnly = true)
public List<SomeValue> getSomeGetCall(List<String> userIds, String applicationName, String propositionName, String since, String... types) {
//Some Logic
try {
return jdbcTemplate.query(sql, rowMapper, paramList.toArray());
} catch (DataAccessException ex) {
throw new ErrorGettingConsent(ex.getMessage(), ex);
}
}
}
Note:we have not added any spring aws jdbc dependency
Let's assume the cloud service name is my_db.
Map the cloud service to the application config appication-cloud.yml used by default in the CF (BTW this is better than using the connector because you can customize the datasource)
spring:
datasource:
type: com.zaxxer.hikari.HikariDataSource
# my_db
url: ${vcap.services.my_db.credentials.url}
username: ${vcap.services.my_db.credentials.username}
password: ${vcap.services.my_db.credentials.password}
hikari:
poolName: Hikari
auto-commit: false
data-source-properties:
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
jpa:
generate-ddl: false
show-sql: true
put the service to the application manifest.yml:
---
applications:
- name: my-app
env:
SPRING_PROFILES_ACTIVE: "cloud" # by default
services:
- my_db
We have several systems, using same core libraries and same Oracle DB. But just one system get some error every day, below is a stacktrace.
The error is: ORA-04042: procedure, function, package, or package body does not exist. The difference between this system and others is, that this system uses several datasources, below you can see the Hikari config, part of build.gradle and stacktrace. All other system uses one datasource.
This is Oracle version info:
BANNER CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Release 12.1.0.1.0 - 64bit Production 0
PL/SQL Release 12.1.0.1.0 - Production 0
CORE 12.1.0.1.0 Production 0
TNS for 64-bit Windows: Version 12.1.0.1.0 - Production 0
NLSRTL Version 12.1.0.1.0 - Production 0
This is a code used for config one of the datasources. Second one is prodused in same way:
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "sourceEntityManagerFactory",
transactionManagerRef = "sourceTransactionManager",
basePackages = { "com.maxi.jpa.source" }
)
public class SourceConfig
{
#Bean(name = "sourceDS")
#Primary
#ConfigurationProperties("spring.datasource.ds-src")
public DataSource dataSource(){
return DataSourceBuilder.create().build();
}
#Bean(name = "sourceEntityManagerFactory")
#DependsOn("sourceDS")
#Primary
public LocalContainerEntityManagerFactoryBean sourceEMF(
EntityManagerFactoryBuilder builder,
#Qualifier("sourceDS")
DataSource ds
) {
return builder
.dataSource(ds)
.packages("com.maxi.jpa.model")
.persistenceUnit("source")
.build();
}
#Bean(name = "sourceEntityManager")
#Primary
public EntityManager sourceEM(
#Qualifier("sourceEntityManagerFactory")
EntityManagerFactory factory
) {
return factory.createEntityManager();
}
#Bean(name = "sourceTransactionManager")
#Primary
public PlatformTransactionManager transactionManager(
#Qualifier("sourceEntityManagerFactory") EntityManagerFactory
entityManagerFactory
) {
return new JpaTransactionManager(entityManagerFactory);
}
}
Here is the code where the exception is:
private List<String> descTable(String owner, String tableName) {
String query =
"select column_name " +
"from all_tab_columns " +
"where upper(owner) = '%s' " +
"and upper(table_name) = '%s'";
query = String.format(query, owner, tableName);
#SuppressWarnings("unchecked")
List<String> result = (List<String>)sourceEntityManager
.createNativeQuery(query)
.getResultList();
return result;
}
This way, I inject sourceEntityManager:
#Service
public class DMLService
{
private EntityManager sourceEntityManager;
private EntityManager destEntityManager;
public DMLService(
#Qualifier("sourceEntityManager")
EntityManager sourceEntityManager,
#Qualifier("destEntityManager")
EntityManager destEntityManager
) {
this.sourceEntityManager = sourceEntityManager;
this.destEntityManager = destEntityManager;
}
....
}
application.yaml
spring:
datasource:
ds-src:
driverClassName: oracle.jdbc.OracleDriver
jdbcUrl: jdbc:oracle:thin:#${DB4VAL_SRC_DB}
username: ${DB4VAL_SRC_USERNAME}
password: ${DB4VAL_SRC_PASSWORD}
poolName: Db4ValidateSource
connectionTestQuery: SELECT 1 FROM DUAL
ds-dest:
driverClassName: oracle.jdbc.OracleDriver
jdbcUrl: jdbc:oracle:thin:#${DB4VAL_DEST_DB}
username: ${DB4VAL_DEST_USERNAME}
password: ${DB4VAL_DEST_PASSWORD}
poolName: Db4ValidateDestination
connectionTestQuery: SELECT 1 FROM DUAL
jpa:
database-platform: org.hibernate.dialect.Oracle12cDialect
application-default.yaml
spring:
datasource:
ds-src:
minimumIdle: 1
maximumPoolSize: 5
idleTimeout: 20000
maxLifetime: 60000
keepaliveTime: 10000
connectionTimeout: 30000
ds-dest:
minimumIdle: 1
maximumPoolSize: 5
idleTimeout: 20000
maxLifetime: 60000
keepaliveTime: 10000
connectionTimeout: 30000
build.gradle
dependencies {
implementation 'org.apache.commons:commons-lang3'
implementation 'commons-net:commons-net:3.6'
implementation 'commons-io:commons-io:2.6'
implementation 'com.zaxxer:HikariCP:4.0.2'
implementation 'org.springframework:spring-context-support'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.boot:spring-boot-starter-jdbc'
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
...
}
stacktrace
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:628)
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:557)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:730)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:291)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:492)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:148)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:928)
at oracle.jdbc.driver.OracleStatement.prepareDefineBufferAndExecute(OracleStatement.java:1158)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1093)
at oracle.jdbc.driver.OracleStatement.executeSQLSelect(OracleStatement.java:1402)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1285)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3735)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3847)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1098)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2297)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2050)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2012)
at org.hibernate.loader.Loader.doQuery(Loader.java:948)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
at org.hibernate.loader.Loader.doList(Loader.java:2843)
at org.hibernate.loader.Loader.doList(Loader.java:2825)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2657)
at org.hibernate.loader.Loader.list(Loader.java:2652)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:338)
at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2141)
at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:1169)
at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:176)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1604)
at org.hibernate.query.Query.getResultList(Query.java:165)
at com.maxi.services.DMLService.descTable(DMLService.java:77)
at com.maxi.services.DMLService.getTableData(DMLService.java:191)
at com.maxi.services.DMLService.processRecord(DMLService.java:203)
at com.maxi.services.ValidateService.processRecord(ValidateService.java:263)
at com.maxi.services.ValidateService.runETL(ValidateService.java:362)
at com.maxi.services.Handler.runProcess(Handler.java:54)
at com.maxi.services.Handler.lambda$handleRequest$0(Handler.java:35)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
As you can see, the system catches all the configuration
So, the problem was, that EntityManager retrievs connection from pool, and doesn't returns it, till it will be closed. But we don't able to close it manually, since it is a bean, that we possible will not able to reopen.
So the solution is not to create the EM manuualy, but fetch it from framework, using #PersistenceContext(unitName = "unit_name").
It is necessary to remove this code (for both EMs):
#Bean(name = "sourceEntityManager")
#Primary
public EntityManager sourceEM(
#Qualifier("sourceEntityManagerFactory")
EntityManagerFactory factory
) {
return factory.createEntityManager();
}
#Bean(name = "sourceTransactionManager")
#Primary
public PlatformTransactionManager transactionManager(
#Qualifier("sourceEntityManagerFactory") EntityManagerFactory
entityManagerFactory
) {
return new JpaTransactionManager(entityManagerFactory);
}
And to Autowire the beans in following way:
#PersistenceContext(unitName = "source")
private EntityManager sourceEntityManager;
#PersistenceContext(unitName = "destination")
private EntityManager destEntityManager;
Don't forget to rewrite code for use #Transactions (jpa or spring). Put the annotation over public method or whole class.
My spring boot app is using route to read from mySQL DB.
from("sql:select * from Students?repeatCount=1").convertBodyTo(String.class)
.to("file:outbox");
Now I want to create route to read from in memory H2 DB but Im not sure which camel components to use and how to create the route.
If you got Spring Boot, then you can actually inject everything via the dataSource=#mysql parameter and introduce separate DataSource-Beans and use the required ones:
#Configuration
public class DataSourceConfig {
#Bean(name = "mysqlProperties")
#ConfigurationProperties("spring.datasource.mysql")
public DataSourceProperties mysqlDataSourceProperties() {
return new DataSourceProperties();
}
#Bean(name = "mysql")
public DataSource mysqlDataSource(#Qualifier("mysqlProperties") DataSourceProperties properties) {
return properties
.initializeDataSourceBuilder()
.build();
}
#Bean(name = "h2Properties")
#ConfigurationProperties("spring.datasource.h2")
public DataSourceProperties h2DataSourceProperties() {
return new DataSourceProperties();
}
#Bean(name = "h2")
public DataSource h2DataSource(#Qualifier("h2Properties") DataSourceProperties properties) {
return properties
.initializeDataSourceBuilder()
.build();
}
}
Afterwards you can declare the different DataSources in you application.yml but don't forget to disable/exclude the datasource autoconfig.
spring:
autoconfigure:
exclude:
- org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration
datasource:
mysql:
url: jdbc:mysql://localhost:3306/db_example
username: user
password: pass
driver-class-name: com.mysql.cj.jdbc.Driver
h2:
url: jdbc:h2:mem:h2testdb
username: sa
password: sa
driver-class-name: org.h2.Driver
Your Camel-Route should look like this to use everything properly:
from("sql:select * from Students?repeatCount=1&dataSource=#mysql")
.convertBodyTo(String.class)
.to("file:outbox");
Hikari datasource configurations always taking as default values. even though i provide the actual configurations in application.yml file.
MainApp.java
#SpringBootApplication
public class MainApp {
public static void main(String[] args) {
SpringApplication.run(MainApp.class, args);
}
#Bean
#Primary
public DataSourceProperties dataSourceProperties() {
return new DataSourceProperties();
}
//I use below method to set password from different approach instead of taking from app.yml
#Bean
public DataSource dataSource(DataSourceProperties properties) {
DataSource ds = properties.initializeDataSourceBuilder()
.password("setting a password from vault")
.build();
return ds;
}
}
application.yml
spring:
application:
name: demo
datasource:
hikari:
connection-timeout: 20000
minimum-idle: 5
maximum-pool-size: 12
idle-timeout: 300000
max-lifetime: 1200000
auto-commit: true
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
url: jdbc:sqlserver://ip:port;databaseName=sample
username: username
Using Spring 2.1.1.RELEASE and when i start the application i logged the Hikari logs and it says all default values. So i did debugged on the second bean datasource and its actually HikariDatasource and except default values rest of them are empty and not reflecting the given values from the application.yml. Please recommend or comment if i misconfigured anything or done any mistake!
I migrated some projects from Springboot 1.5.10 to 2.0.
Springboot 2.0.3Release, JDK10, mysql, hikari-cp
After this work, in JUnit test, all data in test cases remains at database. I think it doesn't works #Tranactional - org.springframework.transaction.annotation.Transactional
Here is part of application.yml and test class.
spring:
datasource:
driver-class-name: com.mysql.jdbc.Driver
jpa:
show-sql: true
hibernate:
ddl-auto: none
database: mysql
database-platform: org.hibernate.dialect.MySQL5Dialect
Here is datasource.
#Configuration
#EnableTransactionManagement
public class DatasourceJPAConfig {
#Bean
#Primary
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
}
Here is part of JUnit test class.
#Transactional
#Rollback(true)
#RunWith(SpringRunner.class)
#ActiveProfiles("local")
#SpringBootTest
public class RepoTests {
#Autowired
private TestRepository testRepository;
#Test
public void saveTest() {
var name = "test";
var description = "test description"
var item = TestDomain.builder()
.name(name)
.description(description)
.build();
testRepository.save(item);
var optional = testRepository.findById(item.getId());
assertTrue(optional.isPresent());
assertEquals(optional.get().getDescription(), description);
assertEquals(optional.get().getName(), name);
}
}
after to run saveTest method, increase 1 row at database.
Add datasource to test and set auto commit to false
#Autowired
private DataSource dataSource;
And inside test
((HikariDataSource)dataSource).setAutoCommit(false);