Trying to reuse Spring JdbcTemplate connection for Postgres JDBC copyIn - getting 'relation does not exist' - spring

I've hit a brick wall in a Spring/Kotlin/JDBC/Postgres project and am hoping the community can help.
What I'm Trying To Do
Use Spring JDBC API (JdbcTemplate) to create a temporary table (temporary_pokemon) in a Postgres database.
Unwrap the JDBC connection from the JdbcTemplate so I can use the same connection to load a CSV into the database using the Postgres JDBC driver's copyIn method (https://jdbc.postgresql.org/documentation/publicapi/org/postgresql/copy/CopyIn.html)
What Is Going Wrong
The copyIn method errors with the message relation "temporary_pokemon" does not exist, and my assumption here is that the unwrapped connection is somehow separate/different to the db.execute command which creates the table.
Ideally there's a way to re-use the same connection while still being able to rely largely on Spring Boot's autoconfiguration and things like automatic connection pooling, etc.
What I've Tried So Far
Adding the #Transactional annotation
Creating a DataSource manually using Spring Boot's DataSourceBuilder (this seems to work, I am assuming that it only creates a single connection which gets reused)
The error message
Caused by: org.postgresql.util.PSQLException: ERROR: relation "temporary_pokemon" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1212) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.core.v3.QueryExecutorImpl.startCopy(QueryExecutorImpl.java:894) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:45) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:177) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:160) ~[postgresql-42.2.18.jar:42.2.18]
at com.example.demo.IngestPostgres.ingest(IngestPostgres.kt:32) ~[main/:na]
at com.example.demo.IngestPostgres$$FastClassBySpringCGLIB$$f1321c17.invoke(<generated>) ~[main/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.2.jar:5.3.2]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) ~[spring-aop-5.3.2.jar:5.3.2]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.2.jar:5.3.2]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.3.2.jar:5.3.2]
... 14 common frames omitted
Code Snippets
My #Component for the CommandLineRunner:
#Component
class Seed : CommandLineRunner {
#Autowired
lateinit var ingester : IngestPostgres
override fun run(vararg args: String?) {
val buffer = BufferedReader(FileReader(File("src/main/resources/ingest.csv")))
ingester.ingest(buffer)
}
}
The IngestPostgres #Component:
#Component
class IngestPostgres {
#Autowired
private lateinit var db: JdbcTemplate
#Transactional
fun ingest(bufferedReader: BufferedReader) {
db.execute("""
DROP TABLE IF EXISTS temporary_pokemon;
CREATE TABLE temporary_pokemon (
pokemon_id INT,
pokemon_name VARCHAR,
pokemon_type VARCHAR
);
""".trimIndent())
val pgConnection = db.dataSource?.connection?.unwrap(PgConnection::class.java)!!
CopyManager(pgConnection).copyIn(
"COPY temporary_pokemon FROM stdin DELIMITER ',' CSV HEADER",
bufferedReader
)
// snipped - later code INSERTS contents of temporary_pokemon into main pokemon table
}
}
My dependencies in build.gradle.kts:
dependencies {
implementation("org.springframework.boot:spring-boot-starter-jdbc")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
implementation("org.postgresql:postgresql")
testImplementation("org.springframework.boot:spring-boot-starter-test")
}
Thanks for any help. I'm not an expert at Spring/JDBC by any means, so apologies in advance if I've missed something that's common knowledge. I've tried searching on SO and Google but to no avail.

Seems like your queries use different connections. So results of first query are not committed when the second query starts. You can get a single connection and use it for execution of both queries or use JdbcTemplate api for importing csv data to your table

Related

Saving a record using JPA in a Spring Boot Scheduler

I'm using a Spring Boot Scheduler to run a query on the DB daily to find some records based on a condition and update the records returned. Fetching the records using JPA works fine, but when I loop through them, update them, and try to save each updated record I get the following error:
Could not commit JPA transaction; nested exception is javax.persistence.RollbackException: Error while committing the transaction
Caused by: javax.persistence.RollbackException: Error while committing the transaction at org.hibernate.internal.ExceptionConverterImpl.convertCommitException(ExceptionConverterImpl.java:81) at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:104) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:562) ... 30 more Caused by: java.lang.NullPointerException at com.xxx.yyy.config.JpaAuditingConfiguration.auditorProvider$lambda-0(JpaAuditingConfiguration.kt:15) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) at com.sun.proxy.$Proxy168.getCurrentAuditor(Unknown Source) at java.base/java.util.Optional.map(Optional.java:265) at org.springframework.data.auditing.AuditingHandler.getAuditor(AuditingHandler.java:109) at org.springframework.data.auditing.AuditingHandler.markModified(AuditingHandler.java:104) at org.springframework.data.jpa.domain.support.AuditingEntityListener.touchForUpdate(AuditingEntityListener.java:112).
Here is the scheduler code I have. If I run the same code inside my service and call it using an endpoint everything works fine:
#Component
class Scheduler(
private val repository: Repository
) {
#Scheduled(cron = "0 0 2 * * *")
fun expire() {
val records = repository.findRecords()
for (record in records) {
try {
// Call some external API using record.id but this part is commented out for now until the saving works
record.active = false
repository.save(record)
} catch (ex: Exception) {
logger.error("Error expiring record " + record.id)
logger.error("Exception: ${ex.printStackTrace()}")
continue
}
}
}
}
the null pointer exception happens in the JpaAuditingConfiguration config I use for storing the created_at and last_modified_at dates. Here is the code I have for that class:
#Configuration
#EnableJpaAuditing(auditorAwareRef = "auditorProvider")
class JpaAuditingConfiguration {
#Bean
fun auditorProvider(): AuditorAware<String> {
return AuditorAware { Optional.of(SecurityContextHolder.getContext().authentication.name) }
}
}
Your JpaAuditingConfiguration requires the security context to be non null when you make modifications. When you're running your task in a scheduler there is no active request, so no active session, and therefore your authentication is null.
Usually, this is solved by making a special app user and manually authenticating them in your scheduled task.

Spring Boot #RequestScope and Hibernate schema based multi-tenancy

I'm working on a schema based multi-tenant app, in which I want to resolve the Tenant Identifier using a #RequestScope bean. My understanding is that #RequestScope uses/injects proxies for the request scoped beans, wherever they are referred (e.g. in other singleton beans). However, this is not working in the #Component that implements CurrentTenantIdentifierResolver and I get the following error when I start my service,
Caused by: org.springframework.beans.factory.support.ScopeNotActiveException: Error creating bean with name 'scopedTarget.userContext': Scope 'request' is not active for the current thread;
Caused by: java.lang.IllegalStateException: No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.
Following are the relevant pieces of code.
#Component
public class CurrentTenant implements CurrentTenantIdentifierResolver {
#Autowired
private UserContext userContext;
#Override
public String resolveCurrentTenantIdentifier() {
return Optional.of(userContext)
.map(u -> u.getDomain())
.get();
}
#Component
#RequestScope
public class UserContext {
private UUID id;
private String domain;
My questions,
Isn't the proxy for the #RequestScope injected (by default)? Do I need to do anything more?
Is Hibernate/Spring trying to establish a connection to the DB at startup (even when there is no tenant available)?
Hibernate properties:
HashMap<String, Object> properties = new HashMap<>();
properties.put("hibernate.dialect", env.getProperty("hibernate.dialect"));
properties.remove(AvailableSettings.DEFAULT_SCHEMA);
properties.put(AvailableSettings.MULTI_TENANT, MultiTenancyStrategy.SCHEMA);
properties.put(AvailableSettings.MULTI_TENANT_IDENTIFIER_RESOLVER, tenantResolver);
properties.put(AvailableSettings.MULTI_TENANT_CONNECTION_PROVIDER, connectionProvider);
For the time being, I'm preventing the NullPointerException by checking if we are in the RequestContext. However, a connection still gets established to the master database (although I've explicitly specified the dialect and am not specifying hbm2ddl.auto). Since this connection is not associated with any schema, I'd like to avoid making it, so that it does not look for any tables that it won't find anyways.
What seems to be happenning is that when a HTTP request is received, hibernate is trying to resolve the current tenant identifier, even before my #RequestScope bean is created (and even before my #RestController method is called.) If a provide the default connection to the databse, I then get the following error. If I don't provide a connection, it throws an exception and aborts.
2021-09-26 11:55:44.882 WARN 19759 --- [nio-8082-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 42P01
2021-09-26 11:55:44.882 ERROR 19759 --- [nio-8082-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: relation "employees" does not exist
Position: 301
2021-09-26 11:55:44.884 ERROR 19759 --- [nio-8082-exec-2] o.t.n.controller.EmployeeController : Exception: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet

Transactions, Spring Boot Starter JDBC & R2DBC

I am trying to migrate a Spring Boot project, version 2.3.0.M3, that have used JDBC template to R2DBC. The project also uses Liquibase so I cannot get rid of JDBC altogether.
I have both the spring-boot-starter-data-r2dbc and the spring-boot-starter-jdbc dependencies in the project with which I get the following exception when trying to run one of my tests:
org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type 'org.springframework.transaction.TransactionManager' available: expected single matching bean but found 2: transactionManager,connectionFactoryTransactionManager
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1180)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:416)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:349)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:342)
at org.springframework.transaction.interceptor.TransactionAspectSupport.determineTransactionManager(TransactionAspectSupport.java:480)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:335)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
...
The bean connectionFactoryTransaction manager is defined like this in the Spring class R2dbcTransactionManagerAutoConfiguration:
#Bean
#ConditionalOnMissingBean(ReactiveTransactionManager.class)
public R2dbcTransactionManager connectionFactoryTransactionManager(ConnectionFactory connectionFactory) {
return new R2dbcTransactionManager(connectionFactory);
}
The bean transactionManager is defined like this in the Spring class DataSourceTransactionManagerAutoConfiguration:
#Bean
#ConditionalOnMissingBean(PlatformTransactionManager.class)
DataSourceTransactionManager transactionManager(DataSource dataSource,
ObjectProvider<TransactionManagerCustomizers> transactionManagerCustomizers) {
DataSourceTransactionManager transactionManager = new DataSourceTransactionManager(dataSource);
transactionManagerCustomizers.ifAvailable((customizers) -> customizers.customize(transactionManager));
return transactionManager;
}
As can be seen, the #ConditionalOnMissingBean annotation contains different types which will cause an instance of both beans to be created.
However, in the Spring class TransactionAspectSupport there is this line of code in the determineTransactionManager method:
defaultTransactionManager = this.beanFactory.getBean(TransactionManager.class);
Since both of the transaction manager types, DataSourceTransactionManager and R2dbcTransactionManager, implement the TransactionManager interface, both the transaction manager beans as above will be matched and the error will occur.
I am now reaching out to hear if there is anyone who has managed to solve or work around this issue?
Thanks in advance!
With inspiration from M. Deinums answer (thanks!), I applied the following steps to my project and the test that failed earlier now runs successfully:
Remove the spring-boot-starter-jdbc dependency.
Add a dependency to spring-jdbc.
Add a dependency to HikariCP (com.zaxxer).
Add spring.liquibase user and password properties (I already had the url and change-log properties).
Remove all spring.datasource properties (I had url and drive-class-name).
I had the spring.r2dbc properties username, password and url defined which I did not need to change.
Update:
In addition, I used Testcontainers in the tests and could not assign a static port. In order to be able to configure the database port on Liquibase, I overrode a bean name liquibase of the type SpringLiquibase and created a DataSource (not exposed as a bean) in the liquibase bean creation method and set it on the liquibase bean.
It's possible to have spring-boot-starter-jdbc and spring-boot-starter-data-r2dbc co-exist. There is a class org.springframework.transaction.annotation.TransactionManagementConfigurer that can be used to resolve the conflict.
Spring Boot 2.3.0 seems to disable automatic datasource config when r2dbc is present. It's possible to manually import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration class to make both co-exist.
#Bean
TransactionManagementConfigurer transactionManagementConfigurer(ReactiveTransactionManager reactiveTransactionManager) {
return new TransactionManagementConfigurer() {
#Override
public TransactionManager annotationDrivenTransactionManager() {
return reactiveTransactionManager;
}
};
}

Spring boot, multi-tenet, multi-module, #Transactional , parallelStream

I am trying to insert some 50k records into db. We have used AbstractRoutingDataSource which resolve Datasource using TenantContext which is a utility class and has a private static final ThreadLocal CURRENT_TENANT = new ThreadLocal<>();
when I am using parallel stream or if I am trying to make the method #Async I am getting the below error
Code:
.parallelStream()
.forEach(row -> {
TenantContext.setCurrentTenant(centerCd);
someDao.insert(row);
});
Error:
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.lang.IllegalStateException: Cannot determine target DataSource for lookup key [null]
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:305)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:474)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot determine target DataSource for lookup key [null]
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.determineTargetDataSource(AbstractRoutingDataSource.java:207)
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:169)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:262)
... 10 common frames omitted
It works exactly like you described: your TenantContext is exactly ThreadLocal and exists in a thread, which is initiated either by parallelStream() or Async method. (in reality, the call inside of the Async or forEach method is a run from Runnable)
The data source is attempted to be injected/resolved at start of the thread: because your transaction have to be started at the thread creation, before your Runnable gets into a run method. And at this moment of time you haven't yet specified your tenant, call TenantContext.setCurrentTenant(centerCd) is performed later in a run method implementation.
I would suggest applying such structure to your code:
class TenantAwareThread extends Thread {
public TenantAwareThread(Runnable target, TenantData tenantData) {
super(target);
TenantContext.setCurrentTenant(tenantData);
}
}
#Autowired
TaskExecutor executor;
void startTask(TenantData tenantData, RowData row) {
executor.execute(
new TenantAwareThread(() -> {
someDao.insert(row);
},
tenantData));
}
You create a new thread type which is aware of tenant data from the very beginning. And simply wraps your executions into such thread.

Spring boot 2 Prometheus not pulling db metrics

I am using spring boot2 with Prometheus. we are using Postgres as db. The Prometheus url is not fetching db metrics.
Any references would be very helpful.
I have tried
#Configuration
#AutoConfigureAfter({DataSourceAutoConfiguration.class})
public class MyConfiguration {
#Autowired
HikariDataSource dataSource;
#Bean
PostgreSQLDatabaseMetrics dbMeterics() {
return new PostgreSQLDatabaseMetrics(dataSource, "database-name");
}
}
When I hit the end point /prometheus, I get this error
java.lang.NullPointerException: null
at io.micrometer.core.instrument.binder.db.PostgreSQLDatabaseMetrics.lambda$bindTo$1(PostgreSQLDatabaseMetrics.java:101)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:40)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$3(PrometheusMeterRegistry.java:235)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:69)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at org.springframework.boot.actuate.metrics.export.prometheus.PrometheusScrapeEndpoint.scrape(PrometheusScrapeEndpoint.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76)
In line 101 the registry received via DI is used to build the pgMetrics object. Since you are getting the null pointer at this line, you might be missing the import of Prometheus registry as a dependency.

Resources