I'm trying to setup tests with Spring Boot and Mongo Embedded (JHipster, Flapdoodle).
In general it works. I can see database in Robomongo with created test collections and objects (migrations using Mongobee).
But when I want to access Clients collection:
List clients = clientRepository.findAll();
It throws:
java.lang.IllegalArgumentException: Given DBObject must be a BasicDBObject! Object of class [org.jongo.bson.RelaxedLazyDBObject] must be an instance of class com.mongodb.BasicDBObject
When I run application with real Mongo instance:
./mvnw -Pdev
there are no errors even though test and dev configurations use the same Mongobee mibrations.
Spring documentation is very laconic:
http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#boot-features-mongo-embedded
My test configuration file application.yml is:
spring:
mongodb:
embedded:
version: 3.2.1
data:
mongodb:
host: localhost
port: 27117
database: mydb-test
I use Mongo (real instance) version v3.2.7, in configuration highest possible version of Embedded Mongo which I set is 3.2.1, maybe this could be a problem?
Maybe someone could share his configuration with example working tests?
I find my self the solution.
Problem was that I used org.jongo.Jongo instead of JHipster default com.mongodb.Db in #ChangeSet.
For some reasons Jongo doesn't work well with Embedded Mongo.
When I switched to Db, all problems has gone.
NOT WORKING:
#ChangeSet(...)
public void someChange(Jongo jongo) throws IOException {
org.jongo.MongoCollection collection = jongo.getCollection("collection");
DBObject basicDBObject = new BasicDBObject();
collection.insert(basicDBObject);
...
}
WORKING:
#ChangeSet(...)
public void someChange(Db db) throws IOException {
com.mongodb.MongoCollection collection = db.getCollection("collection");
DBObject basicDBObject = new BasicDBObject();
collection.insert(basicDBObject);
...
}
Related
I have a junit test, making use of testcontainers-1.15.1. How can I start an explicit image? Because:
#SpringBootTest
public class ContainerTest {
private final JdbcDatabaseContainer DB = new MariaDBContainer("mariadb:10.5.8");
static {
DB.start();
}
#Test
public void test() {
}
}
Result: the default 10.3.6 container is started.
[][] 2021-02-04 14:32:50,741 INFO ?.3.6]: Creating container for image: mariadb:10.3.6
[][] 2021-02-04 14:32:51,597 INFO ?.3.6]: Container mariadb:10.3.6 is starting: d9ccf77f4b9165ccd1690ee5cb8437f43e7d853dfe5121d468a391d67eccef7d
application.properties:
spring.datasource.url=jdbc:tc:mariadb:///test
spring.datasource.username=test
spring.datasource.password=test
This might due to an inconsistent behavior of the constructors of the different Testcontainers modules in the past. It was fixed with this commit and should be available since Testcontainers 1.15.0.
Not sure if your sample was pseudo test code, but the following example is a valid copy-pastable example:
public class MariaDbContainerTest {
private static final JdbcDatabaseContainer DB = new MariaDBContainer("mariadb:10.5.8");
static {
DB.start();
}
#Test
public void test() {
}
}
I've tested it for both Testcontainers 1.15.0 and 1.15.1 and it works on my machine.
UPDATE: I've not seen that you also specify the JDBC support of Testcontainers inside your application.properties file in addition to your manual container definition as part of your test.
Pick either the JDBC support OR the manual container definition and your problem should be resolved.
When using the JDBC support you can also specify the version of your database: jdbc:tc:mariadb:10.5.8:///test
Problem with connection to Neo4j test container using Spring boot 2 and JUnit5
int test context. Container started successfully but spring.data.neo4j.uri property has a wrong default port:7687, I guess this URI must be the same when I call neo4jContainer.getBoltUrl().
Everything works fine in this case:
#Testcontainers
public class ExampleTest {
#Container
private static Neo4jContainer neo4jContainer = new Neo4jContainer()
.withAdminPassword(null); // Disable password
#Test
void testSomethingUsingBolt() {
// Retrieve the Bolt URL from the container
String boltUrl = neo4jContainer.getBoltUrl();
try (
Driver driver = GraphDatabase.driver(boltUrl, AuthTokens.none());
Session session = driver.session()
) {
long one = session.run("RETURN 1",
Collections.emptyMap()).next().get(0).asLong();
assertThat(one, is(1L));
} catch (Exception e) {
fail(e.getMessage());
}
}
}
But SessionFactory is not created for the application using autoconfiguration following to these recommendations - https://www.testcontainers.org/modules/databases/neo4j/
When I try to create own primary bean - SessionFactory in test context I get the message like this - "URI cannot be returned before the container is not loaded"
But Application runs and works perfect using autoconfiguration and neo4j started in a container, the same cannot be told about the test context
You cannot rely 100% on Spring Boot's auto configuration (for production) in this case because it will read the application.properties or use the default values for the connection.
To achieve what you want to, the key part is to create a custom (Neo4j-OGM) Configuration bean. The #DataNeo4jTest annotation is provided by the spring-boot-test-autoconfigure module.
#Testcontainers
#DataNeo4jTest
public class TestClass {
#TestConfiguration
static class Config {
#Bean
public org.neo4j.ogm.config.Configuration configuration() {
return new Configuration.Builder()
.uri(databaseServer.getBoltUrl())
.credentials("neo4j", databaseServer.getAdminPassword())
.build();
}
}
// your tests
}
For a broader explanation have a look at this blog post. Esp. the section Using with Neo4j-OGM and SDN.
I have started a GemFire Server and Locator via Spring Boot and when I try to connect to the Locator from GFSH, I getting the following issue:
gfsh> connect
Connecting to Locator at [host=localhost, port=10334] ..
Connection refused: connect
Below, is the Spring (Java) configuration:
#Configuration
#ComponentScan
#EnableGemfireRepositories(basePackages= "com.gemfire.demo")
#CacheServerApplication(locators = "localhost[10334]")
#EnableManager
public class GemfireConfiguration {
#Bean
Properties gemfireProperties() {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", "SpringDataGemFireApplication");
gemfireProperties.setProperty("mcast-port", "0");
gemfireProperties.setProperty("log-level", "info");
return gemfireProperties;
}
#Bean
#Autowired
CacheFactoryBean gemfireCache() {
CacheFactoryBean gemfireCache = new CacheFactoryBean();
gemfireCache.setClose(true);
gemfireCache.setProperties(gemfireProperties());
return gemfireCache;
}
#Bean(name="employee")
#Autowired
LocalRegionFactoryBean<String, Employee> getEmployee(final GemFireCache cache) {
LocalRegionFactoryBean<String, Employee> employeeRegion = new LocalRegionFactoryBean<String, Employee>();
employeeRegion.setCache(cache);
employeeRegion.setClose(false);
employeeRegion.setName("employee");
employeeRegion.setPersistent(false);
employeeRegion.setDataPolicy(DataPolicy.PRELOADED);
return employeeRegion;
}
}
Ref:Spring Data Gemfire locator
as per John's advice, I have enabled the Manager though I am still unable to connect.
You are not able to connect to the Locator (using Gfsh) because you don't have a Locator (service, either standalone or embedded) running with just the Spring (Java) config shown above.
Note that the #CacheServerApplication(locators = "localhost[10334]") annotation, specifically with the locators attribute as you have configured above, does NOT start an embedded Locator. It simply allows this Spring Boot configured and bootstrapped Apache Geode or Pivotal GemFire peer Cache node to join an existing distributed system (cluster) with an "existing" Locator running on localhost, listening on port 10334.
For instance, you could have started a Locator using Gfsh (e.g. start locator --name=X ...), then started your Spring Boot application with the Spring (Java) config shown above and you would see the Spring Boot app as part of the cluster formed by the Gfsh started Locator.
It is simply a shortcut (convenience) to configure and start an "embedded" Locator, but to do so, you need to use the #EnableLocator annotation.
Therefore, to configure and start an (embedded) Locator service in the same Spring Boot application as the CacheServer (and Manager), you must also include the #EnableLocator annotation, like so:
#SpringBootApplicaton
#CacheServerApplication
#EnableLocator
#EnableManager(start = true)
public class GemFireServerApplication {
...
}
I have plenty of examples of this here, for instance here, and talk about this generally here, etc.
As a side note, your whole configuration (class) is confused and it is clear you don't quite understand what you are doing. For instance, declaring the gemfireProperties and gemfireCache beans in JavaConfig is redundant and unnecessary since you are using the #CacheServerApplication annotation. Your whole configuration could be simplified to:
#CacheServerApplication(
name = "SpringDataGemFireApplication",
locators = "localhost[10334]",
logLevel = "info"
)
#EnableLocator
#EnableManager(start = true)
#EnableGemfireRepositories(basePackages= "com.gemfire.demo")
#ComponentScan
public class GemfireConfiguration {
#Bean(name="employee")
LocalRegionFactoryBean<String, Employee> getEmployee(GemFireCache cache) {
LocalRegionFactoryBean<String, Employee> employeeRegion =
new LocalRegionFactoryBean<String, Employee>();
employeeRegion.setCache(cache);
employeeRegion.setClose(false);
employeeRegion.setName("employee");
employeeRegion.setPersistent(false);
employeeRegion.setDataPolicy(DataPolicy.PRELOADED);
return employeeRegion;
}
}
Two things:
1) First, I would be highly careful about using classpath component scanning (#ComponentScan). I am not a fan of this configuration approach, especially in production where things should be explicit.
2) I would encourage your to considering using the type-safe basePackageClasses attribute on the #EnableGemFireRepositorities annotation rather than the basePackages attribute. With basePackageClasses, you only need to refer to a single interface/class in the desired package (e.g. com.gemfire.demo) rather than every interface/class. The referenced interface/class serves as a pointer to identify the package to scan from, including all sub-packages. It is type-safe and when your interfaces/classes in that package are re-located, then your attribute is still valid after the refactoring.
Anyway...
Hope this helps.
-j
I need to set some specific Oracle JDBC connection properties in order to speed up batch INSERTs (defaultBatchValue) and mass SELECTs (defaultRowPrefetch).
I got suggestions how to achieve this with DBCP (Thanks to M. Deinum) but I would like to:
keep the default Tomcat jdbc connection pool
keep application.yml for configuration
I was thinking about a feature request to support spring.datasource.custom_connection_properties or similar in the future and because of this tried to pretent this was already possible. I did this by passing the relevant information while creating the DataSource and manipulated the creation of the DataSource like this:
#Bean
public DataSource dataSource() {
DataSource ds = null;
try {
Field props = DataSourceBuilder.class.getDeclaredField("properties");
props.setAccessible(true);
DataSourceBuilder builder = DataSourceBuilder.create();
Map<String, String> properties = (Map<String, String>) props.get(builder);
properties.put("defaultRowPrefetch", "1000");
properties.put("defaultBatchValue", "1000");
ds = builder.url( "jdbc:oracle:thin:#xyz:1521:abc" ).username( "ihave" ).password( "wonttell" ).build();
properties = (Map<String, String>) props.get(builder);
log.debug("properties after: {}", properties);
} ... leaving out the catches ...
}
log.debug("We are using this datasource: {}", ds);
return ds;
}
In the logs I can see that I am creating the correct DataSource:
2016-01-18 14:40:32.924 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : We are using this datasource: org.apache.tomcat.jdbc.pool.DataSource#19f040ba{ConnectionPool[defaultAutoCommit=null; ...
2016-01-18 14:40:32.919 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : properties after: {password=wonttell, driverClassName=oracle.jdbc.OracleDriver, defaultRowPrefetch=1000, defaultBatchValue=1000, url=jdbc:oracle:thin:#xyz:1521:abc, username=ihave}
The actuator shows me that my code replaced the datasource:
But the settings are not activated, which I can see while profiling the application. The defaultRowPrefetch is still at 10 which causes my SELECTs to be much slower than they would be if 1000 was activated.
Setting the pools connectionProperties should work. Those will be passed to the JDBC driver. Add this to application.properties:
spring.datasource.connectionProperties: defaultRowPrefetch=1000;defaultBatchValue=1000
Edit (some background information):
Note also that you can configure any of the DataSource implementation
specific properties via spring.datasource.*: refer to the
documentation of the connection pool implementation you are using for
more details.
source: spring-boot documentation
As Spring Boot is EOL for a long time I switched to Spring Boot 2.1 with its new default connection pool Hikari. Here the solution is even more simply and can be done in the application.properties or (like shown here) application.yml:
spring:
datasource:
hikari:
data-source-properties:
defaultRowPrefetch: 1000
(In a real-life config there would be several other configuration items but as they are not of interest for the question asked I simply left them out in my example)
Some additional information to complement the answer by #Cyril. If you want to upvote use his answer, not mine.
I was a little bit puzzled how easy it is to set additional connection properties that in the end get used while creating the database connection. So I did a little bit of research.
spring.datasource.connectionProperties is not mentioned in the reference. I created an issue because of this.
If I had used the Spring Boot YML editor, I would have seen which properties are supported. Here is what STS suggests when you create an application.yml and hit Ctrl+Space:
The dash does not matter because of relaxed binding but if you interpret it literally the propertys name is spring.datasource.connection-properties.
The correct setup in application.yml looks like this:
spring:
datasource:
connection-properties: defaultBatchValue=1000;defaultRowPrefetch=1000
...
This gets honored which is proven by my perf4j measurements of mass SELECTs.
Before:
2016-01-19 08:58:32.604 INFO 15108 --- [ main]
org.perf4j.TimingLogger : start[1453190311227]
time[1377] tag[get elements]
After:
2016-01-19 08:09:18.214 INFO 9152 --- [ main]
org.perf4j.TimingLogger : start[1453187358066]
time[147] tag[get elements]
The time taken to complete the SQL statement drops from 1377ms to 147, which is an enormous gain in performance.
After digging around in the Tomcat code for a bit, I found that the dataSource.getPoolProperties().getDbProperties() is the Properties object that will actually get used to generate connections for the pool.
If you use the BeanPostProcessor approach mentioned by #m-deinum, but instead use it to populate the dbProperties like so, you should be able to add the properties in a way that makes them stick and get passed to the Oracle driver.
import java.util.Properties;
import org.apache.tomcat.jdbc.pool.DataSource;
import org.apache.tomcat.jdbc.pool.PoolConfiguration;
#Component
public class OracleConfigurer implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException {
if (bean instanceof DataSource) {
DataSource dataSource = (DataSource)bean;
PoolConfiguration configuration = dataSource.getPoolProperties();
Properties properties = configuration.getDbProperties();
if (null == properties) properties = new Properties();
properties.put("defaultRowPrefetch", 1000);
properties.put("defaultBatchValue", 1000);
configuration.setDbProperties(properties);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String name) throws BeansException {
return bean;
}
}
I have a project and in it I'm using spring-boot-jdbc-starter and it automatically configures a DataSource for me.
Now I added camel-spring-boot to project and I was able to successfully create routes from Beans of type RouteBuilder.
But when I'm using sql component of camel it can not find datasource. Is there any simple way to add Spring configured datasource to CamelContext? In samples of camel project they use spring xml for datasource configuration but I'm looking for a way with java config. This is what I tried:
#Configuration
public class SqlRouteBuilder extends RouteBuilder {
#Bean
public SqlComponent sqlComponent(DataSource dataSource) {
SqlComponent sqlComponent = new SqlComponent();
sqlComponent.setDataSource(dataSource);
return sqlComponent;
}
#Override
public void configure() throws Exception {
from("sql:SELECT * FROM tasks WHERE STATUS NOT LIKE 'completed'")
.to("mock:sql");
}
}
I have to publish it because although the answer is in the commentary, you may not notice it, and in my case such a configuration was necessary to run the process.
The use of the SQL component should look like this:
from("timer://dbQueryTimer?period=10s")
.routeId("DATABASE_QUERY_TIMER_ROUTE")
.to("sql:SELECT * FROM event_queue?dataSource=#dataSource")
.process(xchg -> {
List<Map<String, Object>> row = xchg.getIn().getBody(List.class);
row.stream()
.map((x) -> {
EventQueue eventQueue = new EventQueue();
eventQueue.setId((Long)x.get("id"));
eventQueue.setData((String)x.get("data"));
return eventQueue;
}).collect(Collectors.toList());
})
.log(LoggingLevel.INFO,"******Database query executed - body:${body}******");
Note the use of ?dataSource=#dataSource. The dataSource name points to the DataSource object configured by Spring, it can be changed to another one and thus use different DataSource in different routes.
Here is the sample/example code (Java DSL). For this I used
Spring boot
H2 embedded Database
Camel
on startup spring-boot, creates table and loads data. Then camel route, runs "select" to pull the data.
Here is the code:
public void configure() throws Exception {
from("timer://timer1?period=1000")
.setBody(constant("select * from Employee"))
.to("jdbc:dataSource")
.split().simple("${body}")
.log("process row ${body}")
full example in github