In my Spring Boot 1.5.10.Final project I use Hibernate Search ORM 5.6.4.Final. It works fine except of the integration tests. There is one test class with several
test methods to test the search logic. If I run just this test class every things works fine. Spring Boot is starting and creates the index. If I run this test class
together with all other integration tests, every test class will throw an LockObtainFailedException and the Hibernate Search tests will fail.
org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: ...LieferantEntity\write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:776) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:126) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:92) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriterDelegate(AbstractWorkspaceImpl.java:203) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:81) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:46) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.applyChangesets(SyncWorkProcessor.java:165) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.run(SyncWorkProcessor.java:151) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
I use the default settings. If I set exclusive_index_use to false, then it works without failure but then the test execution is very slow.
For me it seems the index is initialized during the startup of Spring Boot and interferes between the tests.
Is it possible to use Spring Boot integration tests with Hibernate Search in a way so that locks are cleanly released between tests?
Alternativly I'm looking for a way to disable the Hibernate Search indexing for all integration tests not making use of Hibernate Search
I also tried already the property near-real-time and different lock factories as native, simple and single without luck.
First: do not use exclusive_index_use unless you are a Lucene guru. It is dangerous and probably will not behave as you want.
Now that we got that out of the way... As far as I understand, you are trying to execute integration tests in parallel on the same machine. This means integration tests will probably compete for access to the exact same index, and will write to the same index. This could lead to unpredictable results if your tests perform conflicting writes (one test erasing a document added by another test, before that test has completed).
If you really need to perform tests in parallel, I would recommend to execute each test in an isolated environment:
Dedicated DB, or at least dedicated DB schema
Dedicated Lucene index
etc.
In the case of Hibernate Search, you will have to find a way to use a different physical index in each test execution.
There are two ways to do that:
Just for tests, do not store the indexes on the filesystem, but directly in the heap, by setting hibernate.search.backend.directory.type to local-heap (Hibernate Search 6+) or hibernate.search.default.directory_provider to local-heap (Hibernate Search 5 and below).
It's super easy to implement, but there are a few disadvantages you should be aware of:
your testing environment will not be exactly the same as your production environment anymore
indexes will be lost after the tests have finished executing, which may make post-mortem debugging challenging (you won't be able to use Luke to inspect the state of the indexes anymore)
if your integration tests store a lot of content in the index, you might get an OutOfMemoryError.
If the disadvantages of solution 1 are too much for you, you can continue using the filesystem to store the indexes, but use a different configuration for each test execution, setting the index base path (hibernate.search.backend.directory.root for Hibernate Search 6+, or hibernate.search.default.indexBase for Hibernate Search 5 and below) to some unique path for each test execution. You will have to find how to do that in Spring, but I would be suprised to learn it's not possible. Maybe Spring allows you to use interpolation in the properties, something like hibernate.search.backend.directory.root = /tmp/it/#{testName}?
See the documentation about directory configuration (here for Hibernate Search 6+, or here for Hibernate Search 5 for more information on how to configure index storage.
Related
Since I updated to Spring boot 2.5 my application context won't start in the Test environment.
We have several test environments. Most tests do not need an Elastic search instance. Those that need it share an Elasticsearch test container instance.
Since the Update the creation of repositories causes some kind of query to Elasticsearch. That fails and causes the context not to load.
Is there a way to mock away the Spring Data Elasticsearch part(Not loading is not really an option to load most parts of the context)?
Should I be starting an Elasticsearch Instance for all integration tests(that seems like a little overkill, since few tests actually need it)?
Any ideas are highly appreciated.
I am trying to improve performance of medium tests in Spring Boot.
I am using the Spring Boot - testcontainers library.
For an individual test this works really well, with a few annotations I can get access to kafka, zookeeper, and schema-registry. These are full services so it takes a few seconds to start everything up, all together setup takes about 40 seconds. The test accurately recreates a realistic deployment, it's beautifully simple.
This would be fine if it just happened once but it happens every time a Spring Context is created. That means every test that uses #MockBean incurs that 40 second cost.
I've tried refactoring into a single TestConfiguration class and referencing that. I've looked into using ContextHierarchy but I think that means I'll lose all of the Spring Boot niceties and I'll need to recreate the context (which means it won't look exactly like the context created by the production app).
Is there a better way to do this?
Spring framework already took care of this scenario.
There is a concept of caching the application context for test class/classes.
See the documentation.
Few lines from the documentation:
The Spring TestContext framework stores application contexts in a
static cache. This means that the context is literally stored in a
static variable. In other words, if tests run in separate processes,
the static cache is cleared between each test execution, which
effectively disables the caching mechanism.
So essentially you need to structure your code or context configuration in such a way that you use cached context in your desired test cases.
But use this capability wisely, if not thought through properly this could lead to undesired side-effects
I am working on a sample application right now using Spring Boot, Spring Data JPA, and Spring Data Elasticsearch. I want to be able to run the unit tests as part of a pipeline build, but they require Elasticsearch to be running to work as the service makes calls to said ES server. SQL works fine because I am using an in-memory H2 instance.
I have implemented some code to attempt to launch ES as an "embedded" server. The embedded server works just fine, but it seems like, at least from what I can tell, it is started AFTER the context loads. Most importantly after the ElasticSearchConfiguration does it's thing.
I think I need refactor the code out of AbstractElasticsearchTest into a separate class that can run prior to ElasticSearchConfiguration generates the client/template, but I am not sure how to do it, nor how to Google said process.
Is there some mechanism in Spring Boot that could be used to start the embedded servers prior to running any of the configurations? Or is there some way I could enhance ElasticSearchConfiguration to do it prior to creating the client/template, but only when running the unit tests?
Edit:
So, just to be a little more specific...what I am looking for is a means/way to either run ES 5 in "embedded" mode OR how to mock up the Spring Data ES code enough so that it works for the CI server. The code linked above currently is mixing unit tests with integration tests, I know, as it's currently making calls to a physical ES server. That's what I am trying to correct: I should be able to stub/mock enough of the underlying Spring Data code to make the unit test think it's talking to the real-deal. I can then change the tests that determine if the documents made it to ES and test things like type-ahead searches to be integration tests instead so they do not run when CI or Sonar runs.
Ok, so for those that might come back here in the future, this commit shows the changes I made to get ES to run as "embedded".
The nuts-and-bolts of it was to start the node as "local" then physically return node.client(). Then in the Spring Bean method that gets the client, check if you have "embedded" turned on, if so start the node and return it's Client (the local one), if not just build the client just as normal.
I am new to golang and have two little projects started, where I got to the same problem.
One uses a MySQL database as a datastore and the other elasticsearch. How can I write tests which cover the code for that.
At first I’ve used a HashMapDatastore, but with that I can only test my handlers and business logic, but not the persistence layer (e.g. if the persisting and retrieving works as I expect in elasticsearch) (aka integration test).
So should I test if there is a blank database for testing?
How should I delete this db after each test? I’ve found an approach for this in the integration tests of docker/libcompose but it is relatively complex.
Is there a common approach for that? Testing if there is an elasticsearch with an empty index available and if not failing the tests with a good setupmessage?
Thanks
I have a large spring project, using xml configuration. I'm looking for a quick way to verify changes to the xml configuration.
I can load the whole project locally - the problem is this takes more than 5 minutes, loads a huge amount of data.
My XML editor catches XML formatting errors.
I'm looking for something intermediate - to catch obvious problems like references to beans that aren't defined, or calling constructors with the wrong arguments. Is there a quick way to do this, without having to actually invoke all the constructors and bring up the whole environment?
I'm building with Maven and editing with Eclipse, although my question isn't specific to either.
Since you already use Eclipse, you could try Spring Tool Suite (comes either standalone or as an add-on). It's essentially Eclipse with extra Spring-specific features, like Beans Validator. I'm not sure how thorough the validation is, but it should catch most configuration problems.
It's maintained by SpringSource so its integration with Spring "just works" and it's guaranteed not be more or less in sync with Spring Framework's release cycle.
Beanoh :
http://beanoh.org/overview.html#Verify
this project does exactly what I'm looking for. Verify obvious problems with spring config, but without the overhead of initializing everything.
You can use a Spring testing support to integration test your Spring configuration. However if the loading of the context is taking 5 mins, then the tests will also take the same amount of time. Spring does cache the context so if you have multiple tests using the same set of Spring contexts, then once cached the tests should be very quick.
I can suggest a few ways to more efficiently test your configuration:
Organize your project in modules, with each module being responsible for its own Spring configuration - this way, each module can be independently developed and tested.
If you have a modular structure, the testing can be more localized by mocking out the dependent modules, again this is for speed.