How to start localhost elasticsearch node in my junit test, and use high level rest client( Elasticsearch 6.2.)
Here is the code I tried
String clusterName = "test";
Settings settings = Settings.builder()
.put("path.home", ES_WORKING_DIR)
.build();
new Node(settings).start();
and the error is.
java.lang.IllegalStateException: Unsupported transport.type []
Embedded Elasticsearch node is not supported anymore.
You can read here for more detail
Embedded Elasticsearch not supported
Btw, in case you still want to have that feature (run component integration test, for example), there're some third-parties library may support you:
https://github.com/allegro/embedded-elasticsearch
https://github.com/nitishgoyal13/elasticsearch-6-embedded-cluster
Related
I want to configure the CosmosQueryRequestOptions.maxDegreeOfParallelism while using the CosmosRepository. I didn't find any documentation around it.
This blog shows how to configure and use this setting through a custom client, but I want to use the repository instead. https://medium.com/#middha.nishant173/improve-query-performance-with-azure-cosmosdb-java-sdk-v4-db1fc54cb484
CosmosQueryRequestOptions is implementation detail for Spring Data Cosmos SDK, so customers cannot set it through spring application.
This can be implemented as a new feature, and can be exposed through application.properties via query.maxDegreeOfParallelism - which customers can opt in if they want.
Default value for maxDegreeOfParallelism is 0, which is the right value for single partition queries. For cross partition queries in the current SDK version, you can get the cosmosClient through spring boot applicationContext and run the query directly against the client. This example shows how to do it - https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-spring-data-cosmos-test/src/test/java/com/azure/spring/data/cosmos/repository/integration/PageableAddressRepositoryIT.java#L144
Currently we are on spring-data-elasticsearch 3.2.6 + elasticsearch 6.8.
We are moving to new elasticsearch 7.x. Do I have to update spring-data-elasticsearch to 4.x? We only use ElasticsearchRepository in spring-data-elasticsearch. And we don't need to use any new feature in elasticsearch 7.x.
If we are moving to elasticsearch 8.x in the future, do I need update spring-data-elasticsearch ?
Update:
What Elasticsearch client does Spring-Data-Elasticsearch use under the hood?
All methods in the `ElasticsearchRepository` are deprecated. What should do I use?
I found some discussions in above threads. Here is my summary.
Operations with Templates:
ElasticsearchTemplate implements ElasticSearchOperation. It uses TransportClient(which is deprecated in ES 7 and has been removed in ES8)
ElasticsearchRestTemplate implements ElasticSearchOperation. It uses high level client(which is deprecated in ES 7.16.0. It will be removed in future. #Deprecated(since = "7.16.0", forRemoval = true) )
ReactiveElasticsearchTemplate implements ReactiveElasticsearchOperations. It uses Reactive Client.
Repository
ElasticsearchRepository uses TransportClient as default. All methods in the ElasticsearchRepository are deprecated now.
Reactive Elasticsearch repository builds on ReactiveElasticsearchOperations.
Due to underlying TransportClient or HigLevelRestClient has been deprecated, can I conclude that the correct way is to use Reactive Client(ReactiveElasticsearchTemplate or Reactive Elasticsearch repository) ?
The new Elasticsearch would be 8.
Val already put the link to the compatibility matrix in his comment.
Version 3.2.6 is pretty outdated (March 25 2020) and out of support since October 2020.
The first thing you can try is to see if your application works with a 7 cluster - although I doubt that, I can't tell you exactly what had changed in the API, but there was quite some stuff.
What you should not do is putting newer Elasticsearch libraries on the classpath than the ones that Spring Data Elasticsearch was built with, this will in most cases make problems.
But I'd recommend to upgrade your application anyway and keep it regularly up to date.
As for future upgrade to version 8: It is possible to send a compatibility header in your requests (this can be done in Spring Data Elasticsearch 4) and the Elasticsearch cluster should respond in a format that is compatible with a client expecting version 7. I wrote should, because it does not conform to this in every case - I reported one case that is fixed now. But I wouldn't rely on that.
Again, please update your application and keep it up to date, not only because of Spring Data Elasticsearch, but also because these updates always include bug and/or security fixes.
I am using hapi fhir v5.1.0 with jpa server(hapi-fhir-jpa-server-starter). As per the description, this version contains elastic search library for text search.
How can I configure elastic server here?
I see some entries in the properties file and configured the elastic rest url, but nothing works and always get the following error - HSEARCH000222 - the search factory was not initialized.
Could someone please let me know the configuration steps required to activate elastic search inside hapi fhir jpa server?
I'm struggeling to setup hibernate seach using the elastic search backend in a spring boot setup.
What I have is spring boot and the following dependencies.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
<version>1.4.0.M3</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-search-backend-elasticsearch</artifactId>
<version>5.6.0.Alpha3</version>
</dependency>
What happens is, that hibernate search initializes before elastic search has finished starting.
Using the following property exposes the rest interface as well
spring:
data:
elasticsearch:
properties:
http:
enabled: true
Causing an exception
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect
Now, how do I define a dependency here?
I tried using a custom BeanFactoryPostProcessor to inject a dependency on elastic search, but that seems to be ignored in the auto configuration scenario.
Is there any way to introduce a wait until elastic search is up?
The setup works, when I set the hibernate index_management_strategyto NONE, but then the index is not configured and all custom analyzer annotations are ignored, defaulting to the default mappings in elastic search, which can not be configured in the auto configuration scenario.
Ideally elastic search should be hosted external to the jvm, but it's convenient in testing scenarios.
I'm understanding this is an issue you're hitting during integration tests.
You could have a look at how we start ES during the integration tests within Hibernate Search itself, using a Maven plugin which makes sure the server is started before the tests:
- https://github.com/hibernate/hibernate-search/blob/5.6.0.Beta1/elasticsearch/pom.xml#L341-L368
N.B. this uses a custom ES configuration, tuned to start quickly even though it's only a single node cluster:
- https://raw.githubusercontent.com/hibernate/hibernate-search/5.6.0.Beta1/elasticsearch/elasticsearchconfiguration/elasticsearch.yml
Hibernate Search uses the Jest client to connect to ES, so it will require you to enable the HTTP connector of ES: let's not confuse this with a NodeClient, which is a different operating mode.
If your question isn't related to automated testing but rather production clusters, then I'd suggest using a Service Orchestrator like Kubernetes.
Thanks to some help from the spring boot team, I was able to solve the issue - solution here.
The problem is that there's no dependency between the EntityManagerFactory bean and the Elasticsearch Client bean so there's no guarantee that Elasticsearch will start before Hibernate. As it happens, Hibernate starts first and then fails to connect to Elasticsearch.
This can be fixed by setting up a dependency between the two beans. An easy way to do that is with a subclass of EntityManagerFactoryDependsOnPostProcessor:
#Configuration
static class ElasticsearchJpaDependencyConfiguration extends EntityManagerFactoryDependsOnPostProcessor {
public ElasticsearchJpaDependencyConfiguration() {
super("elasticsearchClient");
}
}
Now all that is needed is to set the number of replicas to 0 to fix the health status of the cluster in the single node deployment. This can be done by specifying an additional property in the application.properties file
spring.data.elasticsearch.properties.index.number_of_replicas= 0
I checked sprint-data docs and looks like you misunderstood this piece (and actually it's confusing, guys don't understand the tech underneath?)
By default the instance will attempt to connect to a local in-memory server (a NodeClient in Elasticsearch terms), but you can switch to a remote server (i.e. a TransportClient) by setting spring.data.elasticsearch.cluster-nodes to a comma-separated ‘host:port’ list.
NodeClient is not "local server", it's special type of ES client. This local client can connect to ES cluster nodes containing data, and as I said in the comment, you don't have any ES data nodes running.
Read this for better understanding https://www.elastic.co/guide/en/elasticsearch/guide/current/_transport_client_versus_node_client.html
I am trying to integrate camel with elasticsearch.
In applicationContext.xml added the following
<route id="timer-to-console">
<from uri="timer://foo?fixedRate=true&period=10s"/>
<transform>
<simple>Hello Web Application, how are you?</simple>
</transform>
<to uri="stream:out"/>
<to uri="elasticsearch://local"/>
</route>
Then when I run
mvn jetty:run
I am getting the following
veryCounter=0, firedTime=Mon Apr 21 13:14:43 PDT 2014}
BodyType String
Body Hello Web Application, how are you?
]
Stacktrace
----------------------------------------------------------------------------------------
java.lang.IllegalArgumentException: operation is missing
at org.apache.camel.component.elasticsearch.ElasticsearchProducer.process(ElasticsearchProducer.java:54)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
My elasticsearch is running locally, I am using ES 1.1.1.
what do I need to specify for
elasticsearch://clusterName[?options]
Thanks,
From a quick glance at the Apache Camel Elasticsearch Component page they show the following example:
elasticsearch://local?operation=INDEX&indexName=twitter&indexType=tweet
This would INDEX (add) into an index named twitter with a type of tweet. You can use whatever values you want for the indexName and indexType.
Update: Looking at the Elasticsearch Camel Component documentation again... I think in order to use local as the server name in the elasticsearch connection, you would need to be running your local Elasticsearch instance with a clustername of local. By default Elasticsearch configuration (elasticsearch.yml) is setup to run with a clustername of elasticsearch
You have to use the cluster name specified in elasticsearch.yml, i.e. if it says cluster.name: elasticsearch123 then you have to route for example as follows:
from("file:data_json?noop=true&moveFailed=data_json/.error")
.convertBodyTo(byte[].class)
.to("elasticsearch://elasticsearch123?operation=INDEX&indexName=myindexname&indexType=mytypename");
Using local simply doesn't work for me. Unfortunately it doesn't even throw any error which makes it difficult to debug.
Also notice the conversion .convertBodyTo(byte[].class) - that's also essential, otherwise Camel converts the JSON files to the class Properties (it looks for a conversion from GenericFile to Map and it finds a fallback type converter to Properties). Of course, you could also convert to the other types supported by the component.
It's also worth mentioning that the camel-elasticsearch component of version 2.14.1 depends on org.elasticsearch:elasticsearch:1.0.0 which is quite an old version given how fast Elasticsearch moves (the current version is 1.4.4) and how often it likes to break compatibility. That said, the component seems to work with Elasticsearch 1.4.3.
Update: the current master branch on GitHub has Elasticsearch upgraded to version 1.4.2: https://github.com/apache/camel/blob/2470023e25b8814279cbadb3ebc8002bed6d8fc8/parent/pom.xml#L144
The parameter/clusterName local in fact means that it will look for an Elasticsearch instance started in the same JVM, from Elasticsearch's JavaDoc (NodeBuilder.local(boolean)):
A local node is a node that uses a local (JVM level) discovery and
transport. Other (local) nodes started within the same JVM (actually, class-loader) will be
discovered and communicated with. Nodes outside of the JVM will not be discovered.