In my test environment, I am running tomcat and I do hot deployment. This works most of the time, but there are rare occasion wherein the connection between tomcat and Elasticsearch gets lost. In my other application with tomcat and Elasticsearch (different version of Elasticsearch), I still encounter the same issue.
Usually, I just restart tomcat and it will establish the connection with Elasticsearch again. So in my other application, I stop tomcat before deploying the war file, then start it again. Since then, I am not getting this issue (lost ES connection) for that project. However, in the project that still uses hot deployment.. I am still getting this weird issue. If I do stop/deploy/start Tomcat, it will not have that issue, but I want this project to use hot deployment. I am thinking that tomcat should reestablish a connection with Elasticsearch, but this is not the case.
Anyone experienced this issue? and how did you manage to fix this hot deployment issue with Elasticsearch? By the way, I use spring data Elasticsearch and rest high level client Elasticsearch on different projects and both have the same issue on hot deploy (rarely happens though).
Related
Everything was working until I introduced Webflux into our Sprint Boot application running on external tomcat inside a docker container. This was needed when I wrote an elasticsearch repository using ReactiveCrudRepository to implement asynchronous save/fetch operations. Now, our Spring Boot apps hosted on docker are not able to connect to any of the datastores.
For Redis the exception is -
Caused by: java.net.UnknownHostException: Failed to resolve
'redis1' after 2 queries
For Elasticsearch -
Caused by:
org.springframework.data.elasticsearch.client.NoReachableHostException:
Host 'es_host:9200' not reachable. Cluster state is offline.
The data nodes are all hosted on separate instances.
On researching around a bit, I came across several issues with netty's DNS resolver, but there is no standard fix mentioned anywhere. Please advice.
I completed the code upgrade and removed all deprecated calls from my service code.
This is a Java Spring application and all services come up (as reported in the log files) when the Cluster (of 5 nodes) starts up.
However, when I try to get a serviceProxy to each service, all services get cancelled as a result of ServiceDeploymentTask attempting to do a redeploy of the services!! The redeploy fails and only cancels all of the services and fails to restart them. This can be demonstrated with both a thick and a thin client.
Why is Ignite trying to redeploy the services? (and why don't the services restart?)
Is there something I'm missing from the move to Ignite 2.10???
Finally, why does a Java Thin Client create a NODE_JOIN event?
Thanks in advance.
Greg
My spring-boot (version 2.4.1) application was successfully connected to an ElasticSearch(v7.9.3) instance using the autowired org.elasticsearch.client.RestHighLevelClient (I just had to specify the application properties and it worked).
In a new phase of the project a dependency with spring-boot-starter-webflux was added to use some reactive logic to call an external webservice. (which has nothing to do with my elasticsearch connection)
But now suddenly the elasticsearch client also tries to connect using reactor and I got errors like:
reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.data.elasticsearch.client.NoReachableHostException:
Host 'https://elastic-dev.company.intra:9200:9200' not reachable. Cluster state is offline.
Caused by: org.springframework.data.elasticsearch.client.NoReachableHostException:
Host 'https://elastic-dev.company.intra:9200:9200' not reachable. Cluster state is offline.
at org.springframework.data.elasticsearch.client.reactive.SingleNodeHostProvider.lambda$lookupActiveHost$4(SingleNodeHostProvider.java:108) ~[spring-data-elasticsearch-4.1.2.jar!/:4.1.2]
I know there is a configuration issue with :9200:9200 but I would like to just disable the use of reactor for my Elasticsearch client so it just uses the old way (I still need my Elasticseach client). Is this possible ?
Thanks.
After searching further I found a solution which was also suggested by P.J.Meish: disable the AutoConfiguration classes regarding the reactive elasticsearch:
I preferred the config in application.properties:
spring.autoconfigure.exclude=\
org.springframework.boot.actuate.autoconfigure.elasticsearch.ElasticSearchReactiveHealthContributorAutoConfiguration,\
org.springframework.boot.autoconfigure.data.elasticsearch.ReactiveElasticsearchRepositoriesAutoConfiguration,\
org.springframework.boot.autoconfigure.data.elasticsearch.ReactiveElasticsearchRestClientAutoConfiguration
I have two docker instances that I launch with docker-compose.
One holds a Cassandra instance
One holds a Spring Boot application that tries to connect to that application.
However, the Spring Boot application will always fail, because it's trying to connect to a Cassandra instance that is not ready yet to take connections.
I have tried:
Using restart:always in Docker-compose
This still doesn't always work, because the Cassandra might be up 'enough' to no longer crash the Spring Boot application, but not up 'enough' to have successfully created the Table/Column family. On top of that, this is a very hacky solution.
Using healthcheck
It seems like healthcheck in compose doesn't have restart capabilities
Using a bash script as entrypoint
In the hope that I could use netstat,ping,... whatever to determine that readiness state of Cassandra
Right now the only thing that really works is using that same bash script and sleep the process for x seconds, then start the jar. This is even more hacky...
Does anyone have an idea on how to solve this?
Thanks!
Does the spring boot service defined in the docker-compose.yml depends_on the cassandara service? If yes then the service is started only if the cassandra service is ready.
https://docs.docker.com/compose/compose-file/#depends_on
Take a look at this github repository, to find a healthcheck for the cassandra service.
https://github.com/docker-library/healthcheck
CONCLUSION
After some discussion we found out that docker-compose seems not to provide a functionality for waiting until services are up and healthy, such as Kubernetes and Openshift provide (See comments below). They recommend to use wrapper script (docker-entrypoint.sh) which waits for the depending service to come up, which make binaries necessary, the actual service shouldn't use such as the cassandra client binary. Additionally the service depending on cassandra could never get up if cassandra doesn't, which shouldn't happen.
A main thing with microservices is that they have to be resilient for failures and are not supposed to die or not to come up if a depending service is currently not available or unexpectedly disappears. Therefore the microservice should be implemented in a way so that it retries to get connection after startup or an unexpected disappearance. Unexpected is a word actually wrongly used in this context, because you should always expect such issues in a distributed environment, and even with docker-compose you will face issues like that as discussed in this topic.
The following link points to a tutorial which helped to integrate cassandra properly into a spring boot application. It provides a way to implement the retrieval of a cassandra connection with a retry behavior, therefore the service is resilient to a non existing cassandra database and will not fail to start anymore. Hope this helps others as well.
https://dzone.com/articles/containerising-a-spring-data-cassandra-application
I have a very rare problem it is a java web application with jsf 2.0, spring 3.0.3 and jpa2 eclipselink implementation running on tomcat 7 when deploying the application to server it run correctly without any errors and after random period of time it return Access Denied for user "user#host" and all i need to do is just restarting tomcat to work again.
Please help if you know anything about this problem.
Thanks,
Are you leaking connections?
It sounds like you're using up all the connections and MySQL won't let you create new ones.