How to fix this issue?
Spring boot version:2.4.5
I am using gradle and dependencies used related to elastic search are:
implementation 'org.springframework.boot:spring-boot-starter-data-elasticsearch'
&
implementation 'co.elastic.clients:elasticsearch-java:7.17.3'
2023-02-10 20:51:45.788 INFO 1 --- [main] o.s.d.elasticsearch.support.VersionInfo : Version Spring Data Elasticsearch: 4.1.8
2023-02-10 20:51:45.793 INFO 1 --- [main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch Client in build: 7.9.3
2023-02-10 20:51:45.794 INFO 1 --- [main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch Client used: 7.9.3
2023-02-10 20:51:45.795 INFO 1 --- [main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch cluster: 8.6.1
2023-02-10 20:51:45.795 WARN 1 --- [main] o.s.d.elasticsearch.support.VersionInfo : Version mismatch in between Elasticsearch Client and Cluster: 7.9.3 - 8.6.1
The issue is a warning because the major version between the Elasticsearch client that is used and the Elasticsearch server do not match. Spring Data Elasticsearch was not tested and released against ES 8. So anything might happen - no guarantees.
How to solve this? Upgrade your application to Spring Boot 3, that uses Spring Data Elasticsearch 5.0, which in turn uses Elasticsearch 8.6.
Spring Data Elasticsearch 4.1 is out of maintenance since May 2021.
Related
I added the sleuth to a microservice and successfully get the Trace Id and span Id, but as you can see after sleuth, missing a part of the info before the log level in the log line.
I think maybe I can disable sleuth in a debug level, to have log line as before and then enable it in the info level.
do you know how or maybe a better solution?
before Sleuth:
2022-03-24 09:53:22.338 [(ID:9dcebfa9-b37f-4a94-a91b-45867ea90811)(CUSTOMER ID:*** TYPE:UNKNOWN)] DEBUG 3180716 --- [https-jsse-nio-48082-exec-4] c.h.d.s.staticdata.StaticDataService : start with version: 1.2 OS: ANDROID
after Sleuth:
2022-03-24 10:19:34.541 DEBUG [*****,efcbdb6dc513e468,efcbdb6dc513e468,true] 3368114 --- [https-jsse-nio-48082-exec-7] c.h.d.s.staticdata.StaticDataService : start with version: 1.2 OS: ios
I just migrated to a new spring boot version lately, namely v2.6.2 and now I find the following exception of some of the services logs.
23:27:45.148 [main] INFO o.s.b.w.e.tomcat.TomcatWebServer - Tomcat started on port(s): 8080 (https) with context path ''
23:27:47.046 [main] INFO o.a.c.i.engine.AbstractCamelContext - Routes startup (total:1 started:1)
23:27:47.048 [main] INFO o.a.c.i.engine.AbstractCamelContext - Started route1 (jms://queue:inp.contentextractor.entry)
23:27:47.049 [main] INFO o.a.c.i.engine.AbstractCamelContext - Apache Camel 3.14.0 (camel-1) started in 2s228ms (build:622ms init:1s32ms start:574ms)
23:27:47.061 [main] INFO eu.hermes.esb.cloud.Application - Started Application in 25.305 seconds (JVM running for 27.737)
23:28:20.356 [https-jsse-nio-8080-exec-4] INFO o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
23:28:20.357 [https-jsse-nio-8080-exec-4] INFO o.s.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
23:28:20.359 [https-jsse-nio-8080-exec-4] INFO o.s.web.servlet.DispatcherServlet - Completed initialization in 2 ms
00:05:19.986 [https-jsse-nio-8080-Acceptor] ERROR org.apache.tomcat.util.net.Acceptor - Socket accept failed
java.io.IOException: Duplicate accept detected. This is a known OS bug. Please consider reporting that you are affected: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1924298
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:545)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:78)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129)
at java.base/java.lang.Thread.run(Thread.java:834)
05:20:49.981 [https-jsse-nio-8080-Acceptor] ERROR org.apache.tomcat.util.net.Acceptor - Socket accept failed
java.io.IOException: Duplicate accept detected. This is a known OS bug. Please consider reporting that you are affected: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1924298
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:545)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:78)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129)
at java.base/java.lang.Thread.run(Thread.java:834)
What does it mean and how harmful is it ?
Following the link provided in the bug (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1924298), and looking into comments made by community members, it seems that there is something in tomcat usage that uncovered a hidden issue in linux kernal 5.10-rc4 and was most likely fixed in 5.10-rc6.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1924298/comments/15
This issue was reported across centOS, amazon linux, Arch linux and even Window 2019 - when using a lower linux kernel than 5.10.
The conclusion from this discussion seems to be the suggestion to update the OS itself to a stable version that uses linux kernel >= 5.10
I have a Spring Boot application which I'm building and running with Java 10. If I run the app using
java -jar
Everything works fine. The app starts just OK.
But if I put my app inside a Docker container with the exactly same Java version, my app throws this exception:
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jgroups.logging.Slf4jLogImpl
at org.jgroups.logging.LogFactory.getLog(LogFactory.java:101)
at org.jgroups.conf.XmlConfigurator.<clinit>(XmlConfigurator.java:33)
at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:62)
at org.jgroups.JChannel.<init>(JChannel.java:122)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.buildChannel(JGroupsTransport.java:591)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.initChannel(JGroupsTransport.java:405)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:389)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
... 104 common frames omitted
I'm using this version of Java:
java version "10.0.2" 2018-07-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.2+13)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode)
Docker version is:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:21:31 2018
OS/Arch: darwin/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:29:02 2018
OS/Arch: linux/amd64
Experimental: true
My Docker is using an Alpine base image alpine:latest. I'm installing java in my container from this link:
curl -jksSLH "Cookie: oraclelicense=accept-securebackup-cookie" -o /tmp/java.tar.gz \
http://download.oracle.com/otn-pub/java/jdk/10.0.2+13/19aef61b38124481863b1413dce1855f/jdk-10.0.2_linux-x64_bin.tar.gz
I'm really confused because from outside a docker container my app works fine, but inside a docker container it doesn't. Either case I'm using the same Java version.
UPDATE
We tried Oracle JDK and OpenJDK, same behavior
UPDATE 2
We even tried java -jar from inside the container, no luck
TL;DR
There are three options to resolve.
Upgrade to version 4.0.16 of JGroups, currently in SNAPSHOT. Edit: Now released here.
Make sure java properties for "user.language" and "user.country" are set.
Force JDKLogImpl with -Djgroups.use.jdk_logger=true. (mentioned by Perimosh)
Explanation
Ran into this issue in the following scenario.
Apache Camel + JGroups worked fine in a local environment. Deployed it elsewhere in a Docker instance, where we got the following stacktrace:
2018-11-19 13:38:03.063 INFO 582 --- [ main] o.a.camel.spring.boot.RoutesCollector : Loading additional Camel XML routes from: classpath:camel/*.xml
2018-11-19 13:38:03.064 INFO 582 --- [ main] o.a.camel.spring.boot.RoutesCollector : Loading additional Camel XML rests from: classpath:camel-rest/*.xml
2018-11-19 13:38:03.107 INFO 582 --- [ main] o.a.camel.spring.SpringCamelContext : Apache Camel 2.22.2 (CamelContext: camel-1) is starting
2018-11-19 13:38:03.111 INFO 582 --- [ main] o.a.c.m.ManagedManagementStrategy : JMX is enabled
2018-11-19 13:38:03.480 INFO 582 --- [ main] o.a.camel.spring.SpringCamelContext : StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2018-11-19 13:38:03.597 INFO 582 --- [ main] o.a.camel.spring.SpringCamelContext : Apache Camel 2.22.2 (CamelContext: camel-1) is shutting down
2018-11-19 13:38:03.616 WARN 582 --- [ main] o.a.camel.spring.SpringCamelContext : Error occurred while shutting down service: org.apache.camel.component.jgroups.cluster. JGroupsLockClusterService#10fa5af5. This exception will be ignored.
java.lang.NullPointerException: null
at org.apache.camel.component.jgroups.cluster.JGroupsLockClusterView.doStop(JGroupsLockClusterView.java:109)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at org.apache.camel.impl.cluster.AbstractCamelClusterService.lambda$doStop$2(AbstractCamelClusterService.java:134)
at org.apache.camel.util.concurrent.LockHelper.doWithReadLockT(LockHelper.java:54)
at org.apache.camel.impl.cluster.AbstractCamelClusterService.doStop(AbstractCamelClusterService.java:130)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:142)
at org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:205)
at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:3663)
at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:3688)
at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:3676)
at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:3567)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at org.apache.camel.impl.DefaultCamelContext.stop(DefaultCamelContext.java:3220)
at org.apache.camel.spring.SpringCamelContext.stop(SpringCamelContext.java:148)
...
2018-11-19 13:38:03.679 INFO 582 --- [ main] o.a.camel.spring.SpringCamelContext : Apache Camel 2.22.2 (CamelContext: camel-1) uptime 0.570 seconds
2018-11-19 13:38:03.680 INFO 582 --- [ main] o.a.camel.spring.SpringCamelContext : Apache Camel 2.22.2 (CamelContext: camel-1) is shutdown in 0.082 seconds
2018-11-19 13:38:03.716 INFO 582 --- [ main] org.mongodb.driver.connection : Closed connection [connectionId{localValue:2, serverValue:2}] to localhost:43115 because the pool has been closed.
As you can see, Apache Camel attempts to start, but never does and ends up shutting down. Thus, JGroups gets a NPE because it expects Camel to be up.
After debugging the code, it appeared that there was an exception being thrown during the Camel start up process which was getting eaten.
From there, discovered that the creation of an instance of Slf4jLogImpl in org.jgroups.logging.LogFactory#getLog(java.lang.Class<?>) (new Slf4jLogImpl(clazz)) was a problem Method threw 'java.lang.ExceptionInInitializerError' exception.:
java.lang.NullPointerException: null
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.jgroups.logging.LogFactory.getLog(LogFactory.java:101)
at org.jgroups.conf.XmlConfigurator.<clinit>(XmlConfigurator.java:33)
at org.jgroups.conf.ConfiguratorFactory.getXmlConfigurator(ConfiguratorFactory.java:210)
at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:91)
at org.jgroups.JChannel.<init>(JChannel.java:130)
...
Running (new Slf4jLogImpl(clazz)) the second time onward in the debugger results in the following stacktrace that mirrors the original posted issue:
java.lang.NoClassDefFoundError: Could not initialize class org.jgroups.logging.Slf4jLogImpl
at org.jgroups.logging.LogFactory.getLog(LogFactory.java:101)
at org.jgroups.conf.XmlConfigurator.<clinit>(XmlConfigurator.java:33)
at org.jgroups.conf.ConfiguratorFactory.getXmlConfigurator(ConfiguratorFactory.java:210)
at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:91)
at org.jgroups.JChannel.<init>(JChannel.java:130)
This difference in results is due to the class loader caching the result of the Class.forName() call previously to determine that the class definition was not found.
Finally, we tracked the previous NPE to be thrown from java.util.Locale#Locale(java.lang.String, java.lang.String, java.lang.String), since country was null. This is because JGroup's org.jgroups.logging.Slf4jLogImpl is definine a LOCALE field using java properties for "user.language" and "user.country". The former was not set in our docker instance, so Locale.java threw the NPE. Adding both of these java properties should fix this issue. Alternatively you can force using the JDKLogImpl so that the Slf4jLogImpl is never attempted to be instantiated. This was mentioned in the previous answer, by passing in -Djgroups.use.jdk_logger=true.
Edit: Fixed in the latest version released here.
Now, it looks like this will be fixed in the upcoming JGroup release of 4.0.16.Final (https://github.com/belaban/JGroups/commit/61578c657138f02178c32a564ac9eae7c3976093#diff-93eb0f6a8a4953312098be459bd7ce76). Until then, you can get the snapshot version with the fix at https://repository.jboss.org/nexus/content/repositories/snapshots/org/jgroups/jgroups/4.0.16-SNAPSHOT/.
This is not a real solution, but since it unblocked us, I will share it. Also, maybe somebody can think about the real problem by looking at this workaround. We added this JVM argument to by pass SLF4J for jgroups and use JDKLogImpl
-Djgroups.use.jdk_logger=true
I have installed Neo4j 2.3.2 Community Edition on Mac OS 10.10. I can launch the application and connect to it from localhost:7474/browser/. So far, so good.
I would like to launch Neo4j 2.3.2 from a Terminal window, so that I don't have the overhead of a windowed application running at the same time. When I run the following command...
$ ~/neo4j/bin/neo4j console
... I get this output in the Terminal window:
WARNING: Max 256 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Starting Neo4j Server console-mode...
Unable to find any JVMs matching version "1.7".
Using additional JVM arguments: -server -XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -XX:hashCode=5 -Dneo4j.ext.udc.source=tarball
2016-02-25 14:03:18.755+0000 INFO [API] Setting startup timeout to: 120000ms based on 120000
2016-02-25 14:03:58.356+0000 INFO [API] Successfully started database
2016-02-25 14:04:04.220+0000 INFO [API] Starting HTTP on port :7474 with 2 threads available
2016-02-25 14:04:13.512+0000 INFO [API] Enabling HTTPS on port :7473
09:04:20.201 [main] INFO org.eclipse.jetty.util.log - Logging initialized #98517ms
2016-02-25 14:04:23.034+0000 INFO [API] Mounting static content at [/webadmin] from [webadmin-html]
2016-02-25 14:04:25.785+0000 INFO [API] Mounting static content at [/browser] from [browser]
09:04:25.993 [main] INFO org.eclipse.jetty.server.Server - jetty-9.2.4.v20141103
09:04:26.722 [main] INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.h.MovedContextHandler#1611ba2{/,null,AVAILABLE}
09:04:27.794 [main] INFO o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /webadmin, did not find org.apache.jasper.servlet.JspServlet
09:04:27.981 [main] INFO o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext#132ea25{/webadmin,jar:file:/Users/james/neo4j/system/lib/neo4j-server-2.2.5-static-web.jar!/webadmin-html,AVAILABLE}
09:04:38.841 [main] INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler#60bfaa02{/db/manage,null,AVAILABLE}
09:04:39.326 [main] INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler#28e2e149{/db/data,null,AVAILABLE}
09:04:39.353 [main] INFO o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /browser, did not find org.apache.jasper.servlet.JspServlet
09:04:39.355 [main] INFO o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext#78e6aa71{/browser,jar:file:/Users/james/neo4j/system/lib/neo4j-browser-2.2.5.jar!/browser,AVAILABLE}
09:04:39.536 [main] INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler#4994d9ab{/,null,AVAILABLE}
09:04:39.745 [main] INFO o.e.jetty.server.ServerConnector - Started ServerConnector#2d19cf20{HTTP/1.1}{localhost:7474}
09:04:40.576 [main] INFO o.e.jetty.server.ServerConnector - Started ServerConnector#43c742c{SSL-HTTP/1.1}{localhost:7473}
09:04:40.577 [main] INFO org.eclipse.jetty.server.Server - Started #119058ms
2016-02-25 14:04:40.577+0000 INFO [API] Server started on: http://localhost:7474/
2016-02-25 14:04:40.590+0000 INFO [API] Remote interface ready and available at [http://localhost:7474/]
I have Java version 8, update 74 installed (build 1.8.0_74-b02), so I assume that I can ignore the warning Unable to find any JVMs matching version "1.7".
However, when I visit http://localhost:7474/ in Chrome Version 45.0.2454.85 (64-bit), I see three errors in the Developer Console: two files that fail to load and a subsequent script error.
localhost/:28 GET http://localhost:7474/browser/styles/68eddd94.main.css
localhost/:466 GET http://localhost:7474/browser/scripts/ded362b3.scripts.js
Uncaught Error: [$injector:modulerr] Failed to instantiate module neo4jApp due to:
Error: [$injector:nomod] Module 'neo4jApp' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.
As a result, the Neo4j interface does not appear in the browser window.
Is it possible to run Neo4j 2.3.2 from the Terminal, and if so, what do I need to do to get http://localhost:7474/ to load correctly?
Shift-reload, or test in an incognito window.
Looks like a JS file mismatch due to aggressive browser caching.
We are using spring ehcache in our application, recently we upgraded the java version of the tomcat in which the app runs to 1.8.66.
Strangely there are no errors in the logs - but caching doesn't work on the linux machine.
On my local (windows) same version of java 1.8.66, it works like a charm... what could be the issue?
We are using aspecJ load time weaving with TomcatInstrumentableClassLoader - version 3.2.3.
Details of dependencies:
ehcache 2.7.4
aspectJ weaver 1.6.2
spring 3.1.1
INFO 2015-12-28 14:09:19,667 (org.springframework.cache.ehcache.EhCacheManagerFactoryBean.afterPropertiesSet(EhCacheManagerFactoryBean.java:100)) - Initializing EHCache CacheManager
INFO 2015-12-28 14:09:19,828 (net.sf.ehcache.pool.sizeof.filter.AnnotationSizeOfFilter.<clinit>(AnnotationSizeOfFilter.java:53)) - Using regular expression provided through VM argument net.sf.ehcache.pool.sizeof.ignore.pattern for IgnoreSizeOf annotation : ^.*cache\..*IgnoreSizeOf$
INFO 2015-12-28 14:09:19,873 (net.sf.ehcache.pool.sizeof.JvmInformation.<clinit>(JvmInformation.java:446)) - Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed OOPs
INFO 2015-12-28 14:09:20,190 (net.sf.ehcache.pool.sizeof.AgentLoader.getAgentFile(AgentLoader.java:198)) - Extracted agent jar to temporary file /opt/projects/tomcat/instance_7/temp/ehcache-sizeof-agent7845091970407661363.jar
INFO 2015-12-28 14:09:20,190 (net.sf.ehcache.pool.sizeof.AgentLoader.loadAgent(AgentLoader.java:138)) - Trying to load agent # /opt/projects/tomcat/instance_7/temp/ehcache-sizeof-agent7845091970407661363.jar
INFO 2015-12-28 14:09:20,199 (net.sf.ehcache.pool.impl.DefaultSizeOfEngine.<init>(DefaultSizeOfEngine.java:111)) - using Agent sizeof engine
INFO 2015-12-28 14:09:20,456 (net.sf.ehcache.pool.impl.DefaultSizeOfEngine.<init>(DefaultSizeOfEngine.java:111)) - using Agent sizeof engine
INFO 2015-12-28 14:09:20,746 (net.sf.ehcache.pool.impl.DefaultSizeOfEngine.<init>(DefaultSizeOfEngine.java:111)) - using Agent sizeof engine
INFO 2015-12-28 14:09:20,768 (net.sf.ehcache.pool.impl.DefaultSizeOfEngine.<init>(DefaultSizeOfEngine.java:111)) - using Agent sizeof engine
INFO 2015-12-28 14:09:20,789 (net.sf.ehcache.pool.impl.DefaultSizeOfEngine.<init>(DefaultSizeOfEngine.java:111)) - using Agent sizeof engine
I can see same logs in both Windows and Linux after application starts up
Could this be a issue with Linux JVM?
I think that it's a problem with spring 3.x.x and Java 8.
Take a look here Spring 3.2.x with Java 8