Hikari Data pool is shutting down in Spring boot application - spring-boot
We are facing the below issue while migrating.
Spring Batch 4.0.x from 3.0.x
Spring Boot 2.0.x from 1.5.x
Spring core 5.0.x from 4.3.x
After restarting the application, I could see after a min it throws the logs as SpringBootJPAHikariCP - Shutdown initiated....
But when we trigger the Job Hikari starts the new connections and working as expected.
But I am not sure why the first connection is shutting down after a minute.
I understood that SpringBoot2.0 uses the Hikari for the DB Pool, but not sure why I am seeing Shut down info logs after one min of ideal time.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.4.RELEASE)
2018-09-07 17:16:38,117 -0700,threadId=1,level=INFO ,logger=,msg="The following profiles are active: local"
2018-09-07 17:16:39,495 -0700,threadId=1,level=WARN ,logger=HikariConfig.1081,msg="SpringBootJPAHikariCP - idleTimeout is close to or more than maxLifetime, disabling it."
2018-09-07 17:16:39,495 -0700,threadId=1,level=INFO ,logger=HikariDataSource.110,msg="SpringBootJPAHikariCP - Starting..."
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-09-07 17:16:39,695 -0700,threadId=1,level=INFO ,logger=HikariDataSource.123,msg="SpringBootJPAHikariCP - Start completed."
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-09-07 17:16:40,002 -0700,threadId=1,level=INFO ,logger=Http11NioProtocol.180,msg="Initializing ProtocolHandler ["http-nio-8080"]"
2018-09-07 17:16:40,010 -0700,threadId=1,level=INFO ,logger=StandardService.180,msg="Starting service [Tomcat]"
2018-09-07 17:16:40,011 -0700,threadId=1,level=INFO ,logger=StandardEngine.180,msg="Starting Servlet Engine: Apache Tomcat/8.5.32"
2018-09-07 17:16:40,014 -0700,threadId=54,level=INFO ,logger=AprLifecycleListener.180,msg="The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/Users/xxxxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.]"
2018-09-07 17:16:40,117 -0700,threadId=54,level=INFO ,logger=[/].180,msg="Initializing Spring embedded WebApplicationContext"
2018-09-07 17:16:40,310 -0700,threadId=54,level=INFO ,logger=ReceiverConfig$$EnhancerBySpringCGLIB$$8c1192a0.68,msg="creating activeMQConnectionFactory"
2018-09-07 17:16:40,786 -0700,threadId=1,level=WARN ,logger=Flyway.53,msg="Flyway.setCallbacks(FlywayCallback) has been deprecated and will be removed in Flyway 6.0. Use Flyway.setCallbacks(Callback) instead."
2018-09-07 17:16:40,804 -0700,threadId=1,level=INFO ,logger=VersionPrinter.49,msg="Flyway Community Edition 5.1.4 by Boxfuse"
2018-09-07 17:16:40,807 -0700,threadId=1,level=INFO ,logger=DatabaseFactory.49,msg="Database: jdbc:mysql://127.0.0.1:3306/invoicing (MySQL 5.7)"
2018-09-07 17:16:40,853 -0700,threadId=1,level=INFO ,logger=DbValidate.49,msg="Successfully validated 2 migrations (execution time 00:00.015s)"
2018-09-07 17:16:40,861 -0700,threadId=1,level=INFO ,logger=DbMigrate.49,msg="Current version of schema `invoicing`: 1.0.13.1"
2018-09-07 17:16:40,862 -0700,threadId=1,level=INFO ,logger=DbMigrate.49,msg="Schema `invoicing` is up to date. No migration necessary."
2018-09-07 17:16:43,166 -0700,threadId=1,level=ERROR,logger=,msg="[ThreadID-1] Warning: big time skew between machina and client: service=1536365803166 client=1536365760000 diffSecs=43"
2018-09-07 17:16:44,425 -0700,threadId=1,level=WARN ,logger=JpaBaseConfiguration$JpaWebConfiguration$JpaWebMvcConfiguration.235,msg="spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning"
2018-09-07 17:16:45,814 -0700,threadId=85,level=INFO ,logger=FailoverTransport.1052,msg="Successfully connected to ssl://qal.message-preprod.a.intuit.com:61617"
2018-09-07 17:16:46,100 -0700,threadId=1,level=INFO ,logger=Http11NioProtocol.180,msg="Starting ProtocolHandler ["http-nio-8080"]"
2018-09-07 17:16:46,120 -0700,threadId=1,level=INFO ,logger=NioSelectorPool.180,msg="Using a shared selector for servlet write/read"
2018-09-07 17:16:46,133 -0700,threadId=1,level=INFO ,logger=LateFeeProcessor.59,msg="Started LateFeeProcessor in 12.283 seconds (JVM running for 13.152)"
2018-09-07 17:16:46,429 -0700,threadId=19,level=INFO ,logger=[/].180,msg="Initializing Spring FrameworkServlet 'dispatcherServlet'"
2018-09-07 17:17:47,868 -0700,threadId=93,level=INFO ,logger=HikariDataSource.381,msg="SpringBootJPAHikariCP - Shutdown initiated..."
2018-09-07 17:17:47,876 -0700,threadId=93,level=INFO ,logger=HikariDataSource.383,msg="SpringBootJPAHikariCP - Shutdown completed."
I have some issue and so far found that somehow spring issues shutdown command to Hikari thinking that its disposable bean.
Related
What is the default TLS version in Spring boot?
In the documentation the default value for server.ssl.protocol is TLS, but it does not specify which version will be used. I read that TLS 1.3 is available since java 11 but is it used by default in Sprint boot when available? Is there any configuration that can tell me which version is used in my project? Or any documentation depending on the Spring boot version that could tell the TLS version used by the framework?
I am using Spring Boot 2.7.3 and JDK 17 and by default, it supports TLSv1.3 You can check that by running the below command. My application is running locally on port 8080 so I passed 127.0.01:8080 after -connect openssl s_client -connect 127.0.01:8080 Output CONNECTED(00000003) 140704377439424:error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version:/AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/ssl/tls13_lib.c:151: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 5 bytes and written 294 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.3 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Start Time: 1668006818 Timeout : 7200 (sec) Verify return code: 0 (ok) --- You can change the TLS version by this property. server.ssl.enabled-protocols=TLSv1.2 Want to read more about this? refer below links https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto.webserver.configure-ssl https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#application-properties.server.server.ssl.enabled-protocols
How extend Tomcat v7.0.47 to support new TLS 1.2 ciphers?
Our application still runs on a Tomcat v7.0.47. Now our client requested that we limit the TLS connections to TLSv1.2+ and only allow a specific subset of ciphers. So in server.xml inside the Connector element I specified sslEnabledProtocols="TLSv1.2,TLSv1.3" and
ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ECDHE-RSA-AES256-GCM-SHA384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 ECDHE-RSA-AES256-SHA384
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 DHE-RSA-AES256-GCM-SHA384
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 DHE-RSA-AES256-SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ECDHE-RSA-AES128-GCM-SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 ECDHE-RSA-AES128-SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 DHE-RSA-AES128-GCM-SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 DHE-RSA-AES128-SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ECDHE-ECDSA-AES128-GCM-SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 ECDHE-ECDSA-AES256-GCM-SHA384"
as requested.
But when the client (actually a reverse proxy) connects to that Tomcat I get the following error in the catalina.out:
Oct 11, 2022 11:20:16 AM org.apache.tomcat.util.net.NioEndpoint setSocketOptions
SEVERE:
javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
at sun.security.ssl.ServerHandshakeContext.<init>(ServerHandshakeContext.java:62)
at sun.security.ssl.TransportContext.kickstart(TransportContext.java:220)
at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:97)
at org.apache.tomcat.util.net.SecureNioChannel.reset(SecureNioChannel.java:89)
at org.apache.tomcat.util.net.SecureNioChannel.<init>(SecureNioChannel.java:71)
at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:666)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:808)
at java.lang.Thread.run(Thread.java:750)
javax.net.ssl|FINE|B5|http-nio-8443-Acceptor-0|2022-10-11 11:20:16.045 CEST|HandshakeContext.java:304|No available cipher suite for TLS12
javax.net.ssl|SEVERE|B5|http-nio-8443-Acceptor-0|2022-10-11 11:20:16.045 CEST|TransportContext.java:316|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
"throwable" : {
javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
at sun.security.ssl.ServerHandshakeContext.<init>(ServerHandshakeContext.java:62)
at sun.security.ssl.TransportContext.kickstart(TransportContext.java:220)
at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:97)
at org.apache.tomcat.util.net.SecureNioChannel.reset(SecureNioChannel.java:89)
at org.apache.tomcat.util.net.SecureNioChannel.<init>(SecureNioChannel.java:71)
at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:666)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:808)
at java.lang.Thread.run(Thread.java:750)}
)```
So, obviously this Tomcat does not "know" or is unable to locate the required ciphers.
Any idea how I can add or configure these? Is there an additional library required to support these ciphers? Or some config that needs to be enabled?
The Tomcat runs on Java 8 (jdk8u322-b06)
sonarqube | Wrapper Stopped | 7.1
I am getting below error in sonar.log. I have installed Sonarqube 7.1 version in AWS Other Linux OS. Java -version is openjdk version "1.8.0_201". And nginx.repo configuration is:- [nginx] name=nginx repo baseurl=http://nginx.org/packages/rhel/7/x86_64/ gpgcheck=0 enabled=1 Error logs:- sonar.log --> Wrapper Started as Daemon Launching a JVM... Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved. 2019.09.27 13:13:26 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/efs/sonar/temp 2019.09.27 13:13:26 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001 2019.09.27 13:13:26 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/efs/sonar/elasticsearch]: /opt/efs/sonar/elasticsearch/bin/elasticsearch -Epath.conf=/opt/efs/sonar/temp/conf/es 2019.09.27 13:13:26 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running 2019.09.27 13:13:27 INFO app[][o.e.p.PluginsService] no modules loaded 2019.09.27 13:13:27 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin] 2019.09.27 13:13:52 INFO app[][o.s.a.SchedulerImpl] Process[es] is up 2019.09.27 13:13:52 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/efs/sonar]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/efs/sonar/temp -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -cp ./lib/common/*:/opt/efs/sonar/lib/jdbc/mysql/mysql-connector-java-5.1.42.jar org.sonar.server.app.WebServer /opt/efs/sonar/temp/sq-process3612618067556786670properties Fri Sep 27 13:13:55 UTC 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Fri Sep 27 13:13:56 UTC 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2019.09.27 13:13:59 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped 2019.09.27 13:13:59 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped 2019.09.27 13:13:59 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143 2019.09.27 13:13:59 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped <-- Wrapper Stopped Adding wrapper.conf wrapper.java.command=java wrapper.java.additional.1=-Dsonar.wrapped=true wrapper.java.additional.2=-Djava.awt.headless=true wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp wrapper.java.classpath.1=../../lib/jsw/*.jar wrapper.java.classpath.2=../../lib/common/*.jar wrapper.java.classpath.3=../../lib/*.jar wrapper.java.library.path.1=./lib wrapper.app.parameter.1=org.sonar.application.App wrapper.java.initmemory=8 wrapper.java.maxmemory=32 wrapper.console.format=PM wrapper.console.loglevel=INFO wrapper.logfile=../../logs/sonar.log wrapper.logfile.format=M wrapper.logfile.loglevel=INFO wrapper.syslog.loglevel=NONE wrapper.console.title=SonarQube wrapper.single_invocation=true wrapper.ntservice.name=SonarQube wrapper.ntservice.displayname=SonarQube wrapper.ntservice.description=SonarQube wrapper.ntservice.dependency.1= wrapper.ntservice.starttype=AUTO_START wrapper.ntservice.interactive=false wrapper.disable_restarts=TRUE wrapper.ping.timeout=0 wrapper.shutdown.timeout=300 wrapper.jvm_exit.timeout=300 List of plugins installed until now under /opt/efs/sonar/extension/plugins folder:- sonar-apigee-plugin-2.0.0.jar sonar-auth-bitbucket-plugin-1.0.jar sonar-csharp-plugin-7.0.1.4822.jar sonar-flex-plugin-2.4.0.1222.jar sonar-java-plugin-5.2.0.13398.jar sonar-javascript-plugin-4.1.0.6085.jar sonar-php-plugin-2.13.0.3107.jar sonar-python-plugin-1.9.1.2080.jar sonar-scm-git-plugin-1.4.0.1037.jar sonar-scm-svn-plugin-1.7.0.1017.jar sonar-typescript-plugin-1.6.0.2388.jar sonar-xml-plugin-2.0.1.2020.jar
After giving 644 permission to *.jar plugin it started working
Pentaho Data Integration How to run job with kitchen on carte cluster?
I had set up a carte cluster (1 master and 2 slaves) and run a job on carte cluster with spoon. But when I ran with kitchen command or carte http access, it ran as standalone (just run in master node). Did I miss anything in the configuration? Or doesn't it support cluster mode? Here was what I tried: my config: ran in spoon with "Enviroment Type -- Local" master output: 2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2017/11/28 04:47:10 - sortcluster111 (master) - Dispatching started for transformation [sortcluster111 (master)] Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2017/11/28 04:47:10 - output.0 - Connected to database [102] (commit=1000) 2017/11/28 04:47:10 - input.0 - Finished reading query, closing connection. 2017/11/28 04:47:10 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0) 2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40001], reading from server Dynamic slave [kettleslave02:8083] 2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40000], reading from server Dynamic slave [kettleslave01:8082] 2017/11/28 04:47:10 - output.0 - Finished processing (I=47, O=47, R=0, W=47, U=0, E=0) slave01 output: 2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2017/11/28 04:47:10 - sortcluster111 (cluster1:Dynamic slave [kettleslave01:8082]) - Dispatching started for transformation [sortcluster111 (cluster1:Dynamic slave [kettleslave01:8082])] 2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01 2017/11/28 04:47:10 - sort.0 - Finished processing (I=24, O=0, R=0, W=24, U=0, E=0) slave02 output: 2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml 2017/11/28 04:47:09 - General - Unable to connect to the repository with name 'Mysqlrep' 2017/11/28 04:47:10 - sortcluster111 (cluster1:Dynamic slave [kettleslave02:8083]) - Dispatching started for transformation [sortcluster111 (cluster1:Dynamic slave [kettleslave02:8083])] 2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01 2017/11/28 04:47:10 - sort.0 - Finished processing (I=23, O=0, R=0, W=23, U=0, E=0) ran with kitchen: kitchen.sh -rep=Mysqlrep -user=admin -pass=admin -job trans1 master output: 2017/11/28 04:10:19 - trans1 - Starting entry [sorttrans] 2017/11/28 04:10:19 - sorttrans - Loading transformation from repository [sortcluster111] in directory [/] 2017/11/28 04:10:19 - sorttrans - Using run configuration [cluster config] 2017/11/28 04:10:19 - sorttrans - Using legacy execution engine 2017/11/28 04:10:19 - sortcluster111 - Dispatching started for transformation [sortcluster111] Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2017/11/28 04:10:19 - output.0 - Connected to database [102] (commit=1000) 2017/11/28 04:10:19 - input.0 - Finished reading query, closing connection. 2017/11/28 04:10:19 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0) 2017/11/28 04:10:19 - sort.0 - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0) 2017/11/28 04:10:19 - output.0 - Finished processing (I=0, O=47, R=47, W=47, U=0, E=0) 2017/11/28 04:10:19 - trans1 - Starting entry [finish] 2017/11/28 04:10:19 - trans1 - Finished job entry [finish] (result=[true]) 2017/11/28 04:10:19 - trans1 - Finished job entry [sorttrans] (result=[true]) 2017/11/28 04:10:19 - trans1 - Finished job entry [SQL] (result=[true]) 2017/11/28 04:10:19 - trans1 - Job execution finished 2017/11/28 04:10:19 - Kitchen - Finished! 2017/11/28 04:10:19 - Kitchen - Start=2017/11/28 04:10:00.586, Stop=2017/11/28 04:10:19.739 2017/11/28 04:10:19 - Kitchen - Processing ended after 19 seconds. has no output in the slave Regards John
There is a bug in new versions of pdi: The option "Run this transformation in a clustered mode?" not exists, so to fix that you need open the job xml file and remove the property run_configuration and set cluster to Y from entry's of transformation that you wanna run in clustered mode. Hope this helps.
First, start carte on distant server: ./carte.sh hostname port check in your browser: hostname:port/kettle/status/ respond with an empty status page Second, define carte in spoon: in spoon view (left panel): click on Slave server, then new server name = carte hostname = previously defined hostname port = previously defined port web app name = blank (important!) username = cluster password = cluster Is the master = Yes Third, configure spoon run: in spoon view (left panel): click on Run configuration, then new Name: carte/servername Engine: Pentaho (default) Slave server: Yes Location: carte (server name as defined in second step) Send resource to this server: Yes (unless you read the doc) Forth, run the transformaton/job the Run configuration drop-down box let you choose between Pentaho local or carte/hostname, or any configuration defined in third step. if you disabled the Run option panel, then Drop-down the Run tool or Top menu/Action/Run option or F8. And check the Always show dialog on run, you'll need it in the future. Fifth in your browser: hostname:port/kettle/status/ have fun.
KrbException connecting to Hadoop cluster with Zookeeper client - UNKNOWN_SERVER
My Zookeeper client is having trouble connecting to the Hadoop cluster.
This works fine from a Linux VM, but I am using a Mac.
I set the -Dsun.security.krb5.debug=true flag on the JVM and get the following output:
Found ticket for solr#DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM#DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for solr#DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM#DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000, number of retries =3, #bytes=682
>>> KDCCommunication: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000,Attempt =1, #bytes=682
>>> KrbKdcReq send: #bytes read=217
>>> KdcAccessibility: remove oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
cTime is Thu Dec 24 11:18:15 GMT 2015 1450955895000
sTime is Fri Apr 28 15:15:06 BST 2017 1493388906000
suSec is 925863
error code is 7
error Message is Server not found in Kerberos database
cname is solr#DDA.MYCO.COM
sname is zookeeper/oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com#DDA.MYCO.COM
msgType is 30
KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:366)
at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:362)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:348)
at org.apache.zookeeper.client.ZooKeeperSaslClient.sendSaslPacket(ZooKeeperSaslClient.java:420)
at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:458)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1057)
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 18 more
ERROR 2017-04-28 15:15:07,046 5539 org.apache.zookeeper.client.ZooKeeperSaslClient [main-SendThread(oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com:2181)]
An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)])
occurred when evaluating Zookeeper Quorum Member's received SASL token.
This may be caused by Java's being unable to resolve the Zookeeper Quorum Member's hostname correctly.
You may want to try to adding '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment.
Zookeeper Client will go to AUTH_FAILED state.
I've tested Kerberos config as follows:
>kinit -kt /etc/security/keytabs/solr.headless.keytab solr
>klist
Credentials cache: API:3451691D-7D5E-49FD-A27C-135816F33E4D
Principal: solr#DDA.MYCO.COM
Issued Expires Principal
Apr 28 16:58:02 2017 Apr 29 04:58:02 2017 krbtgt/DDA.MYCO.COM#DDA.MYCO.COM
Following the instructions from hortonworks I managed to get the kerberos ticket stored in a file:
>klist -c FILE:/tmp/krb5cc_501
Credentials cache: FILE:/tmp/krb5cc_501
Principal: solr#DDA.MYCO.COM
Issued Expires Principal
Apr 28 17:10:25 2017 Apr 29 05:10:25 2017 krbtgt/DDA.MYCO.COM#DDA.MYCO.COM
Also I tried the suggested JVM option suggested in the stack trace (-Dsun.net.spi.nameservice.provider.1=dns,sun), but this led to a different error along the lines of Client session timed out, which suggests that this JVM param is preventing the client from connecting correctly in the first place.
==EDIT==
Seems that the Mac version of Kerberos is not the latest:
> krb5-config --version
Kerberos 5 release 1.7-prerelease
I just tried brew install krb5 to install a newer version, then adjusting the path to point to the new version.
> krb5-config --version
Kerberos 5 release 1.15.1
This has had no effect whatsoever on the outcome.
NB this works fine from a linux VM on my Mac, using exactly the same jaas.conf, keytab files, and krb5.conf.
krb5.conf:
[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = DDA.MYCO.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
DDA.MYCO.COM = {
admin_server = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
kdc = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
}
Reverse DNS:
I checked that the FQDN hostname I'm connecting to can be found using a reverse DNS lookup:
> host 10.252.132.160
160.132.252.10.in-addr.arpa domain name pointer oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com.
This is exactly as per the response to the same command from the linux VM.
===WIRESHARK ANALYSIS===
Using Wireshark configured to use the system key tabs allows a bit more detail in the analysis.
Here I have found that a failed call looks like this:
client -> host AS-REQ
host -> client AS-REP
client -> host AS-REQ
host -> client AS-REP
client -> host TGS-REQ <-- this call is detailed below
host -> client KRB error KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
The erroneous TGS-REQ call shows the following:
Kerberos
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 1 item
req-body
Padding: 0
kdc-options: 40000000 (forwardable)
realm: DDA.MYCO.COM
sname
name-type: kRB5-NT-UNKNOWN (0)
sname-string: 2 items
SNameString: zookeeper
SNameString: oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
till: 1970-01-01 00:00:00 (UTC)
nonce: 797021964
etype: 3 items
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
Here is the corresponding successful call from the linux box, which is followed by several more exchanges.
Kerberos
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 1 item
req-body
Padding: 0
kdc-options: 40000000 (forwardable)
realm: DDA.MYCO.COM
sname
name-type: kRB5-NT-UNKNOWN (0)
sname-string: 2 items
SNameString: zookeeper
SNameString: d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
till: 1970-01-01 00:00:00 (UTC)
nonce: 681936272
etype: 3 items
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
So it looks like the client is sending
oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
as the server host, when it should be sending:
d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
So the question is, how do I fix that? Bear in mind this is a Java piece of code.
My /etc/hosts has the following:
10.252.132.160 b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
10.252.134.51 d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
10.252.132.139 d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
And my krb5.conf file has:
kdc = d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
kdc = b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
kdc = d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
I tried adding -Dsun.net.spi.nameservice.provider.1=file,dns as a JVM param but got the same result.
I fixed this by setting up a local dnsmasq instance to supply the forward and reverse DNS lookups. So now from the command line, host d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com returns 10.252.134.51 See also here and here.
Looks like some DNS issue. Could this SO question help you resolving your problem? Also, here is an Q&A about the problem. It also could be because of non Sun JVM.