I have integrated a simple service call in Kony, When I run my App on iOS simulator i cannot see any errors, instead in the middleware.log files I can see below error, Can anyone please help.
Logs
X-Forwarded-For=null] 12:15:55,270 DEBUG factory.KonyAppFactory - Memcache is Enabled and Memcache Session Management instance is created
[appID=ProjectBestBuy01
requestID=61c164cc-1743-437f-8288-8e6ec0903d86
UA=BestBuy/1.0 CFNetwork/711.1.12 Darwin/15.0.0
rcid=NA
referer=NA
node.no=1
REMOTEADDRESS=127.0.0.1
REMOTEADDRESS=127.0.0.1
X-Forwarded-For=null] 11:39:51,112 ERROR cache.MemCacheWrapper - Unable to store object in Memcache node /127.0.0.1:21201 for Key: 10e6351c7-fe0d-44e4-bbc5-3e448d0af7a6 in 3 attempt from Server Node Num: 1
java.util.concurrent.TimeoutException: Timed out waiting for operation
at net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1656)
at com.konylabs.middleware.cache.MemCacheWrapper.storebyNoofattempts(MemCacheWrapper.java:277)
at com.konylabs.middleware.cache.MemCacheWrapper.store(MemCacheWrapper.java:235)
at com.konylabs.middleware.common.AbstractCacheSessionManager.store2Cache(AbstractCacheSessionManager.java:280)
at com.konylabs.middleware.common.AbstractCacheSessionManager.exit(AbstractCacheSessionManager.java:212)
at com.konylabs.middleware.common.MemCacheDCFilterAction.doChainDCFilter(MemCacheDCFilterAction.java:115)
at com.konylabs.middleware.common.MiddlewareMemCacheDCFilter.doFilter(MiddlewareMemCacheDCFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.konylabs.middleware.common.XSSFilter.doFilter(XSSFilter.java:169)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.konylabs.middleware.common.AddAdditionalResponseHeaderAttribute.doFilter(AddAdditionalResponseHeaderAttribute.java:94)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[appID=ProjectBestBuy01
requestID=3dc52d1e-9b5a-45d5-8b64-92f8cf1f0cb7
UA=BestBuy/1.0 CFNetwork/711.1.12 Darwin/15.0.0
rcid=NA
You question don't have enough information.
1.Are you trying to save any information on middle ware meme cache ?
2.if yes , how you are trying to save preprocessor,postprocessor or javaservice ?
3.Are you set scope as session in service def. file ?
Related
for Integration with IBM MQ, I am using Spring Boot to create a REST API with services that will attempt to put messages in a queue as well as to retrieve messages from a queue.
I am using the following versions:
Spring Boot: 2.3.1
IBM MQ: 9.1.4.0
My SpringBoot application has the dependencies:
1. mq-jms-spring-boot-starter (dependency): 2.0.9
2. spring-boot-starter-web
I have install IBM MQ using the recommended installation, and chose NOT to configure a domain user id for IBM MQ to run under.
As for the Spring Boot application, I have built it according to the following:
com.mi.bootibmmqdemo
BootIbmMqDemoApplication.java ----> Spring boot runnable class
com.mi.bootibmmqdemo.controllers.mq
BootIbmMqDemoController.java ----> Rest Controller
com.mi.bootibmmqdemo.model.mq
MqRequestMessage.java ----> POJO request object
MqResponseMessage.java ----> POJO response object
I constructed the REST controller like so:
package com.mi.bootibmmqdemo.controllers.mq;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jms.JmsException;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import com.mi.bootibmmqdemo.model.mq.MqRequestMessage;
import com.mi.bootibmmqdemo.model.mq.MqResponseMessage;
#RestController
#RequestMapping("/boot-ibm-mq")
public class BootIbmMqDemoController {
Logger logger = LoggerFactory.getLogger(BootIbmMqDemoController.class);
#Autowired
JmsTemplate jmsTemplate;
#PostMapping("/sendMq")
public MqResponseMessage sendToQueue(#RequestBody MqRequestMessage mqRequestMessage) {
try {
jmsTemplate.convertAndSend(mqRequestMessage.getQueueName(), mqRequestMessage.getMessage());
return new MqResponseMessage("OK");
} catch (JmsException ex) {
ex.printStackTrace();
return new MqResponseMessage("FAIL");
}
}
#GetMapping("/receiveMq")
String recv(){
try{
return jmsTemplate.receive("DEV.QUEUE.1").toString();
}catch(JmsException ex){
ex.printStackTrace();
return "FAIL";
}
}
}
My application.properties file contains the following configuration:
ibm.mq.queueManager=QM1
ibm.mq.channel=DEV.ADMIN.SVRCONN
ibm.mq.connName=localhost(1414)
ibm.mq.user=admin
ibm.mq.password=passw0rd
Testing with Postman using the below JSON request:
{
"queueName":"DEV.QUEUE.1",
"message":"Hello World!"
}
I am receiving the 'FAIL' outcome, and in the logs I am getting the following error:
org.springframework.jms.JmsSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'QM1' with connection mode 'Client' and host name 'localhost(1414)'.; nested exception is com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'QM1' with connection mode 'Client' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:286)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:185)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:507)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:584)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:661)
at com.mi.bootibmmqdemo.controllers.mq.BootIbmMqDemoController.sendToQueue(BootIbmMqDemoController.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'QM1' with connection mode 'Client' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:531)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:424)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:303)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6016)
at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6041)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:196)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
... 53 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
While searching online for the potential cause and starting from the error code, I have found the suggestion indicating that the channel authority rule for whole queue manager needs to be disabled, by using command prompt (windows) navigate to the MQ installation directory, in my case it is: D:\IBM\MQ\bin
and running the following:
D:\IBM\MQ\bin>runmqsc QM1
5724-H72 (C) Copyright IBM Corp. 1994, 2019.
Starting MQSC for queue manager QM1.
alter qmgr chlauth(disabled)
1 : alter qmgr chlauth(disabled)
AMQ8005I: IBM MQ queue manager changed.
end
2 : end
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
D:\IBM\MQ\bin>runmqsc QM1
5724-H72 (C) Copyright IBM Corp. 1994, 2019.
Starting MQSC for queue manager QM1.
ALTER QMGR CHLAUTH(DISABLED)
1 : ALTER QMGR CHLAUTH(DISABLED)
AMQ8005I: IBM MQ queue manager changed.
end
2 : end
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
D:\IBM\MQ\bin>
I restarted the queue manager and restarted the Spring Boot application and re-tested with postman, same error is repeated.
UPDATE: adding MQ logs:
6/17/2020 08:17:06 - Process(24384.13) User(MUSR_MQADMIN) Program(amqzlaa0.exe)
Host(E107) Installation(Installation1)
VRMF(9.1.4.0) QMgr(QM1)
Time(2020-06-17T05:17:06.756Z)
CommentInsert1(admin)
CommentInsert2(BootIbmMqDemoApplication)
CommentInsert3(N/A)
AMQ5534E: User ID 'admin' authentication failed
EXPLANATION:
The user ID and password supplied by the 'BootIbmMqDemoApplication' program
could not be authenticated.
Additional information: 'N/A'.
ACTION:
Ensure that the correct user ID and password are provided by the application.
Ensure that the authentication repository is correctly configured. Look at
previous error messages for any additional information.
----- amqzfuca.c : 4716 -------------------------------------------------------
6/17/2020 08:17:06 - Process(24384.13) User(MUSR_MQADMIN) Program(amqzlaa0.exe)
Host(E107) Installation(Installation1)
VRMF(9.1.4.0) QMgr(QM1)
Time(2020-06-17T05:17:06.756Z)
CommentInsert1(admin)
CommentInsert2(DEV.AUTHINFO)
CommentInsert3(CHCKCLNT(REQDADM))
AMQ5542I: The failed authentication check was caused by the queue manager
CONNAUTH CHCKCLNT(REQDADM) configuration.
EXPLANATION:
The user ID 'admin' and its password were checked because the queue manager
connection authority (CONNAUTH) configuration refers to an authentication
information (AUTHINFO) object named 'DEV.AUTHINFO' with CHCKCLNT(REQDADM).
This message accompanies a previous error to clarify the reason for the user ID
and password check.
ACTION:
Refer to the previous error for more information.
Ensure that a password is specified by the client application and that the
password is correct for the user ID. The authentication configuration of the
queue manager connection determines the user ID repository. For example, the
local operating system user database or an LDAP server.
If the CHCKCLNT setting is OPTIONAL, the authentication check can be avoided by
not passing a user ID across the channel. For example, by omitting the MQCSP
structure from the client MQCONNX API call.
To avoid the authentication check, you can amend the authentication
configuration of the queue manager connection, but you should generally not
allow unauthenticated remote access.
----- amqzfuca.c : 4739 -------------------------------------------------------
Kind Regards,
I have a multi-module spring gradle app. 2 modules are executable, both modules use vaadin 14.1.17. I use a remote server to deploy two jar at the same time, but every time I run the second application, the first breaks. help to understand what the problem is. stacktrace broken application:
com.vaadin.flow.server.BootstrapException: Unable to read webpack stats file.
at com.vaadin.flow.server.BootstrapHandler$BootstrapPageBuilder.setupFrameworkLibraries(BootstrapHandler.java:872) ~[flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.server.BootstrapHandler$BootstrapPageBuilder.setupDocumentHead(BootstrapHandler.java:749) ~[flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.server.BootstrapHandler$BootstrapPageBuilder.getBootstrapPage(BootstrapHandler.java:524) ~[flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.server.BootstrapHandler.synchronizedHandleRequest(BootstrapHandler.java:465) ~[flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1540) ~[flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:247) [flow-server-2.1.5.jar!/:2.1.5]
at com.vaadin.flow.spring.SpringServlet.service(SpringServlet.java:95) [vaadin-spring-12.1.2.jar!/:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:352) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.springframework.web.servlet.mvc.ServletForwardingController.handleRequestInternal(ServletForwardingController.java:141) [spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
each application runs on a separate port. In addition, before the first call through a web browser, the first application does not break. I launch applications on a Linux server as a systemd service.
I repeat again. Before I access through the browser, both services are running and do not contain errors. But after I access the FIRST running application, it crashes, but the second continues to work
After upgrading to Spring Boot 2.0.1, I am experiencing inexplicable problems with my applications.
There are two applications involved. Requests to App 1 make a back-end call to App 2 to fetch some data. The call to App 2 goes through an AWS elastic load balancer (ELB). The client used to make the call is Apache Commons HttpClient wrapped by my utility.
After App 1 was upgraded to Boot 2.0.1, I am seeing a small minority of the calls from App 1 to App 2 hang for a long time (15 minutes). When I use JConsole to get a thread dump of the hung threads, I see this stack trace:
Stack trace:
java.net.PlainSocketImpl.socketClose0(Native Method)
java.net.AbstractPlainSocketImpl.socketPreClose(AbstractPlainSocketImpl.java:693)
java.net.AbstractPlainSocketImpl.close(AbstractPlainSocketImpl.java:530)
- locked java.lang.Object#29a91ad3
java.net.SocksSocketImpl.close(SocksSocketImpl.java:1075)
java.net.Socket.close(Socket.java:1495)
- locked java.lang.Object#71e238ff
- locked java.net.Socket#73f58606
sun.security.ssl.BaseSSLSocketImpl.close(BaseSSLSocketImpl.java:624)
- locked sun.security.ssl.SSLSocketImpl#22264f18
sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1585)
sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1723)
sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
- locked sun.security.ssl.SSLSocketImpl#22264f18
- locked java.lang.Object#4338a60d
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked sun.security.ssl.AppInputStream#23fd5b55
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(BHttpConnectionBase.java:344)
org.apache.http.impl.BHttpConnectionBase.isStale(BHttpConnectionBase.java:364)
org.apache.http.impl.conn.CPool.validate(CPool.java:71)
org.apache.http.impl.conn.CPool.validate(CPool.java:45)
org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:249)
- locked org.apache.http.pool.AbstractConnPool$2#7c672c9a
org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:282)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:269)
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
com.kendelong.util.http.HttpConnectionService.doExecuteAndGetResponse(HttpConnectionService.java:243)
com.kendelong.util.http.HttpConnectionService.getResult(HttpConnectionService.java:189)
com.kendelong.util.http.IHttpConnectionService$getResult$0.call(Unknown Source)
com.hatchbaby.sub.util.MainSiteHttpProxyService.getMemberData(MainSiteHttpProxyService.groovy:68)
com.hatchbaby.sub.util.MainSiteHttpProxyService$$FastClassBySpringCGLIB$$3cc5cf68.invoke(<generated>)
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
com.kendelong.util.performance.PerformanceMonitoringAspect.monitorInvocation(PerformanceMonitoringAspect.java:108)
com.kendelong.util.performance.PerformanceMonitoringAspect.monitorAnnotatedClasses(PerformanceMonitoringAspect.java:83)
sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
com.hatchbaby.sub.util.MainSiteHttpProxyService$$EnhancerBySpringCGLIB$$edc027d4.getMemberData(<generated>)
com.hatchbaby.sub.service.security.HatchBabyAuthenticationUserDetailsService.loadUserDetails(HatchBabyAuthenticationUserDetailsService.java:39)
com.hatchbaby.sub.service.security.HatchBabyAuthenticationUserDetailsService.loadUserDetails(HatchBabyAuthenticationUserDetailsService.java:20)
org.springframework.security.web.authentication.preauth.PreAuthenticatedAuthenticationProvider.authenticate(PreAuthenticatedAuthenticationProvider.java:103)
org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doAuthenticate(AbstractPreAuthenticatedProcessingFilter.java:184)
org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:118)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.AbstractRequestLoggingFilter.doFilterInternal(AbstractRequestLoggingFilter.java:245)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
com.hatchbaby.sub.web.RequestInfoFilter.doFilter(RequestInfoFilter.java:99)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
com.hatchbaby.sub.web.SpecialRedirectFilter.doFilter(SpecialRedirectFilter.java:43)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.ebaysf.web.cors.CORSFilter.handleSimpleCORS(CORSFilter.java:302)
org.ebaysf.web.cors.CORSFilter.doFilter(CORSFilter.java:160)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#6ca9f281
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748)
The thread hangs in this state for roughly 15 minutes (actally 15 min 44 seconds plus/minus 5 seconds), or a rough integer multiple thereof. Eventually the request succeeds.
What is even odder is that when I compare the logs, I see the request enter App 1 and make the call to App 2 (say, at 8:00). The thread hangs in the socketClose0() method as seen in the thread dump. When the request finally completes 15 minutes later (at 8:15), the timestamps in the logs of App 2 and its ELB show that the request arrived the later time (8:15): it's as if the socket is trying to close for 15 minutes, it finally closes (without having contacted the App 2 server), and immediately retries. Usually it succeeds, but sometimes it gets stuck again for another 15 minutes (hence the multiples of 15 minutes). But the request doesn't even register at App 2's ELB until 15 minutes later (8:15).
I have seen this behavior in my QA environment (one server each) and in my production environment (2 servers each). If I roll the code back to the version with Spring Boot 1.5.9, the problem disappears. However, I have reverted most of the suspect libraries one-by-one to the Boot 1.5.9 version, and the problem persists. I have reverted httpclient (obviously), groovy, tomcat, and commons-codec. I was unable to revert Spring (reset spring.version property) as the app would not start. As part of the upgrade to Boot 2.0.1, I switched from EhCache distributed by JGroups to Hazelcast as the Hibernate second-level cache, but I have completely removed Hazelcast from the app and the problem persists. Reverting Hibernate or Spring Data JPA is more difficult because of code changes.
There are no timeouts set at 15 minutes in my app, of course. Both connection and socket timeouts for httpclient are 5 seconds.
Does anyone have any clue, or any troubleshooting suggestions?
UPDATE
Setting -Djavax.net.debug=all yields this output:
http-nio-8080-exec-6, setSoTimeout(1) called
[Raw read]: length = 5
0000: 15 03 03 00 1A .....
[Raw read]: length = 26
0000: 98 1C 7C C4 27 0B 2B 40 E5 AC D7 38 A9 6F F7 73 ....'.+#...8.o.s
0010: 83 18 E8 15 14 7C 52 10 47 5E ......R.G^
http-nio-8080-exec-6, READ: TLSv1.2 Alert, length = 26
Padded plaintext after DECRYPTION: len = 2
0000: 01 00 ..
http-nio-8080-exec-6, RECV TLSv1.2 ALERT: warning, close_notify
http-nio-8080-exec-6, called closeInternal(false)
http-nio-8080-exec-6, SEND TLSv1.2 ALERT: warning, description = close_notify
Padded plaintext before ENCRYPTION: len = 2
0000: 01 00 ..
http-nio-8080-exec-6, WRITE: TLSv1.2 Alert, length = 26
[Raw write]: length = 31
0000: 15 03 03 00 1A 00 00 00 00 00 00 00 02 1F A3 99 ................
0010: 4C CD 61 EB 02 2C 14 0D 00 27 03 51 05 F7 1F L.a..,...'.Q...
http-nio-8080-exec-6, called closeSocket(false)
At this point everything hangs. Looking with netstat on the client, I see the socket is in LAST_ACK state, meaning it's waiting for a final ACK packet.
I removed App 2's ELB from the mix and connected directly to the server for App 2. The problem persists. When client connections are in LAST_ACK state, there are no corresponding connections showing up in netstat on the server side. If the server had not sent the final ACK, it would be in FIN_WAIT_2, but since the connection is already gone, it must have sent the final ACK but it never arrived at the client.
How could a Spring Boot upgrade lead to blocked ACK packets on socket close operations?
All of this is happening on Amazon Linux 2017.9.
UPDATE 2
Five months later, same thing. I upgraded App2 to Spring Boot 2.0.5, no problems.
App1, when upgraded to 2.0.5, starts hanging it's threads. I turned on HttpClient wire logging, and what I see just where it's about to make the network call, there is no outgoing data on the wire logger, and 15m33s later I get an "end of stream" coming in and the request finally closes.
Halleuja, I figure it out. My HTTP library was setting socketLinger = socketTimeout + 500. However, socketLinger and socketTimeout are quite different! My socketTimeout is set to 5000 ms. I released version 1.7.0 which stopped setting socketLinger and presto the problem went away.
I still don't know why I never saw problems with this before, or why the Spring Boot 2 upgrade caused the bug to manifest, but at least it's gone now!
IN MY HADOOP 2.6.5 HA and oozie (use oozie-4.1.0-cdh5.12.1) when I run the oozie example.
[oozie#master shell]$ cat job.properties
nameNode=hdfs://cluster1:8020
jobTracker=master:8032
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/oozie/${examplesRoot}/apps/shell
[hadoop#master sbin]$
[hadoop#master sbin]$ oozie job -oozie http://master.bigdata.com:11000/oozie -config /home/hadoop/app/oozie/examples/apps/map-reduce/job.properties -run
Error: HTTP error code: 500 : Internal Server Error
[hadoop#master sbin]$
[hadoop#master shell]$ oozie job -oozie http://master.bigdata.com:11000/oozie -config /home/hadoop/app/oozie/examples/apps/shell/job.properties -run
Error: HTTP error code: 500 : Internal Server Error
[hadoop#master shell]$
The error is:
[oozie#master logs]$ pwd
/home/hadoop/app/oozie/logs
[oozie#master logs]$ vi oozie.log
2017-09-06 00:33:19,850 WARN AuthenticationFilter:532 - SERVER[master.bigdata.com] AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
2017-09-06 00:33:19,924 ERROR SubmitXCommand:517 - SERVER[master.bigdata.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Error,
java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsServerDefaults.<init>(JIISIZJLorg/apache/hadoop/util/DataChecksum$Type;)V
at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1327)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getServerDefaults(ClientNamenodeProtocolTranslatorPB.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy28.getServerDefaults(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getServerDefaults(DFSClient.java:996)
at org.apache.hadoop.hdfs.DFSClient.shouldEncryptData(DFSClient.java:2032)
at org.apache.hadoop.hdfs.DFSClient.newDataEncryptionKey(DFSClient.java:2038)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:208)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:159)
at org.apache.hadoop.hdfs.net.TcpPeerServer.peerFromSocketAndKey(TcpPeerServer.java:90)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3093)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:778)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:693)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:354)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:617)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:841)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:889)
at java.io.DataInputStream.read(DataInputStream.java:149)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.Reader.read(Reader.java:140)
at org.apache.oozie.util.IOUtils.copyCharStream(IOUtils.java:171)
at org.apache.oozie.service.WorkflowAppService.readDefinition(WorkflowAppService.java:135)
at org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:46)
at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:165)
at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:76)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.DagEngine.submitJob(DagEngine.java:114)
at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:192)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:92)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:76)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.DagEngine.submitJob(DagEngine.java:114)
at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:192)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:92)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:579)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:745)
2017-09-06 00:33:58,499 INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[master.bigdata.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
IN MY HADOOP 2.6.5 HA and oozie (use oozie-4.1.0-cdh5.12.1)
oozie-4.1.0+cdh5.12.1 is primarily target to work with hadoop-2.6.0+cdh5.12.1
Trying to mix versions or compile any later versions yourself is only asking for issues.
Specifically, you have a CLASSPATH issue
java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsServerDefaults.<init>
If you insist on using Cloudera packaging, you can find the necessary downloads here.
https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd_cdh_package_tarball_512.html#cm_vd_cdh_package_tarball_512
My recommendation would be to install Cloudera Manager and let it install and configure the CDH components for you
Cloudera Community Post
Using Hue in Cloudera 5.4.4, when trying to run a Sqoop 2 Job it says the (i) The job is starting... but it never actually runs the job. I see nothing in the Job Browser and nothing in the job's SUBMISSIONS list. I also do not see any errors in the logs.
This is on a stock Cloudera 5.4.4 (QuickStart VM) which I'm assuming has all of the components pre-configured correctly. Unfortunately this is all I have to go on is the lack of error messages & helpful reporting.
on CDH5, the sqoop 2 server does make some logs available under /var/log/sqoop2/sqoop.log. In my case, the error was not providing a partition column. This is an optional parameter, but the error reporting in the Hue interface doesn't surface any of these details.
2015-07-27 07:52:09,728 ERROR server.SqoopProtocolServlet [org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:84)] Exception in PUT http://quickstart.cloudera:12000/sqoop/v1/job/1/start
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0005:No column is found to partition data
at org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.configurePartitionProperties(GenericJdbcFromInitializer.java:147)
at org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.initialize(GenericJdbcFromInitializer.java:51)
at org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.initialize(GenericJdbcFromInitializer.java:40)
at org.apache.sqoop.driver.JobManager.initializeConnector(JobManager.java:449)
at org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:372)
at org.apache.sqoop.driver.JobManager.start(JobManager.java:276)
at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:379)
at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:115)
at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:277)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)