WAS 6.1, Windows 7, Error Starting and Stopping Server - websphere

Hi I had WAS 6.1 running on Windows 7 for 3-4 months without a problem and then one day it started stalling when I stopped it and it wouldn’t start backup. That has been 2 months ago or so. I have uninstalled it and reinstalled it 5 or 6 times to try and resolve the issue with no luck; same symptoms.
When I try and start the server sometimes it works and other times it doesn’t. When it doesn’t I get the following exception:
Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host PHXJ05308513.ads.xxx.com at port 8880
I have tried to go into servers.xml and change that port but it seems to have no effect. I have tried to delete the .pid file before starting as well (in the logs folder for the profile).
In addition, when I attempt to create a datasource connection (that I know works) I get :
Exception attempting to get free port for status socket java.net.BindException: Address already in use: NET_Bind
I have also seen CORBA errors in the log while trying to troubleshoot this. I have read enough to know that there is a networking conflict somewhere, but I have tried many of the solutions and they don't resolve the issue.
Start Server Stack Trace:
[6/29/15 18:29:21:102 GMT] 0000000a AdminTool A ADMU3100I: Reading configuration for server: server1
[6/29/15 18:29:23:741 GMT] 0000000a AdminTool A ADMU3012E: Exception attempting to get free port for status socket Address already in use: NET_Bind
[6/29/15 18:29:23:741 GMT] 0000000a AdminTool E ADMU3012E: Exception attempting to get free port for status socket java.net.BindException: Address already in use: NET_Bind
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:398)
at java.net.ServerSocket.bind(ServerSocket.java:331)
at java.net.ServerSocket.<init>(ServerSocket.java:197)
at java.net.ServerSocket.<init>(ServerSocket.java:109)
at com.ibm.ws.management.tools.AdminTool.initStatusPort(AdminTool.java:983)
at com.ibm.ws.management.tools.AdminTool.getStatusPort(AdminTool.java:964)
at com.ibm.ws.management.tools.WsServerLauncher.runTool(WsServerLauncher.java:282)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:261)
at com.ibm.ws.management.tools.WsServerController.executeUtilityOnWindows(WsServerController.java:136)
at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:263)
[6/29/15 18:29:25:441 GMT] 0000000a AdminTool A ADMU3200I: Server launched. Waiting for initialization status.
[6/29/15 18:29:25:441 GMT] 0000000a WsServerLaunc E ADMU3002E: Exception attempting to process server server1
[6/29/15 18:29:25:441 GMT] 0000000a WsServerLaunc E ADMU3007E: Exception java.lang.NullPointerException
at com.ibm.ws.management.tools.WsServerController.waitForServerInit(WsServerController.java:208)
at com.ibm.ws.management.tools.WsServerLauncher.runTool(WsServerLauncher.java:310)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:261)
at com.ibm.ws.management.tools.WsServerController.executeUtilityOnWindows(WsServerController.java:136)
at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:263)
[6/29/15 18:29:25:441 GMT] 0000000a AdminTool E ADMU0111E: Program exiting with error: java.lang.NullPointerException
at com.ibm.ws.management.tools.WsServerController.waitForServerInit(WsServerController.java:208)
at com.ibm.ws.management.tools.WsServerLauncher.runTool(WsServerLauncher.java:310)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:261)
at com.ibm.ws.management.tools.WsServerController.executeUtilityOnWindows(WsServerController.java:136)
at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:263)
[6/29/15 18:29:25:457 GMT] 0000000a AdminTool A ADMU0111E: Program exiting with error: java.lang.NullPointerException
at com.ibm.ws.management.tools.WsServerController.waitForServerInit(WsServerController.java:208)
at com.ibm.ws.management.tools.WsServerLauncher.runTool(WsServerLauncher.java:310)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:261)
at com.ibm.ws.management.tools.WsServerController.executeUtilityOnWindows(WsServerController.java:136)
at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:263)
Stop Server Stack Trace:
[6/29/15 18:31:28:858 GMT] 0000000a WsServerStop E ADMU3002E: Exception attempting to process server server1
[6/29/15 18:31:28:859 GMT] 0000000a WsServerStop E ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host PHXJ05308513.ads.xxxx.com at port 8881.
[6/29/15 18:31:28:860 GMT] 0000000a WsServerStop A ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host PHXJ05308513.ads.xxxx.com at port 8881.
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:479)
at com.ibm.ws.management.tools.LaunchUtils.getAdminClient(LaunchUtils.java:372)
at com.ibm.ws.management.tools.LaunchUtils.getAdminClient(LaunchUtils.java:312)
at com.ibm.ws.management.tools.WsServerStop.runTool(WsServerStop.java:325)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:261)
at com.ibm.ws.management.tools.WsServerController.executeUtilityOnWindows(WsServerController.java:136)
at com.ibm.ws.management.tools.WsServerStop.main(WsServerStop.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:183)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:90)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:72)
at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:226)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:376)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:334)
at org.eclipse.core.launcher.Main.basicRun(Main.java:278)
at org.eclipse.core.launcher.Main.run(Main.java:973)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:245)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:73)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:67)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:521)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:331)
... 25 more
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.BindException: Address already in use: NET_Bind; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.BindException: Address already in use: NET_Bind]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:338)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.<init>(SOAPConnectorClient.java:175)
... 30 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.BindException: Address already in use: NET_Bind; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.BindException: Address already in use: NET_Bind]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source)
at org.apache.soap.rpc.Call.invoke(Unknown Source)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:303)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:296)
... 31 more

Use the netstat command and search on all the default ports assigned to your JVM. This way you can see if one of the several ports that the JVM uses is being used by some other app. It may not be only your soap port causing the issue, but one of the many other defined ports on the container. You can also try and telnet to all the JVM ports in order to see if another app is listening on that specific port.

Related

FNRPE2131090482E - Server is being initialized. Retry the operation

I am using Filnet 5.2.1. I have everything working fine for years.
but suddenly today when trying to connect to process engine from workplace application I am getting the error :
[FNRPE2131090482E]Server is being initialized. Retry the operation.
The Process Engine server is not yet initialized. The software automatically retries your operation but if the problem persists, try again.
at filenet.jpe.rpc.appserver.PEEngineServlet.<clinit>(PEEngineServlet.java:512)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at <unknown class>.<unknown method>(Unknown Source)
at java.lang.Class.forNameImpl(Native Method)
at <unknown class>.<unknown method>(Unknown Source)
at java.beans.Beans.instantiate(Beans.java:189)
at java.beans.Beans.instantiate(Beans.java:80)
at com.ibm.ws.webcontainer.servlet.ServletWrapper$1.run(ServletWrapper.java:1486)
at <unknown class>.<unknown method>(Unknown Source)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadServlet(ServletWrapper.java:1475)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1354)
at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:623)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:589)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1177)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:776)
at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2195)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2200)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1865)
Edit
Looking at pesvr_system.log I found the following :
RPCHandler PEServer.init [Error] FNRPE2131090405E Start up PE server failed; Exception: Unexpected exception -- see chained exception
at filenet.jpe.general.RuntimeServiceException.ThrowUnchecked(RuntimeServiceException.java:93)
at filenet.jpe.rpc.http.HTTPListenerThread.init(HTTPListenerThread.java:144)
at filenet.jpe.rpc.main.PEServer.execute(PEServer.java:365)
at filenet.jpe.rpc.appserver.PEEngineServlet$ServerStartupThread.run(PEEngineServlet.java:120)
PEServer.init [Error] Error starting listening on 9092; Exception:
Caused by: java.net.BindException: Address already in use: JVM_Bind
2021/02/11 11:35:14.937+0300 RPCHandler 7adf27d9 [Info] PEServer: shutting down...
2021/02/11 11:35:14.937+0300 RPCHandler 7adf27d9 [Info] Component Manager.Shutdown...
2021/02/11 11:35:14.937+0300 RPCHandler 7adf27d9 [Info] PEServer: shutting down worker threads..
2021/02/11 11:35:14.953+0300 RPCHandler 7adf27d9 [Info] PEServer: Finished shutting down.
Issue was fixed by restoring old backup for filenet GCD database.

ADMC0016E: The system cannot create a SOAP connector to connect to host xxx at port 16313

Getting following error on WAS8558.
This WAS8558 is being used by Netcool/OMNIbus WebGUI8.1.
I am having problems while stopping WAS and getting below error.
Earlier I had done the SSL configuration and performed steps mentioned on below link to enable TLSv1.2
https://developer.ibm.com/answers/questions/206952/how-do-i-configure-websphere-application-server-ss.html
ERROR:
8:46:59:065 EDT] 00000001 WsServerStop A ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: com.ibm.websphere.management.exc
eption.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host xxxxx.cap.dev.net at port 16313.
at com.ibm.ws.management.connector.ConnectorHelper.createConnector(ConnectorHelper.java:628)
at com.ibm.ws.management.tools.WsServerStop.runTool(WsServerStop.java:373)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:271)
at com.ibm.ws.management.tools.WsServerStop.main(WsServerStop.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
at org.eclipse.core.launcher.Main.run(Main.java:981)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:401)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:164)
Caused by: com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host xxxxx.cap.dev.net at port 16313.
at com.ibm.websphere.management.AdminClientFactory.createAdminClientPrivileged(AdminClientFactory.java:635)
at com.ibm.websphere.management.AdminClientFactory.access$000(AdminClientFactory.java:127)
at com.ibm.websphere.management.AdminClientFactory$1.run(AdminClientFactory.java:210)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:206)
at com.ibm.ws.management.connector.ConnectorHelper.tryProtocol(ConnectorHelper.java:665)
at com.ibm.ws.management.connector.ConnectorHelper.createConnector(ConnectorHelper.java:597)
... 29 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:86)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:58)
at java.lang.reflect.Constructor.newInstance(Constructor.java:542)
at com.ibm.websphere.management.AdminClientFactory.createAdminClientPrivileged(AdminClientFactory.java:457)
... 35 more
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOExce
ption: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure; targetException=java.lang.IllegalArgumentException: Error opening socket:
java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:429)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.(SOAPConnectorClient.java:228)
... 40 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert:
handshake_failure; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Receive
d fatal alert: handshake_failure]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:475)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:510)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:387)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:372)
... 41 more
try these Solutions
1- write "xxxxx.cap.dev.net:16313/soap" on the browser and hit enter to check if your problem relevant to firewall or not.
if you get a reply, we become to ensure that your problem is not a firewall problem.
2- ensure that the certificate has not been expired
3- in some cases, SOAP connector does not work with TLSv1.2, but it works with v1.
4- try to work with eclipse instead of IBM integration designer.

ZuulException: Forwarding error, Caused by: java.net.SocketException: Broken pipe

Am facing the below ZuulException exception due to Broken pipe issue in my Integeration testing environment. Have just a single service deployed but still am getting the below exception and ideally it should be available all the time.
This issue is intermittent, sometime it is working and sometime it isn't. Have anyone faced this issue and is there any addition configurations to be done?
LOG TRACE
com.netflix.zuul.exception.ZuulException: Forwarding error
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:145) ~[spring-cloud-netflix-core-1.0.0.RELEASE.jar!/:1.0.0.RELEASE]
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:110) ~[spring-cloud-netflix-core-1.0.0.RELEASE.jar!/:1.0.0.RELEASE]
at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112) ~[zuul-core-1.0.28.jar!/:na]
at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197) ~[zuul-core-1.0.28.jar!/:na]
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:161) ~[zuul-core-1.0.28.jar!/:na]
at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:120) ~[zuul-core-1.0.28.jar!/:na]
at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:84) ~[zuul-core-1.0.28.jar!/:na]
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: userManagementRibbonCommand failed and no fallback available.
at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:816) ~[hystrix-core-1.4.0-RC6.jar!/:na]
at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:798) ~[hystrix-core-1.4.0-RC6.jar!/:na]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77) ~[rxjava-1.0.4.jar!/:1.0.4]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70) ~[rxjava-1.0.4.jar!/:1.0.4]
Caused by: com.netflix.client.ClientException: null
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:123) ~[ribbon-loadbalancer-2.0-RC13.jar!/:na]
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:81) ~[ribbon-loadbalancer-2.0-RC13.jar!/:na]
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45]
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159) ~[httpcore-4.3.3.jar!/:4.3.3]
Not sure if this will fix your problem but if you already don't have it then add this to your application.yml file
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 60000
ribbon:
ConnectTimeout: 3000
ReadTimeout: 60000

How to bind port 80 for my tomcat 7 on centos 7

log:
Dec 05, 2015 2:22:10 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-80]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-80]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:813)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.startup.Catalina.load(Catalina.java:638)
at org.apache.catalina.startup.Catalina.load(Catalina.java:663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:280)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:454)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:980)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
... 12 more
Caused by: java.net.BindException: Permission denied <null>:80
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:411)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:640)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:434)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:978)
... 13 more
Caused by: java.net.BindException: Permission denied
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:398)
... 17 more
And I followed the instruction here to install tomcat.
It seams that I should grant permissions to my tomcat but how? And when I googled this problem, many says that I should modify this file /etc/defaults/tomcat7 But, I do not have this file on my centOS 7.
Thanks for your help.
I found a solution by forwarding:
firewall-cmd --zone=public --add-forward-port=port=80:proto=tcp:toport=8080 --permanent
and
firewall-cmd --reload

Upload data to HDFS running in Amazon EC2 from local non-Hadoop Machine

I set up a hadoop cluster of two nodes on Amazon EC2. It works well. I can upload data to HDFS from master node or other instances in the same Amazon zone as the hadoop cluster by using hadoop api (java program is attached).
However, when I want to do this from my local non-hadoop machine, it turns out with exceptions as below:
I then login to the hadoop namenode to check with command line. The folder "testdir" is created, but the size of the uploaded file "myfile" is 0.
==================this is separator===============================
These are the exceptions
Apr 18, 2013 10:40:47 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream createBlockOutputStream
INFO: Exception in createBlockOutputStream 10.196.153.215:50010 java.net.ConnectException: Connection timed out
Apr 18, 2013 10:40:47 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream nextBlockOutputStream
INFO: Abandoning block blk_560654195674249927_1002
Apr 18, 2013 10:40:47 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream nextBlockOutputStream
INFO: Excluding datanode 10.196.153.215:50010
Apr 18, 2013 10:41:09 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream createBlockOutputStream
INFO: Exception in createBlockOutputStream 10.195.171.154:50010 java.net.ConnectException: Connection timed out
Apr 18, 2013 10:41:09 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream nextBlockOutputStream
INFO: Abandoning block blk_1747509888999401559_1002
Apr 18, 2013 10:41:10 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream nextBlockOutputStream
INFO: Excluding datanode 10.195.171.154:50010
Apr 18, 2013 10:41:10 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer run
WARNING: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/ubuntu/testdir/myfile could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)
Apr 18, 2013 10:41:10 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream processDatanodeError
WARNING: Error Recovery for block blk_1747509888999401559_1002 bad datanode[0] nodes == null
Apr 18, 2013 10:41:10 AM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream processDatanodeError
WARNING: Could not get block locations. Source file "/user/ubuntu/testdir/myfile" - Aborting...
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/ubuntu/testdir/myfile could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)
==================this is separator===============================
Here is my java codes
Path output = new Path("testdir");
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://ec2-23-22-12-173.compute-1.amazonaws.com:9000");
conf.set("hadoop.job.user",ubuntu);
FileSystem.mkdirs(FileSystem.get(conf), output, FsPermission.valueOf("drwxr-xr-x"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("./myfile"), output);
==================this is separator===============================
PS. I have open port 9000, 50010 in the security group and turned off the linux firewall already.
Anyone has any thoughts?
Thanks.
There could be several reasons behind this error :
1- DataNodes are not up and running. Make sure it is not the case. If you don't get anything try to dig the DN logs on each server.
2- The space on the machines where DNs are running is lesser than the space specified by you through "dfs.datanode.du.reserved" property.
3- There is actually no space left on your DN machines.
4- The path specified by "dfs.data.dir" in your hdfs-site.xml file has no space left.(Perhaps the disk serving as dfs.data.dir has run out of space).
5- DNs are not able to send heartbeats/block-reports to the NN. Make sure there is no network related issue.
HTH
Did you find any answer to this problem ... if not, here is the potential "REASON" ==> your client is trying to access the data nodes from their private IP address on EC2 (which is only visible to the cluster) not the public IP. you can verify that looking at your error log: excluding datanode private ip not the public ip, but I have no clue how shall we overcome that. I have the same issue. for more information check this link: http://www.hadoopinrealworld.com/could-only-be-replicated-to-0-nodes/

Resources