Why is the non persistent ejb timer not triggering? - websphere

I'm trying to replace an existing logic in our application that makes use of persistent proggramatic timers to non persistent timers due to a recent issue that we encountered.
The file system that WAS uses to write the persistent information of the timers was full and the Derby database got restarted in read only mode , making our application not able to trigger the timers post that, we had to restart the server to fix the same.
this is the current piece of code that is used to trigger the programmatic timer
Timer timer = timerService.createTimer(100,eventContextWrapper);
As per the examples quoted in http://docs.oracle.com/javaee/6/tutorial/doc/bnboy.html, changed our code to
TimerConfig tC = new TimerConfig(eventContextWrapper, false);
Timer timer = timerService.createSingleActionTimer(100, tC);
but we are facing the below exception
[5/31/17 22:18:23:348 GMT+08:00] 000000bf SystemErr R org.omg.CORBA.MARSHAL: Unable to read value from underlying bridge : null vmcid: IBM minor code: 896 completed: No
[5/31/17 22:18:23:348 GMT+08:00] 000000bf SystemErr R at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:771)
[5/31/17 22:18:23:348 GMT+08:00] 000000bf SystemErr R at com.ibm.rmi.iiop.EncoderInputStream.read_value(EncoderInputStream.java:840)
[5/31/17 22:18:23:348 GMT+08:00] 000000bf SystemErr R at com.ibm.ws.ejbcontainer.util.ORBObjectCopierImpl.copy(ORBObjectCopierImpl.java:52)
[5/31/17 22:18:23:348 GMT+08:00] 000000bf SystemErr R at com.ibm.ws.ejbcontainer.util.ObjectUtil.copy(ObjectUtil.java:159)
[5/31/17 22:18:23:349 GMT+08:00] 000000bf SystemErr R at com.ibm.ejs.container.TimerNpImpl.<init>(TimerNpImpl.java:240)
[5/31/17 22:18:23:349 GMT+08:00] 000000bf SystemErr R at com.ibm.ejs.container.TimerNpImpl.<init>(TimerNpImpl.java:276)
[5/31/17 22:18:23:349 GMT+08:00] 000000bf SystemErr R at com.ibm.ws.runtime.component.WASEJBRuntimeImpl.createTimer(WASEJBRuntimeImpl.java:1371)
[5/31/17 22:18:23:349 GMT+08:00] 000000bf SystemErr R at com.ibm.ejs.container.BeanO.createSingleActionTimer(BeanO.java:2551)
com.test.customer.event.EventTimerBean.triggerTimer(EventTimerBean.java:62)
Line 62 is where the createSingleActionTimer is invoked.
Have done some analysis and unable to fix the same. any lead would be helpful..

Eventually found that one of the classes in the EventContextWrapper is not serializable and that's why we were getting the above error. We changed that class to implement serializable and it worked. Still ain't sure , how come it worked when persistent timer was used.

Related

Getting the following exception when trying to render a large PDF in Oracle EWPS

While trying to process a large PDF file I get the following stack trace. This is only happening when we try a large PDF and does not impact the majority of work. This is Oracle Enterprise Web Processing Services (EWPS) Axis2 running on WebSphere Application Server version 8.5.5.13. We think it is an issue with a timeout setting but are not sure what setting to modify.
[5/10/19 3:38:50:458 CDT] 000000b1 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause AxisServlet: org.apache.axis2.AxisFault: Error writing stream: OutputStream encountered error during write
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:417)
at org.apache.axis2.description.AxisService.printUserWSDL(AxisService.java:936)
at org.apache.axis2.description.AxisService.printWSDL(AxisService.java:1056)
at org.apache.axis2.transport.http.ListingAgent.processListService(ListingAgent.java:280)
at org.apache.axis2.transport.http.AxisServlet.doGet(AxisServlet.java:229)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Caused by: Error writing stream: OutputStream encountered error during write: com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during write
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:188)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java:590)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flushBytes(BufferedServletOutputStream.java:433)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flush(BufferedServletOutputStream.java:405)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:310)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:154)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:288)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:84)
at org.apache.axis2.description.AxisService.printDefinitionObject(AxisService.java:920)
at org.apache.axis2.description.AxisService.printUserWSDL(AxisService.java:934)
at org.apache.axis2.description.AxisService.printWSDL(AxisService.java:1056)
at org.apache.axis2.transport.http.ListingAgent.processListService(ListingAgent.java:280)
at org.apache.axis2.transport.http.AxisServlet.doGet(AxisServlet.java:229)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Caused by: java.io.IOException: Async IO operation failed (1), reason: RC: 104 Connection reset by peer
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.<init>(AsyncLibrary.java:924)
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.get(AsyncLibrary.java:937)
at com.ibm.io.async.AsyncLibrary.getIOException(AsyncLibrary.java:951)
at com.ibm.io.async.AbstractAsyncChannel.multiIO(AbstractAsyncChannel.java:482)
at com.ibm.io.async.AsyncSocketChannelHelper.write(AsyncSocketChannelHelper.java:478)
at com.ibm.io.async.AsyncSocketChannelHelper.write(AsyncSocketChannelHelper.java:396)
at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.writeAIO(AioSocketIOChannel.java:282)
at com.ibm.ws.tcp.channel.impl.AioTCPWriteRequestContextImpl.processAsyncWriteRequest(AioTCPWriteRequestContextImpl.java:54)
at com.ibm.ws.tcp.channel.impl.TCPWriteRequestContextImpl.writeInternal(TCPWriteRequestContextImpl.java:382)
at com.ibm.ws.tcp.channel.impl.TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:353)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.asynchWrite(HttpServiceContextImpl.java:2518)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2302)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:901)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBufferAsynch(WCChannelLink.java:517)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBufferResponse(WCChannelLink.java:494)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBuffer(WCChannelLink.java:438)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.flushWriteBuffer(WCCByteBufferOutputStream.java:719)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:225)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java:590)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flushBytes(BufferedServletOutputStream.java:433)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flush(BufferedServletOutputStream.java:405)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:310)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:154)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:288)
at java.io.PrintWriter.flush(PrintWriter.java:331)
at oracle.j2ee.ws.wsdl.extensions.AbstractSerializer.endMarshall(AbstractSerializer.java:47)
at oracle.j2ee.ws.wsdl.extensions.soap.SOAPOperationSerializer.marshall(SOAPOperationSerializer.java:47)
at oracle.j2ee.ws.wsdl.extensions.ParseUtils.writeExtensibilityElements(ParseUtils.java:128)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeBindingOperation(WSDLWriterImpl.java:340)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeBinding(WSDLWriterImpl.java:323)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeDefinition(WSDLWriterImpl.java:197)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:93)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:83)
... 30 more
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:86)
at org.apache.axis2.description.AxisService.printDefinitionObject(AxisService.java:920)
at org.apache.axis2.description.AxisService.printUserWSDL(AxisService.java:934)
... 28 more
Means the other side most likely (as in very very most likely) closed the connection during the file transfer/data IO.

403 Forbidden (RP) encountered a failure during the login when configuring App ID with tWAS

I'm trying to configure integration between App ID and tWAS. I've followed the 'Overview' of App ID service and successfully configured with Liberty using provided example, but same config is failing for tWAS. What I already did:
Configured TAI with the following props:
provider_1.identifier=app-idp
provider_1.clientId=xxx
provider_1.clientSecret=zzz
provider_1.authorizeEndpointUrl=https://eu-gb.appid.cloud.ibm.com/...
provider_1.tokenEndpointUrl=https://eu-gb.appid.cloud.ibm.com/oauth/...
provider_1.signatureAlgorithm=RS256
provider_1.jwkEndpointUrl= https://eu-gb.appid.cloud.ibm.com/oauth/...
provider_1.interceptedPathFilter=/appContext
provider_1.redirectToRPHostAndPort=https://host:443
provider_1.tokenEndpointAuthMethod=basic
provider_1.issuerIdentifier=https://eu-gb.appid.cloud.ibm.com/oauth...
Installed OIDCRP
When I try to log in, I'm correctly redirected to the App ID login page, but after I provide credentials, it redirects and shows 403 Forbidden.
Here is stack trace, any hints what might be wrong or what am I missing??
[4/17/19 16:51:28:016 UTC] 000000bf RelyingPartyU < getSecuredConnection returns [not null] Exit
[4/17/19 16:51:28:016 UTC] 000000bf SessionCache 3 getOpServerConnTimeout returns [20000])
[4/17/19 16:51:28:016 UTC] 000000bf RelyingParty 3 getRevokeEndpoint returns [null]
[4/17/19 16:51:28:016 UTC] 000000bf RelyingPartyU 3 isRevokeEndpoint[false]
[4/17/19 16:51:28:016 UTC] 000000bf RelyingParty 3 getTokenEndpointAuthMethod returns [basic]
[4/17/19 16:51:28:016 UTC] 000000bf RelyingPartyU 3 Setting [Authorization] property on request with basicauth info
[4/17/19 16:51:28:016 UTC] 000000bf RelyingParty 3 getClientBasicAuth returns [Basic OGExZmM2NjQtYTY3Zi00YTk4LWJmM2YtN2E0N2FmYjE4NTdlOk5XTmhaVGN5TVRFdFlUbGhaaTAwTlRkbUxXRTNOemd0TVdSa1lXVTVNelkxTXpWaw==]
[4/17/19 16:51:28:105 UTC] 000000bf RelyingPartyU 3 Response code: 403
[4/17/19 16:51:28:105 UTC] 000000bf RelyingPartyU > getData(inStream[not null]) Entry
[4/17/19 16:51:28:108 UTC] 000000bf RelyingPartyU < getData returns [<!DOCTYPE html>
....
[4/17/19 16:51:28:110 UTC] 000000bf RelyingParty E CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [ResponseCode: 403]. Check the logs for details that lead to this exception.
[4/17/19 16:51:28:111 UTC] 000000bf WebAuthentica E SECJ0126E: Trust Association failed during validation. The exception is com.ibm.websphere.security.WebTrustAssociationFailedException: CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [ResponseCode: 403]. Check the logs for details that lead to this exception.
at com.ibm.ws.security.oidc.client.RelyingParty.handleSigninCallback(RelyingParty.java:526)
at com.ibm.ws.security.oidc.client.RelyingParty.negotiateValidateandEstablishTrust(RelyingParty.java:301)
at com.ibm.ws.security.web.TAIWrapper.negotiateAndValidateEstablishedTrust(TAIWrapper.java:103)
at com.ibm.ws.security.web.WebAuthenticator.handleTrustAssociation(WebAuthenticator.java:438)
at com.ibm.ws.security.web.WebAuthenticator.authenticate(WebAuthenticator.java:3103)
at com.ibm.ws.security.web.WebCollaborator.SetAuthenticatedSubjectIfNeeded(WebCollaborator.java:3522)
at com.ibm.ws.security.web.WebCollaborator.authorize(WebCollaborator.java:851)
at com.ibm.ws.security.web.EJSWebCollaborator.preInvoke(EJSWebCollaborator.java:447)
at com.ibm.ws.webcontainer.collaborator.WebAppSecurityCollaboratorImpl.preInvoke(WebAppSecurityCollaboratorImpl.java:230)
at com.ibm.wsspi.webcontainer.collaborator.CollaboratorHelper.preInvokeCollaborators(CollaboratorHelper.java:436)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1091)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
. Make sure that the setup is correct and that the user credentials are valid.
[4/17/19 16:51:28:116 UTC] 000000bf WebCollaborat A SECJ0056E: Authentication failed for reason CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [ResponseCode: 403]. Check the logs for details that lead to this exception.
Also tried with changing auth method to
provider_1.tokenEndpointAuthMethod=post
but same result, attaching full trace from the call trace1.txt
UPDATE
After your change and some changes I did in the settings I've moved further. I've reset this setting to Basic
provider_1.tokenEndpointAuthMethod=Basic
Now the error is:
[4/18/19 12:07:58:638 UTC] 000000a6 WebCollaborat A SECJ0056E: Authentication failed for reason CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [com.ibm.ws.security.oidc.client.RelyingPartyException: Failed to validate id token, exception thrown during verify [JsonObject]]. Check the logs for details that lead to this exception.
And trace shows:
[4/18/19 12:07:58:614 UTC] 000000a6 JWT E CWWKS1756E: Validation failed for the ID token requested by [....] using the [RS256] algorithm due to a signature verification failure: [JsonObject].
[4/18/19 12:07:58:617 UTC] 000000a6 SessionData 3 Failed to validate id token, exception thrown during verify [JsonObject]
[4/18/19 12:07:58:629 UTC] 000000a6 WebAuthentica E SECJ0126E: Trust Association failed during validation. The exception is com.ibm.websphere.security.WebTrustAssociationFailedException: CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [com.ibm.ws.security.oidc.client.RelyingPartyException: Failed to validate id token, exception thrown during verify [JsonObject]]. Check the logs for details that lead to this exception.
at com.ibm.ws.security.oidc.client.RelyingParty.handleSigninCallback(RelyingParty.java:532)
at com.ibm.ws.security.oidc.client.RelyingParty.negotiateValidateandEstablishTrust(RelyingParty.java:301)
at com.ibm.ws.security.web.TAIWrapper.negotiateAndValidateEstablishedTrust(TAIWrapper.java:103)
at com.ibm.ws.security.web.WebAuthenticator.handleTrustAssociation(WebAuthenticator.java:438)
at com.ibm.ws.security.web.WebAuthenticator.authenticate(WebAuthenticator.java:3171)
at com.ibm.ws.security.web.WebCollaborator.SetAuthenticatedSubjectIfNeeded(WebCollaborator.java:3522)
at com.ibm.ws.security.web.WebCollaborator.authorize(WebCollaborator.java:851)
at com.ibm.ws.security.web.EJSWebCollaborator.preInvoke(EJSWebCollaborator.java:447)
at com.ibm.ws.webcontainer.collaborator.WebAppSecurityCollaboratorImpl.preInvoke(WebAppSecurityCollaboratorImpl.java:230)
at com.ibm.wsspi.webcontainer.collaborator.CollaboratorHelper.preInvokeCollaborators(CollaboratorHelper.java:436)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1091)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
Caused by: com.ibm.ws.security.oidc.client.RelyingPartyException: Failed to validate id token, exception thrown during verify [JsonObject]
at com.ibm.ws.security.oidc.client.SessionData.setIdToken(SessionData.java:483)
at com.ibm.ws.security.oidc.client.SessionData.processJSON(SessionData.java:236)
at com.ibm.ws.security.oidc.client.SessionData.createData(SessionData.java:166)
at com.ibm.ws.security.oidc.client.SessionData.<init>(SessionData.java:137)
at com.ibm.ws.security.oidc.client.SessionCache.createEntry(SessionCache.java:184)
at com.ibm.ws.security.oidc.client.RelyingPartyUtils.createSessionDataObject(RelyingPartyUtils.java:1299)
at com.ibm.ws.security.oidc.client.RelyingParty.handleSigninCallback(RelyingParty.java:529)
... 31 more
Caused by: java.lang.UnsupportedOperationException: JsonObject
at com.google.gson.JsonElement.getAsString(JsonElement.java:191)
at com.ibm.ws.security.openidconnect.token.JsonTokenUtil.fromJsonToken(JsonTokenUtil.java:154)
at com.ibm.ws.security.openidconnect.token.JWT.fromJsonToken(JWT.java:553)
at com.ibm.ws.security.openidconnect.token.JWT.verify(JWT.java:886)
at com.ibm.ws.security.openidconnect.token.IDToken.verify(IDToken.java:578)
at com.ibm.ws.security.oidc.client.SessionData.setIdToken(SessionData.java:471)
... 37 more
Your logs suggest the request to our /token endpoint has been blocked following falsely identifying your server as a potential security risk.
We've slightly changed our configuration, please try again and let us know if you are still facing any issues.
Thanks,
Tal
From your logs -
...
POST Request to URL [https://eu-gb.appid.cloud.ibm.com/oauth/v4/4bef41a0-fafa-4a39-87b2-34e3d0a9a288/token
...
[4/18/19 7:56:29:187 UTC] 000000c1 RelyingPartyU < getData returns
...
<p>The owner of this website (eu-gb.appid.cloud.ibm.com) has banned your access based on your browser's signature (4c95150c9d6d9abe-ua21).</p>
A bug in tWAS OIDC Client was found, which crashes it when parsing non-flat JSON structure JWTs. This is being addressed by the tWAS team.

Access Remote EJB in one Websphere profile from another WebSphere profile

In short: I make a remote JNDI call for a stateless bean from one Websphere server to another (different profiles in different JVMs on the same machine) but after the call is done, the InitialContext gets changed and I can no longer access my local beans.
The situation is as follows:
A Websphere server Server1 has a complex application with many beans and also needs to access a stateless bean (BeanX) on Websphere Server2 that runs another application.
I have managed to access BeanX on Server2 using either of the two methods:
Connection via code:
private static Context;
static {
Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, com.ibm.websphere.naming.WsnInitialContextFactory");
env.put(Context.PROVIDER_URL, "iiop://localhost:9101");
try {
ctx = NamingManager.getInitialContext(env);
mgr = (RemoteBeanXManager) PortableRemoteObject.narrow(ctx.lookup("BeanXManagerBean/remote"), RemoteBeanXManager.class);
} catch (NamingException e) {//logging
}
}
Connection via Webshere configuration
Configured in Server1: Environment -> Naming -> Name Space binding -> New ... Indirect
And introduced the required values, corresponding to the exposed bean from Server2.
Provider URL: corbaloc:iiop:localhost:9101
Initial context factory name: com.ibm.websphere.naming.WsnInitialContextFactory
And then proceeded to obtain the bean in the code as if it was in my local context.
I removed security for RMI/IIOP for Outbound in Server1 and for Inbound in Server2, so that the servers can communicate.
The bean is exposed by Server2, the remote interface RemoteBeanXManager exists in a common package for both applications and the call goes through correctly.
The problem appears after the call, when subsequent EJB injections in Server1 for its local beans return a com.ibm.ejs.container.util.ExceptionUtil.NoSuchEJBException
Basically, at the next #EJB annotation encountered, Server1 will throw:
com.ibm.ejs.container.EJBNotFoundException: EJB named SomeOtherBean
not present in application My Enterprise Application. at
com.ibm.ejs.container.HomeOfHomes.resolveEJBLink(HomeOfHomes.java:751)
What I managed to understand as I investigate, is that the InitialContext on Server1 is changed by the connection to Server2, and cannot 'reset' back after the call through the remote interface. I tried re-obtaining the InitialContext with the default settings after the remote manager is obtained, but to no avail.
I tried to add the optional configuration parameters:
com.ibm.websphere.naming.jndicache.cachename providerURL
com.ibm.websphere.naming.jndicache.cacheobject none
Still the same exception.
If you have any ideas on how to isolate/cache the context on Server1 so that it can resume normal operation after a RMI-IIOP call, please let me know
Updated to add:
When I comment out the code that performs the remote call, the application starts correctly, and all #EJB injections work.
In Server2, at the call of the method from Server1 - no exception, just these messages:
[11/27/18 9:28:13:525 EET] 00000139 BeanXManagerB I com.my.app.manager.impl.BeanXManagerBean performTaskForServer1 End
[11/27/18 9:28:13:883 EET] 00000139 RegisteredRes E WTRN0064E: An illegal attempt to commit a one phase capable resource in a subordinate transaction branch has occurred.
[11/27/18 9:28:13:920 EET] 00000139 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile2\logs\ffdc\Server2_db3b4f88_18.11.27_09.28.13.8878526370681854823401.txt com.ibm.tx.jta.TransactionImpl.prepareResources 467
[11/27/18 9:28:13:932 EET] 00000139 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile2\logs\ffdc\Server2_db3b4f88_18.11.27_09.28.13.9214065828085434219239.txt com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505
The complete stacktrace of Server1's error is:
[I thought it would be too long to post all the incident reports. ]
[11/27/18 9:28:13:946 EET] 0000011a DMAdapter I com.ibm.ws.ffdc.impl.DMAdapter getAnalysisEngine FFDC1009I: Analysis Engine using data base: D:\programs\IBM\WebSphere\AppServer\properties\logbr\ffdc\adv\ffdcdb.xml
[11/27/18 9:28:13:967 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9411059231941807719848.txt com.ibm.tx.jta.impl.RegisteredResources.prepareResource 1216
[11/27/18 9:28:13:991 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9683456920548561326798.txt com.ibm.tx.jta.TransactionImpl.prepareResources 467
[11/27/18 9:28:14:014 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9913497264494769300154.txt com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505
[11/27/18 9:28:14:032 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0285054752089388843670.txt com.ibm.ejs.csi.TranStrategy.commit 294
[11/27/18 9:28:14:039 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.036797394156202410033.txt com.ibm.ejs.container.EJSHome.createBeanO 1047
[11/27/18 9:28:14:044 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0403016172875256112876.txt com.ibm.ejs.csi.EJBApplicationMetaData.createStartupBeans 6921
[11/27/18 9:28:14:044 EET] 0000011a EJBApplicatio E CNTR0190E: The StarterBean startup singleton session bean in the app1.jar module failed initialization with exception:
javax.ejb.NoSuchEJBException: An error occurred during initialization of singleton session bean MY Enterprise Application#app1.jar#StarterBean, resulting in the discarding of the singleton instance.; nested exception is: javax.ejb.EJBTransactionRolledbackException: nested exception is: javax.transaction.RollbackException
at com.ibm.ejs.container.util.ExceptionUtil.NoSuchEJBException(ExceptionUtil.java:540)
at com.ibm.ejs.container.EJSHome.createSingletonBeanO(EJSHome.java:3752)
at com.ibm.ejs.csi.EJBApplicationMetaData.createStartupBeans(EJBApplicationMetaData.java:959)
at com.ibm.ejs.csi.EJBApplicationMetaData.startedModule(EJBApplicationMetaData.java:680)
at com.ibm.ws.runtime.component.EJBContainerImpl.stateChanged(EJBContainerImpl.java:4525)
at com.ibm.ws.runtime.component.ApplicationMgrImpl$ComparableDeployedObjectListener.stateChanged(ApplicationMgrImpl.java:2652)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stateChanged(ApplicationMgrImpl.java:1178)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectEvent(DeployedApplicationImpl.java:1558)
at com.ibm.ws.runtime.component.DeployedModuleImpl.setState(DeployedModuleImpl.java:252)
at com.ibm.ws.runtime.component.DeployedModuleImpl.setState(DeployedModuleImpl.java:248)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1150)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:800)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1450)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2311)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5574)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5700)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1340)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:83)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:287)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1263)
at java.security.AccessController.doPrivileged(AccessController.java:666)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1257)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1096)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:813)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1350)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1243)
at com.ibm.ws.management.application.sync.StartDeploymentTask.startDeployment(StartDeploymentTask.java:249)
at com.ibm.ws.management.application.sync.StartDeploymentTask.fullAppUpdate(StartDeploymentTask.java:121)
at com.ibm.ws.management.application.sync.StartDeploymentTask.performTask(StartDeploymentTask.java:109)
at com.ibm.ws.management.application.sync.AppBinaryProcessor$ExpandApp.expand(AppBinaryProcessor.java:1770)
at com.ibm.ws.management.application.sync.AppBinaryProcessor.postProcessSynchronousExt(AppBinaryProcessor.java:811)
at com.ibm.ws.management.bla.sync.BLABinaryProcessor.postProcess(BLABinaryProcessor.java:599)
at com.ibm.ws.management.bla.sync.BLABinaryProcessor.onChangeCompletion(BLABinaryProcessor.java:476)
at com.ibm.ws.management.bla.sync.BinaryProcessorWrapper.onChangeCompletion(BinaryProcessorWrapper.java:109)
at com.ibm.ws.management.repository.FileRepository.postNotify(FileRepository.java:1938)
at com.ibm.ws.management.repository.FileRepository.update(FileRepository.java:1442)
at com.ibm.ws.management.repository.client.LocalConfigRepositoryClient.update(LocalConfigRepositoryClient.java:189)
at com.ibm.ws.sm.workspace.impl.WorkSpaceMasterRepositoryAdapter.update(WorkSpaceMasterRepositoryAdapter.java:667)
at com.ibm.ws.sm.workspace.impl.RepositoryContextImpl.update(RepositoryContextImpl.java:1998)
at com.ibm.ws.sm.workspace.impl.RepositoryContextImpl.synch(RepositoryContextImpl.java:1946)
at com.ibm.ws.sm.workspace.impl.WorkSpaceImpl.synch(WorkSpaceImpl.java:549)
at com.ibm.ws.console.core.action.SyncWorkSpaceAction$1.run(SyncWorkSpaceAction.java:284)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5574)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5700)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.console.core.action.SyncWorkSpaceAction.execute(SyncWorkSpaceAction.java:288)
at org.apache.struts.action.RequestProcessor.processActionPerform(Unknown Source)
at org.apache.struts.action.RequestProcessor.process(Unknown Source)
at org.apache.struts.action.ActionServlet.process(Unknown Source)
at org.apache.struts.action.ActionServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistance(WSCUrlFilter.java:971)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:518)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:339)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:969)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1109)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
Caused by: javax.ejb.EJBTransactionRolledbackException: nested exception is: javax.transaction.RollbackException
at com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapCSIException(BusinessExceptionMappingStrategy.java:152)
at com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapCSITransactionRolledBackException(BusinessExceptionMappingStrategy.java:613)
at com.ibm.ejs.container.EJSDeployedSupport.mapCSITransactionRolledBackException(EJSDeployedSupport.java:609)
at com.ibm.ejs.container.EJSContainer.postInvokeRolledbackException(EJSContainer.java:4392)
at com.ibm.ejs.container.EJSContainer.postInvokeForLifecycleInterceptors(EJSContainer.java:4584)
at com.ibm.ejs.container.SingletonBeanO.callTransactionalLifecycleInterceptors(SingletonBeanO.java:248)
at com.ibm.ejs.container.SingletonBeanO.initialize(SingletonBeanO.java:330)
at com.ibm.ejs.container.BeanOFactory.create(BeanOFactory.java:105)
at com.ibm.ejs.container.EJSHome.createSingletonBeanO(EJSHome.java:3738)
... 101 more
Caused by: javax.transaction.RollbackException
at com.ibm.tx.jta.impl.TransactionImpl.stage3CommitProcessing(TransactionImpl.java:1279)
at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:1053)
at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:974)
at com.ibm.ws.tx.jta.TranManagerImpl.commit(TranManagerImpl.java:439)
at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:191)
at com.ibm.ejs.csi.TranStrategy.commit(TranStrategy.java:866)
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:188)
at com.ibm.ejs.csi.RequiresNew.postInvoke(RequiresNew.java:118)
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:482)
at com.ibm.ejs.container.EJSContainer.postInvokeForLifecycleInterceptors(EJSContainer.java:4576)
... 105 more
[11/27/18 9:28:14:051 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0481845564063299454983.txt com.ibm.ws.runtime.component.EJBContainerImpl.stateChanged 6637
HomeOfHomes.resolveEJBLink is normally used for #EJB annotations that are resolved locally; an InitialContext is not used, nor is a JNDI lookup performed. The code processing the #EJB skips JNDI entirely and locates the EJB directly in the EJB Container. The fact the failure occurs after a remote lookup could just be a coincidence.
I recommend looking at the beanName parameter of the #EJB and verifying that the beanName does in fact match a bean name in the same application. If the beanName parameter is of the format myModule.jar#SomeOtherBean or myModule/SomeOtherBean, then make sure the module name is correct as well. The names myModule and SomeOtherBean must match the names on one of the CNTR0167I messages in SystemOut.log, in the same application.
For WebSphere, resolving an EJB based on the beanName parameter only works within the same application. If the #EJB is referencing an EJB in a different application, even if that application is on the same server process, then you must provide a binding that identifies the JNDI name of the referenced EJB. The binding would be specified in one of the files, ibm-ejb-jar-bnd.xml/xmi or ibm-web-bnd.xml/xmi. A binding may also be provided during application deployment (and the deployment would add it to the appropriate *bnd.xml file).
If the beanName parameter does appear to be correct, and a JNDI lookup is occurring for the #EJB annotation, then you'll need to provide the full exception stacks for both the client and server side of the lookup, to better understand how the #EJB processes was re-directed to the wrong server. If this is occurring, then the EJBNotFoundException should appear in the Server2 logs, and then flow back to Server1, where it would be reported again; so exception stacks from both server1 and server2 would be needed to identify how the lookup was incorrectly routed.

WSVR0009E: Error occurred during startup, org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4

I am trying to ripple start my cluster, but getting below error:
[10/25/18 20:30:31:311 CEST] 00000001 WsServerImpl E WSVR0009E: Error occurred during startup
com.ibm.ws.exception.RuntimeError: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4 vmcid: 0x49421000 minor code: 56 completed: No
at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:490)
at com.ibm.ws.runtime.component.ContainerHelper.startComponents(ContainerHelper.java:540)
at com.ibm.ws.runtime.component.ContainerImpl.startComponents(ContainerImpl.java:627)
at com.ibm.ws.runtime.component.ContainerImpl.start(ContainerImpl.java:618)
at com.ibm.ws.runtime.component.ServerImpl.start(ServerImpl.java:555)
at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:311)
at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:224)
at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:697)
at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:101)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
at org.eclipse.core.launcher.Main.run(Main.java:981)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:413)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:174)
Caused by: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4 vmcid: 0x49421000 minor code: 56 completed: No
at com.ibm.ws.orbimpl.transport.WSTransport.createListener(WSTransport.java:867)
at com.ibm.ws.orbimpl.transport.WSTransport.initTransports(WSTransport.java:605)
at com.ibm.rmi.iiop.TransportManager.initTransports(TransportManager.java:157)
at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1362)
at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1697)
at org.omg.CORBA.ORB.init(ORB.java:473)
at com.ibm.ws.orb.GlobalORBFactory.init(GlobalORBFactory.java:95)
at com.ibm.ejs.oa.EJSORBImpl.initializeORB(EJSORBImpl.java:169)
at com.ibm.ejs.oa.EJSServerORBImpl.(EJSServerORBImpl.java:88)
at com.ibm.ejs.oa.EJSORB.init(EJSORB.java:50)
at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:482)
... 34 more
As per other posts in stackoverflow, this is problem with bootstrap port. As I am using cluster, I assume default ports will be 9809/9810. But, there is no entry for these ports in /etc/services.
How I can proceed here? I also tried to start individual servers, but got same error. netstat do not have any entry for these ports too.
/etc/services do have an entry for 2809, but I am using cluster. SO, I was not concerned about 2809.

SQLMX/IBM-liberty-datasource: Valid jdbc driver not found

I'm getting the below jdbc driver not found error, though I've included t4sqlmx-3.3.jar in <liberty_server>/lib folder. What additional changes are required?
<library id="sqlmx-lib">
<fileset dir="lib" includes="t4sqlmx-3.3.jar"/>
</library>
Error :
Caused by: java.sql.SQLNonTransientException: DSRA4000E: A valid JDBC driver implementation class was not found for the jdbcDriver dataSource[default-0]/jdbcDriver[default-0] using the library sqlmx-lib. []
at com.ibm.ws.jdbc.internal.JDBCDriverService.classNotFound(JDBCDriverService.java:207)
at com.ibm.ws.jdbc.internal.JDBCDriverService.createAnyDataSource(JDBCDriverService.java:368)
at com.ibm.ws.jdbc.DataSourceService.init(DataSourceService.java:592)
at com.ibm.ws.jca.cm.AbstractConnectionFactoryService$2.run(AbstractConnectionFactoryService.java:460)
at com.ibm.ws.jca.cm.AbstractConnectionFactoryService$2.run(AbstractConnectionFactoryService.java:457)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.ws.jca.cm.AbstractConnectionFactoryService.initPrivileged(AbstractConnectionFactoryService.java:457)
at com.ibm.ws.jca.cm.AbstractConnectionFactoryService.createResource(AbstractConnectionFactoryService.java:144)
at com.ibm.ws.resource.internal.ResourceFactoryTrackerData$1.getService(ResourceFactoryTrackerData.java:120)
... 107 more
After datasource fix, some sqlmx specific error:
[12/05/18 14:39:26:473 EDT] 000003f8 SystemErr R Caused by: java.sql.SQLException: Pool requests blocked for dataSource[default-0]/connectionManager, connection pool is being shut down.
[12/05/18 14:39:26:473 EDT] 000003f8 SystemErr R at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:822)
[12/05/18 14:39:26:473 EDT] 000003f8 SystemErr R at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:152)
[12/05/18 14:39:26:474 EDT] 000003f8 SystemErr R at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:116)
[12/05/18 14:39:26:474 EDT] 000003f8 SystemErr R at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
[12/05/18 14:39:26:474 EDT] 000003f8 SystemErr R at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
[12/05/18 14:39:26:474 EDT] 000003f8 SystemErr R ... 95 more
[12/05/18 14:39:26:474 EDT] 000003f8 SystemErr R Caused by: javax.resource.spi.ResourceAllocationException: Pool requests blocked for dataSource[default-0]/connectionManager, connection pool is being shut down.
[12/05/18 14:39:26:475 EDT] 000003f8 SystemErr R at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:1275)
[12/05/18 14:39:26:475 EDT] 000003f8 SystemErr R at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:573)
[12/05/18 14:39:26:475 EDT] 000003f8 SystemErr R at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:309)
[12/05/18 14:39:26:475 EDT] 000003f8 SystemErr R at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:143)
[12/05/18 14:39:26:475 EDT] 000003f8 SystemErr R ... 98 more
Based on the stack trace, it appears that you have configured a <dataSource> element in your server.xml, but not included it in your question. Using the <library> configuration you did provide, your configuration might look like this:
<dataSource jndiName="jdbc/myDB" type="javax.sql.XADataSource">
<jdbcDriver libraryRef="sqlmx-lib" javax.sql.XADataSource="com.tandem.t4jdbc.SQLMXDataSource"/>
<properties databaseName="SAMPLEDB" hostName="localhost" port="12345"/>
</dataSource>
For more information on how to configure a DataSource in Liberty, see:
Configuring relational database connectivity in Liberty
NOTE: Based on googling for the SQL/MX JDBC driver documentation, I found that their XADataSource implementation class name is com.tandem.t4jdbc.SQLMXDataSource. If that class name is not correct for your JDBC driver jar, then use whatever class your jar has. Also, if your driver does not provide any implementation of javax.sql.XADataSource, then change the configuration to be javax.sql.ConnectionPoolDataSource instead.

Resources