403 Forbidden (RP) encountered a failure during the login when configuring App ID with tWAS - websphere

I'm trying to configure integration between App ID and tWAS. I've followed the 'Overview' of App ID service and successfully configured with Liberty using provided example, but same config is failing for tWAS. What I already did:
Configured TAI with the following props:
provider_1.identifier=app-idp
provider_1.clientId=xxx
provider_1.clientSecret=zzz
provider_1.authorizeEndpointUrl=https://eu-gb.appid.cloud.ibm.com/...
provider_1.tokenEndpointUrl=https://eu-gb.appid.cloud.ibm.com/oauth/...
provider_1.signatureAlgorithm=RS256
provider_1.jwkEndpointUrl= https://eu-gb.appid.cloud.ibm.com/oauth/...
provider_1.interceptedPathFilter=/appContext
provider_1.redirectToRPHostAndPort=https://host:443
provider_1.tokenEndpointAuthMethod=basic
provider_1.issuerIdentifier=https://eu-gb.appid.cloud.ibm.com/oauth...
Installed OIDCRP
When I try to log in, I'm correctly redirected to the App ID login page, but after I provide credentials, it redirects and shows 403 Forbidden.
Here is stack trace, any hints what might be wrong or what am I missing??
[4/17/19 16:51:28:016 UTC] 000000bf RelyingPartyU < getSecuredConnection returns [not null] Exit
[4/17/19 16:51:28:016 UTC] 000000bf SessionCache 3 getOpServerConnTimeout returns [20000])
[4/17/19 16:51:28:016 UTC] 000000bf RelyingParty 3 getRevokeEndpoint returns [null]
[4/17/19 16:51:28:016 UTC] 000000bf RelyingPartyU 3 isRevokeEndpoint[false]
[4/17/19 16:51:28:016 UTC] 000000bf RelyingParty 3 getTokenEndpointAuthMethod returns [basic]
[4/17/19 16:51:28:016 UTC] 000000bf RelyingPartyU 3 Setting [Authorization] property on request with basicauth info
[4/17/19 16:51:28:016 UTC] 000000bf RelyingParty 3 getClientBasicAuth returns [Basic OGExZmM2NjQtYTY3Zi00YTk4LWJmM2YtN2E0N2FmYjE4NTdlOk5XTmhaVGN5TVRFdFlUbGhaaTAwTlRkbUxXRTNOemd0TVdSa1lXVTVNelkxTXpWaw==]
[4/17/19 16:51:28:105 UTC] 000000bf RelyingPartyU 3 Response code: 403
[4/17/19 16:51:28:105 UTC] 000000bf RelyingPartyU > getData(inStream[not null]) Entry
[4/17/19 16:51:28:108 UTC] 000000bf RelyingPartyU < getData returns [<!DOCTYPE html>
....
[4/17/19 16:51:28:110 UTC] 000000bf RelyingParty E CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [ResponseCode: 403]. Check the logs for details that lead to this exception.
[4/17/19 16:51:28:111 UTC] 000000bf WebAuthentica E SECJ0126E: Trust Association failed during validation. The exception is com.ibm.websphere.security.WebTrustAssociationFailedException: CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [ResponseCode: 403]. Check the logs for details that lead to this exception.
at com.ibm.ws.security.oidc.client.RelyingParty.handleSigninCallback(RelyingParty.java:526)
at com.ibm.ws.security.oidc.client.RelyingParty.negotiateValidateandEstablishTrust(RelyingParty.java:301)
at com.ibm.ws.security.web.TAIWrapper.negotiateAndValidateEstablishedTrust(TAIWrapper.java:103)
at com.ibm.ws.security.web.WebAuthenticator.handleTrustAssociation(WebAuthenticator.java:438)
at com.ibm.ws.security.web.WebAuthenticator.authenticate(WebAuthenticator.java:3103)
at com.ibm.ws.security.web.WebCollaborator.SetAuthenticatedSubjectIfNeeded(WebCollaborator.java:3522)
at com.ibm.ws.security.web.WebCollaborator.authorize(WebCollaborator.java:851)
at com.ibm.ws.security.web.EJSWebCollaborator.preInvoke(EJSWebCollaborator.java:447)
at com.ibm.ws.webcontainer.collaborator.WebAppSecurityCollaboratorImpl.preInvoke(WebAppSecurityCollaboratorImpl.java:230)
at com.ibm.wsspi.webcontainer.collaborator.CollaboratorHelper.preInvokeCollaborators(CollaboratorHelper.java:436)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1091)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
. Make sure that the setup is correct and that the user credentials are valid.
[4/17/19 16:51:28:116 UTC] 000000bf WebCollaborat A SECJ0056E: Authentication failed for reason CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [ResponseCode: 403]. Check the logs for details that lead to this exception.
Also tried with changing auth method to
provider_1.tokenEndpointAuthMethod=post
but same result, attaching full trace from the call trace1.txt
UPDATE
After your change and some changes I did in the settings I've moved further. I've reset this setting to Basic
provider_1.tokenEndpointAuthMethod=Basic
Now the error is:
[4/18/19 12:07:58:638 UTC] 000000a6 WebCollaborat A SECJ0056E: Authentication failed for reason CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [com.ibm.ws.security.oidc.client.RelyingPartyException: Failed to validate id token, exception thrown during verify [JsonObject]]. Check the logs for details that lead to this exception.
And trace shows:
[4/18/19 12:07:58:614 UTC] 000000a6 JWT E CWWKS1756E: Validation failed for the ID token requested by [....] using the [RS256] algorithm due to a signature verification failure: [JsonObject].
[4/18/19 12:07:58:617 UTC] 000000a6 SessionData 3 Failed to validate id token, exception thrown during verify [JsonObject]
[4/18/19 12:07:58:629 UTC] 000000a6 WebAuthentica E SECJ0126E: Trust Association failed during validation. The exception is com.ibm.websphere.security.WebTrustAssociationFailedException: CWTAI2007E: The OpenID Connect relying party (RP) encountered a failure during the login. The exception is [com.ibm.ws.security.oidc.client.RelyingPartyException: Failed to validate id token, exception thrown during verify [JsonObject]]. Check the logs for details that lead to this exception.
at com.ibm.ws.security.oidc.client.RelyingParty.handleSigninCallback(RelyingParty.java:532)
at com.ibm.ws.security.oidc.client.RelyingParty.negotiateValidateandEstablishTrust(RelyingParty.java:301)
at com.ibm.ws.security.web.TAIWrapper.negotiateAndValidateEstablishedTrust(TAIWrapper.java:103)
at com.ibm.ws.security.web.WebAuthenticator.handleTrustAssociation(WebAuthenticator.java:438)
at com.ibm.ws.security.web.WebAuthenticator.authenticate(WebAuthenticator.java:3171)
at com.ibm.ws.security.web.WebCollaborator.SetAuthenticatedSubjectIfNeeded(WebCollaborator.java:3522)
at com.ibm.ws.security.web.WebCollaborator.authorize(WebCollaborator.java:851)
at com.ibm.ws.security.web.EJSWebCollaborator.preInvoke(EJSWebCollaborator.java:447)
at com.ibm.ws.webcontainer.collaborator.WebAppSecurityCollaboratorImpl.preInvoke(WebAppSecurityCollaboratorImpl.java:230)
at com.ibm.wsspi.webcontainer.collaborator.CollaboratorHelper.preInvokeCollaborators(CollaboratorHelper.java:436)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1091)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
Caused by: com.ibm.ws.security.oidc.client.RelyingPartyException: Failed to validate id token, exception thrown during verify [JsonObject]
at com.ibm.ws.security.oidc.client.SessionData.setIdToken(SessionData.java:483)
at com.ibm.ws.security.oidc.client.SessionData.processJSON(SessionData.java:236)
at com.ibm.ws.security.oidc.client.SessionData.createData(SessionData.java:166)
at com.ibm.ws.security.oidc.client.SessionData.<init>(SessionData.java:137)
at com.ibm.ws.security.oidc.client.SessionCache.createEntry(SessionCache.java:184)
at com.ibm.ws.security.oidc.client.RelyingPartyUtils.createSessionDataObject(RelyingPartyUtils.java:1299)
at com.ibm.ws.security.oidc.client.RelyingParty.handleSigninCallback(RelyingParty.java:529)
... 31 more
Caused by: java.lang.UnsupportedOperationException: JsonObject
at com.google.gson.JsonElement.getAsString(JsonElement.java:191)
at com.ibm.ws.security.openidconnect.token.JsonTokenUtil.fromJsonToken(JsonTokenUtil.java:154)
at com.ibm.ws.security.openidconnect.token.JWT.fromJsonToken(JWT.java:553)
at com.ibm.ws.security.openidconnect.token.JWT.verify(JWT.java:886)
at com.ibm.ws.security.openidconnect.token.IDToken.verify(IDToken.java:578)
at com.ibm.ws.security.oidc.client.SessionData.setIdToken(SessionData.java:471)
... 37 more

Your logs suggest the request to our /token endpoint has been blocked following falsely identifying your server as a potential security risk.
We've slightly changed our configuration, please try again and let us know if you are still facing any issues.
Thanks,
Tal
From your logs -
...
POST Request to URL [https://eu-gb.appid.cloud.ibm.com/oauth/v4/4bef41a0-fafa-4a39-87b2-34e3d0a9a288/token
...
[4/18/19 7:56:29:187 UTC] 000000c1 RelyingPartyU < getData returns
...
<p>The owner of this website (eu-gb.appid.cloud.ibm.com) has banned your access based on your browser's signature (4c95150c9d6d9abe-ua21).</p>

A bug in tWAS OIDC Client was found, which crashes it when parsing non-flat JSON structure JWTs. This is being addressed by the tWAS team.

Related

Getting the following exception when trying to render a large PDF in Oracle EWPS

While trying to process a large PDF file I get the following stack trace. This is only happening when we try a large PDF and does not impact the majority of work. This is Oracle Enterprise Web Processing Services (EWPS) Axis2 running on WebSphere Application Server version 8.5.5.13. We think it is an issue with a timeout setting but are not sure what setting to modify.
[5/10/19 3:38:50:458 CDT] 000000b1 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause AxisServlet: org.apache.axis2.AxisFault: Error writing stream: OutputStream encountered error during write
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:417)
at org.apache.axis2.description.AxisService.printUserWSDL(AxisService.java:936)
at org.apache.axis2.description.AxisService.printWSDL(AxisService.java:1056)
at org.apache.axis2.transport.http.ListingAgent.processListService(ListingAgent.java:280)
at org.apache.axis2.transport.http.AxisServlet.doGet(AxisServlet.java:229)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Caused by: Error writing stream: OutputStream encountered error during write: com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during write
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:188)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java:590)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flushBytes(BufferedServletOutputStream.java:433)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flush(BufferedServletOutputStream.java:405)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:310)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:154)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:288)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:84)
at org.apache.axis2.description.AxisService.printDefinitionObject(AxisService.java:920)
at org.apache.axis2.description.AxisService.printUserWSDL(AxisService.java:934)
at org.apache.axis2.description.AxisService.printWSDL(AxisService.java:1056)
at org.apache.axis2.transport.http.ListingAgent.processListService(ListingAgent.java:280)
at org.apache.axis2.transport.http.AxisServlet.doGet(AxisServlet.java:229)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Caused by: java.io.IOException: Async IO operation failed (1), reason: RC: 104 Connection reset by peer
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.<init>(AsyncLibrary.java:924)
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.get(AsyncLibrary.java:937)
at com.ibm.io.async.AsyncLibrary.getIOException(AsyncLibrary.java:951)
at com.ibm.io.async.AbstractAsyncChannel.multiIO(AbstractAsyncChannel.java:482)
at com.ibm.io.async.AsyncSocketChannelHelper.write(AsyncSocketChannelHelper.java:478)
at com.ibm.io.async.AsyncSocketChannelHelper.write(AsyncSocketChannelHelper.java:396)
at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.writeAIO(AioSocketIOChannel.java:282)
at com.ibm.ws.tcp.channel.impl.AioTCPWriteRequestContextImpl.processAsyncWriteRequest(AioTCPWriteRequestContextImpl.java:54)
at com.ibm.ws.tcp.channel.impl.TCPWriteRequestContextImpl.writeInternal(TCPWriteRequestContextImpl.java:382)
at com.ibm.ws.tcp.channel.impl.TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:353)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.asynchWrite(HttpServiceContextImpl.java:2518)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2302)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:901)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBufferAsynch(WCChannelLink.java:517)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBufferResponse(WCChannelLink.java:494)
at com.ibm.ws.webcontainer.channel.WCChannelLink.writeBuffer(WCChannelLink.java:438)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.flushWriteBuffer(WCCByteBufferOutputStream.java:719)
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:225)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java:590)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flushBytes(BufferedServletOutputStream.java:433)
at com.ibm.wsspi.webcontainer.util.BufferedServletOutputStream.flush(BufferedServletOutputStream.java:405)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:310)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:154)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:288)
at java.io.PrintWriter.flush(PrintWriter.java:331)
at oracle.j2ee.ws.wsdl.extensions.AbstractSerializer.endMarshall(AbstractSerializer.java:47)
at oracle.j2ee.ws.wsdl.extensions.soap.SOAPOperationSerializer.marshall(SOAPOperationSerializer.java:47)
at oracle.j2ee.ws.wsdl.extensions.ParseUtils.writeExtensibilityElements(ParseUtils.java:128)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeBindingOperation(WSDLWriterImpl.java:340)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeBinding(WSDLWriterImpl.java:323)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeDefinition(WSDLWriterImpl.java:197)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:93)
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:83)
... 30 more
at oracle.j2ee.ws.wsdl.xml.WSDLWriterImpl.writeWSDL(WSDLWriterImpl.java:86)
at org.apache.axis2.description.AxisService.printDefinitionObject(AxisService.java:920)
at org.apache.axis2.description.AxisService.printUserWSDL(AxisService.java:934)
... 28 more
Means the other side most likely (as in very very most likely) closed the connection during the file transfer/data IO.

Access Remote EJB in one Websphere profile from another WebSphere profile

In short: I make a remote JNDI call for a stateless bean from one Websphere server to another (different profiles in different JVMs on the same machine) but after the call is done, the InitialContext gets changed and I can no longer access my local beans.
The situation is as follows:
A Websphere server Server1 has a complex application with many beans and also needs to access a stateless bean (BeanX) on Websphere Server2 that runs another application.
I have managed to access BeanX on Server2 using either of the two methods:
Connection via code:
private static Context;
static {
Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, com.ibm.websphere.naming.WsnInitialContextFactory");
env.put(Context.PROVIDER_URL, "iiop://localhost:9101");
try {
ctx = NamingManager.getInitialContext(env);
mgr = (RemoteBeanXManager) PortableRemoteObject.narrow(ctx.lookup("BeanXManagerBean/remote"), RemoteBeanXManager.class);
} catch (NamingException e) {//logging
}
}
Connection via Webshere configuration
Configured in Server1: Environment -> Naming -> Name Space binding -> New ... Indirect
And introduced the required values, corresponding to the exposed bean from Server2.
Provider URL: corbaloc:iiop:localhost:9101
Initial context factory name: com.ibm.websphere.naming.WsnInitialContextFactory
And then proceeded to obtain the bean in the code as if it was in my local context.
I removed security for RMI/IIOP for Outbound in Server1 and for Inbound in Server2, so that the servers can communicate.
The bean is exposed by Server2, the remote interface RemoteBeanXManager exists in a common package for both applications and the call goes through correctly.
The problem appears after the call, when subsequent EJB injections in Server1 for its local beans return a com.ibm.ejs.container.util.ExceptionUtil.NoSuchEJBException
Basically, at the next #EJB annotation encountered, Server1 will throw:
com.ibm.ejs.container.EJBNotFoundException: EJB named SomeOtherBean
not present in application My Enterprise Application. at
com.ibm.ejs.container.HomeOfHomes.resolveEJBLink(HomeOfHomes.java:751)
What I managed to understand as I investigate, is that the InitialContext on Server1 is changed by the connection to Server2, and cannot 'reset' back after the call through the remote interface. I tried re-obtaining the InitialContext with the default settings after the remote manager is obtained, but to no avail.
I tried to add the optional configuration parameters:
com.ibm.websphere.naming.jndicache.cachename providerURL
com.ibm.websphere.naming.jndicache.cacheobject none
Still the same exception.
If you have any ideas on how to isolate/cache the context on Server1 so that it can resume normal operation after a RMI-IIOP call, please let me know
Updated to add:
When I comment out the code that performs the remote call, the application starts correctly, and all #EJB injections work.
In Server2, at the call of the method from Server1 - no exception, just these messages:
[11/27/18 9:28:13:525 EET] 00000139 BeanXManagerB I com.my.app.manager.impl.BeanXManagerBean performTaskForServer1 End
[11/27/18 9:28:13:883 EET] 00000139 RegisteredRes E WTRN0064E: An illegal attempt to commit a one phase capable resource in a subordinate transaction branch has occurred.
[11/27/18 9:28:13:920 EET] 00000139 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile2\logs\ffdc\Server2_db3b4f88_18.11.27_09.28.13.8878526370681854823401.txt com.ibm.tx.jta.TransactionImpl.prepareResources 467
[11/27/18 9:28:13:932 EET] 00000139 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile2\logs\ffdc\Server2_db3b4f88_18.11.27_09.28.13.9214065828085434219239.txt com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505
The complete stacktrace of Server1's error is:
[I thought it would be too long to post all the incident reports. ]
[11/27/18 9:28:13:946 EET] 0000011a DMAdapter I com.ibm.ws.ffdc.impl.DMAdapter getAnalysisEngine FFDC1009I: Analysis Engine using data base: D:\programs\IBM\WebSphere\AppServer\properties\logbr\ffdc\adv\ffdcdb.xml
[11/27/18 9:28:13:967 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9411059231941807719848.txt com.ibm.tx.jta.impl.RegisteredResources.prepareResource 1216
[11/27/18 9:28:13:991 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9683456920548561326798.txt com.ibm.tx.jta.TransactionImpl.prepareResources 467
[11/27/18 9:28:14:014 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9913497264494769300154.txt com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505
[11/27/18 9:28:14:032 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0285054752089388843670.txt com.ibm.ejs.csi.TranStrategy.commit 294
[11/27/18 9:28:14:039 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.036797394156202410033.txt com.ibm.ejs.container.EJSHome.createBeanO 1047
[11/27/18 9:28:14:044 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0403016172875256112876.txt com.ibm.ejs.csi.EJBApplicationMetaData.createStartupBeans 6921
[11/27/18 9:28:14:044 EET] 0000011a EJBApplicatio E CNTR0190E: The StarterBean startup singleton session bean in the app1.jar module failed initialization with exception:
javax.ejb.NoSuchEJBException: An error occurred during initialization of singleton session bean MY Enterprise Application#app1.jar#StarterBean, resulting in the discarding of the singleton instance.; nested exception is: javax.ejb.EJBTransactionRolledbackException: nested exception is: javax.transaction.RollbackException
at com.ibm.ejs.container.util.ExceptionUtil.NoSuchEJBException(ExceptionUtil.java:540)
at com.ibm.ejs.container.EJSHome.createSingletonBeanO(EJSHome.java:3752)
at com.ibm.ejs.csi.EJBApplicationMetaData.createStartupBeans(EJBApplicationMetaData.java:959)
at com.ibm.ejs.csi.EJBApplicationMetaData.startedModule(EJBApplicationMetaData.java:680)
at com.ibm.ws.runtime.component.EJBContainerImpl.stateChanged(EJBContainerImpl.java:4525)
at com.ibm.ws.runtime.component.ApplicationMgrImpl$ComparableDeployedObjectListener.stateChanged(ApplicationMgrImpl.java:2652)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stateChanged(ApplicationMgrImpl.java:1178)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectEvent(DeployedApplicationImpl.java:1558)
at com.ibm.ws.runtime.component.DeployedModuleImpl.setState(DeployedModuleImpl.java:252)
at com.ibm.ws.runtime.component.DeployedModuleImpl.setState(DeployedModuleImpl.java:248)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1150)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:800)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1450)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2311)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5574)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5700)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1340)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:83)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:287)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1263)
at java.security.AccessController.doPrivileged(AccessController.java:666)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1257)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1096)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:813)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1350)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1243)
at com.ibm.ws.management.application.sync.StartDeploymentTask.startDeployment(StartDeploymentTask.java:249)
at com.ibm.ws.management.application.sync.StartDeploymentTask.fullAppUpdate(StartDeploymentTask.java:121)
at com.ibm.ws.management.application.sync.StartDeploymentTask.performTask(StartDeploymentTask.java:109)
at com.ibm.ws.management.application.sync.AppBinaryProcessor$ExpandApp.expand(AppBinaryProcessor.java:1770)
at com.ibm.ws.management.application.sync.AppBinaryProcessor.postProcessSynchronousExt(AppBinaryProcessor.java:811)
at com.ibm.ws.management.bla.sync.BLABinaryProcessor.postProcess(BLABinaryProcessor.java:599)
at com.ibm.ws.management.bla.sync.BLABinaryProcessor.onChangeCompletion(BLABinaryProcessor.java:476)
at com.ibm.ws.management.bla.sync.BinaryProcessorWrapper.onChangeCompletion(BinaryProcessorWrapper.java:109)
at com.ibm.ws.management.repository.FileRepository.postNotify(FileRepository.java:1938)
at com.ibm.ws.management.repository.FileRepository.update(FileRepository.java:1442)
at com.ibm.ws.management.repository.client.LocalConfigRepositoryClient.update(LocalConfigRepositoryClient.java:189)
at com.ibm.ws.sm.workspace.impl.WorkSpaceMasterRepositoryAdapter.update(WorkSpaceMasterRepositoryAdapter.java:667)
at com.ibm.ws.sm.workspace.impl.RepositoryContextImpl.update(RepositoryContextImpl.java:1998)
at com.ibm.ws.sm.workspace.impl.RepositoryContextImpl.synch(RepositoryContextImpl.java:1946)
at com.ibm.ws.sm.workspace.impl.WorkSpaceImpl.synch(WorkSpaceImpl.java:549)
at com.ibm.ws.console.core.action.SyncWorkSpaceAction$1.run(SyncWorkSpaceAction.java:284)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5574)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5700)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.console.core.action.SyncWorkSpaceAction.execute(SyncWorkSpaceAction.java:288)
at org.apache.struts.action.RequestProcessor.processActionPerform(Unknown Source)
at org.apache.struts.action.RequestProcessor.process(Unknown Source)
at org.apache.struts.action.ActionServlet.process(Unknown Source)
at org.apache.struts.action.ActionServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistance(WSCUrlFilter.java:971)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:518)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:339)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:969)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1109)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
Caused by: javax.ejb.EJBTransactionRolledbackException: nested exception is: javax.transaction.RollbackException
at com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapCSIException(BusinessExceptionMappingStrategy.java:152)
at com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapCSITransactionRolledBackException(BusinessExceptionMappingStrategy.java:613)
at com.ibm.ejs.container.EJSDeployedSupport.mapCSITransactionRolledBackException(EJSDeployedSupport.java:609)
at com.ibm.ejs.container.EJSContainer.postInvokeRolledbackException(EJSContainer.java:4392)
at com.ibm.ejs.container.EJSContainer.postInvokeForLifecycleInterceptors(EJSContainer.java:4584)
at com.ibm.ejs.container.SingletonBeanO.callTransactionalLifecycleInterceptors(SingletonBeanO.java:248)
at com.ibm.ejs.container.SingletonBeanO.initialize(SingletonBeanO.java:330)
at com.ibm.ejs.container.BeanOFactory.create(BeanOFactory.java:105)
at com.ibm.ejs.container.EJSHome.createSingletonBeanO(EJSHome.java:3738)
... 101 more
Caused by: javax.transaction.RollbackException
at com.ibm.tx.jta.impl.TransactionImpl.stage3CommitProcessing(TransactionImpl.java:1279)
at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:1053)
at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:974)
at com.ibm.ws.tx.jta.TranManagerImpl.commit(TranManagerImpl.java:439)
at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:191)
at com.ibm.ejs.csi.TranStrategy.commit(TranStrategy.java:866)
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:188)
at com.ibm.ejs.csi.RequiresNew.postInvoke(RequiresNew.java:118)
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:482)
at com.ibm.ejs.container.EJSContainer.postInvokeForLifecycleInterceptors(EJSContainer.java:4576)
... 105 more
[11/27/18 9:28:14:051 EET] 0000011a FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0481845564063299454983.txt com.ibm.ws.runtime.component.EJBContainerImpl.stateChanged 6637
HomeOfHomes.resolveEJBLink is normally used for #EJB annotations that are resolved locally; an InitialContext is not used, nor is a JNDI lookup performed. The code processing the #EJB skips JNDI entirely and locates the EJB directly in the EJB Container. The fact the failure occurs after a remote lookup could just be a coincidence.
I recommend looking at the beanName parameter of the #EJB and verifying that the beanName does in fact match a bean name in the same application. If the beanName parameter is of the format myModule.jar#SomeOtherBean or myModule/SomeOtherBean, then make sure the module name is correct as well. The names myModule and SomeOtherBean must match the names on one of the CNTR0167I messages in SystemOut.log, in the same application.
For WebSphere, resolving an EJB based on the beanName parameter only works within the same application. If the #EJB is referencing an EJB in a different application, even if that application is on the same server process, then you must provide a binding that identifies the JNDI name of the referenced EJB. The binding would be specified in one of the files, ibm-ejb-jar-bnd.xml/xmi or ibm-web-bnd.xml/xmi. A binding may also be provided during application deployment (and the deployment would add it to the appropriate *bnd.xml file).
If the beanName parameter does appear to be correct, and a JNDI lookup is occurring for the #EJB annotation, then you'll need to provide the full exception stacks for both the client and server side of the lookup, to better understand how the #EJB processes was re-directed to the wrong server. If this is occurring, then the EJBNotFoundException should appear in the Server2 logs, and then flow back to Server1, where it would be reported again; so exception stacks from both server1 and server2 would be needed to identify how the lookup was incorrectly routed.

java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

I had the client server connection working earlier, but it stopped working since few months. Below are the logs I found.
To fix this, I have imported the server's Base-64 encoded x.509 cert into the client's IBM Websphere Keystore. But it didn't make any difference!
What could be the issue?
[1/29/18 15:25:04:701 EST] 000000be XXLogger E User: [system] App: [com.xx.xyz.atom.UvwConnector] java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at org.apache.abdera.protocol.client.AbderaClient.execute(AbderaClient.java:701)
at org.apache.abdera.protocol.client.AbderaClient.get(AbderaClient.java:216)
at org.apache.abdera.protocol.client.AbderaClient.get(AbderaClient.java:404)
at com.xx.xyz.atom.UvwConnector.callUvwGetString(UvwConnector.java:326)
at com.xx.xyz.atom.UvwConnector.callUvwLearning(UvwConnector.java:298)
at com.xx.xyz.atom.UvwConnector.getAllCourses(UvwConnector.java:42)
at com.xx.xyz.web.SearchListServlet.doPost(SearchListServlet.java:35)
at com.xx.xyz.web.SearchListServlet.doGet(SearchListServlet.java:26)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1225)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1032)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:908)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1815)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at com.ibm.jsse2.o.a(o.java:31)
at com.ibm.jsse2.o.a(o.java:37)
at com.ibm.jsse2.SSLSocketImpl.b(SSLSocketImpl.java:468)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:527)
at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:554)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:234)
at com.ibm.jsse2.k.write(k.java:24)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:76)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:134)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.apache.abdera.protocol.client.AbderaClient.execute(AbderaClient.java:688)
... 32 more
"Handshake failure" often means there is no overlap between the server's TLS level (e.g. TLS1.2) and cipher suites, and what your WebSphere can handle.
I'd run a test like https://www.ssllabs.com/ssltest/ against the server, then look at your QoP settings in WebSphere Security > SSL certificate and key management > SSL configurations

Illegal 'sec-websocket-protocol' header: Invalid input 'EOI', expected sec-websocket-protocol

I'm trying to connect on Kubernetes websocket API but I get this error message:
java.lang.IllegalStateException: Message failed with status code 400 Bad Request; Error info: ErrorInfo(Illegal 'sec-websocket-protocol' header: Invalid input 'EOI', expected sec-websocket-protocol (line 1, column 1),
^)
at akka.http.impl.engine.ws.WebSocketClientBlueprint$UpgradeStage$1$$anon$1.onPush(WebSocketClientBlueprint.scala:107)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:747)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:649)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:471)
at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:410)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618)
at akka.actor.Actor.aroundReceive(Actor.scala:502)
at akka.actor.Actor.aroundReceive$(Actor.scala:500)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:529)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
I must be missing some configuration, do you knwo where this comes from?
Cheers
Got over it by defining subprotocol to channel.k8s.io:
WebSocketRequest(
uri,
extraHeaders = List(token),
subprotocol = Option("channel.k8s.io")
)

Websphere - AJAX POST request error

I am using ajax to send data from the client to my server in JSON format and am getting the error message below. Has anybody else managed to resolved this issue ?, because it seems like a lot of people have been getting it.
com.ibm.ws.webcontainer.srt.SRTServletRequest parseParameters SRVE0133E: An error occurred while parsing parameters. {0}
java.io.IOException: SRVE0216E: post body contains less bytes than specified by content-length
at com.ibm.ws.webcontainer.servlet.RequestUtils.parsePostData(RequestUtils.java:306)
at com.ibm.ws.webcontainer.srt.SRTServletRequest.parseParameters(SRTServletRequest.java:1887)
at com.ibm.ws.webcontainer.srt.SRTServletRequest.getParameter(SRTServletRequest.java:1519)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:184)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:932)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:500)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3826)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)

Resources