Exposing Websocket via WSO2 API Manager with wss:// onwards - websocket

Question

I am trying to expose a WebSocket service via WSO2 API manager. I have a requirement that all inter-machine communication is secured, so I want to use encrypted WebSocket (wss://). I have a proof-of-concept web service that just spits out one message per second. This is not natively implementing wss:// but I have a reverse proxy (Apache) in front on the same machine which is OK.
The problem
This works:
I can do calls over wss:// to the gateway if I allow the gateway to use ws:// to call on to the actual API service.
I can do calls from the gateway machine to the actual API service machine over wss://
This doesn't:
But if I try to configure the gateway to call on over wss:// then I get a null pointer exception in the gateway.
This final one is the problem. I think it's some configuration in the gateway but can't see what. The wso2carbon.log shows:
TID: [-1] [] [2019-06-07 11:02:34,051] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Unexpected error during sending message out {org.apache.synapse.core.axis2.Axis2Sender}
java.lang.NullPointerException
at org.wso2.carbon.websocket.transport.WebsocketConnectionFactory.cacheNewConnection(WebsocketConnectionFactory.java:123)
at org.wso2.carbon.websocket.transport.WebsocketConnectionFactory.getChannelHandler(WebsocketConnectionFactory.java:79)
at org.wso2.carbon.websocket.transport.WebsocketTransportSender.sendMessage(WebsocketTransportSender.java:106)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:592)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:83)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:548)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:382)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:65)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:121)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:1005)
at org.wso2.carbon.inbound.endpoint.protocol.websocket.InboundWebsocketSourceHandler.injectToSequence(InboundWebsocketSourceHandler.java:469)
at org.wso2.carbon.inbound.endpoint.protocol.websocket.InboundWebsocketSourceHandler.handleHandshake(InboundWebsocketSourceHandler.java:182)
at org.wso2.carbon.inbound.endpoint.protocol.websocket.InboundWebsocketSourceHandler.channelRead(InboundWebsocketSourceHandler.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at org.wso2.carbon.apimgt.gateway.handlers.WebsocketInboundHandler.channelRead(WebsocketInboundHandler.java:125)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-06-07 11:02:34,057] WARN {org.apache.synapse.core.axis2.Axis2SynapseEnvironment} - Executing fault handler due to exception encountered {org.apache.synapse.core.axis2.Axis2SynapseEnvironment}
TID: [-1] [] [2019-06-07 11:02:34,058] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : AnonymousEndpoint with address wss://icarcdevapi01.internal.carus.com/notifications/notifications will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1] [] [2019-06-07 11:02:34,058] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : AnonymousEndpoint with address wss://icarcdevapi01.internal.carus.com/notifications/notifications - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Fri Jun 07 11:03:04 EEST 2019 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1] [] [2019-06-07 11:02:34,058] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE = Unexpected error during sending message out {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2019-06-07 11:02:34,123] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : AnonymousEndpoint ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
Extra Details
I added the following to the repository/conf/axis2/axis2.xml next to the transportSender for ws:
<transportSender name="wss" class="org.wso2.carbon.websocket.transport.WebsocketTransportSender">
<parameter name="ws.outflow.dispatch.sequence" locked="false">outflowDispatchSeq</parameter>
<parameter name="ws.outflow.dispatch.fault.sequence" locked="false">outflowFaultSeq</parameter>
<parameter name="ws.trust.store" locked="false">
<ws.trust.store.location>repository/resources/security/client-truststore.jks</ws.trust.store.location>
<ws.trust.store.Password>wso2carbon</ws.trust.store.Password>
</parameter>
The reverse proxy on the gateway includes:
RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule /(.*) ws://127.0.0.1:9099/$1 [P,L]
which maps all websocket connections to port 9099. I assume since proxy->wso2 is over ws:// I don't need a different port.
If I configure an API in the WSO2 Publisher pointing to my service directly (ws://myservice.example.com:8001/foo) then it works. I can call over wss:// to the gateway but it calls over unencrypted ws:// on to the API. I also have a reverse proxy on the actual API service machine. If I run my test client on the API Gateway machine and call to wss://myservice.example.com/foo then it works so the service machine end is configured OK.
But, if I put that same URL (wss://myservice.example.com/foo) as the endpoint in the API definition in the WSO2 Publisher then the client just hangs and I get the null pointer above.
We are still on version 2.1 of the API Manager

Answer 1

Possible Reason:
NullPointerException is a bug in wso2/carbon-apimgt project and that is fixed by this PR
But i have seen in your logs
org.wso2.carbon.websocket.transport.WebsocketConnectionFactory.cacheNewConnection
It is interesting with API publishing while 'TokenCacheExpiry' config is enabled in the project. Fixed with this PR
Possible Solution:
In your case i have find a sheen example Can you try to change just your port settings