I am trying to add an API Gateway to a client-server system, which communicates through WebSockets.
The Server is an audio recognition engine that sits in a remote machine, and clients are connecting to it through WebSockets ('wss://...'), sending audio files and receiving their text translate representation.
My main goal is to add a free open-source API Gateway in the middle in order to authenticate\authorize\rate limit\etc (Tried Kong\Tyk\krakenD, but they are not free).
After searching the web, I focused on Apache APISIX gateway.
As a test, I managed to connect the APISIX to a client-server which communicates by http/https and it worked fine.
Moreover, I managed to connect the client with the server on regular websocket connection (ws://) and the data transactions were successful, but no matter what I do I cant connect to the secured connection. Are there any special configurations I should edit in the dashboard?
I'm wondering if anyone knows from experience if it's possible to use
the secured WebSockets with Apache APISIX and if yes, how exactly it should be done - because the docs are missing any informative info\examples, or if there are better solutions known to my problem.
The only WebSocket reference in APISIX dashboard -
What you should do is create an SSL object (you can find it in the sidebar on the left side). See https://apisix.apache.org/docs/apisix/admin-api/#ssl to learn the APISIX SSL object.
You can see an example at https://apisix.apache.org/docs/apisix/certificate
The SSL object provides the required TLS Certificate and Private Key so that this key pair can be used in a TLS handshake according to the TLS SNI sent from clients. After that, your client established a secured connection with APISIX, and now you can send the WebSocket traffic securely.
Related
I am trying to implement a GraphQL WebSocket-based #subscription on a server (using NestJS #subscription). The server is hosted on an AWS ECS and is behind an ALB.
We currently have an AWS API GW connection via VPC-link to our ALB.
I tried to build a dedicated Websocket API GW with the same VPC link we use in the HTTP API GW.
I also tried to spin up a new NLB (Network Load Balancer) over our ECS and a new REST VPC link to be used in the dedicated Websocket API GW.
The client and server are communicating over a graphql-transport-ws sub-protocol using graphql-ws library and the communication is working fine on a localhost setup.
When running the following command on our local host I am able to establish a web socket connection:
wscat -c ws://localhost:3000/graphql -s graphql-transport-ws
When running the same against the WebSocket API GW URL
wscat -c wss://*****.execute-api.*****.amazonaws.com/**** -s graphql-transport-ws
I’m getting this:
error: Server sent no subprotocol
The error indicates a problem with the sub-protocol so when removing the sub-protocol a connection is established and I am getting a prompt:
Connected (press CTRL+C to quit)
>
However, there’s no indication of reaching the server and it seems like the connection is only made with the WebSocket API GW itself.
When I circumvent the gateway and directly connect an internet-facing NLB I'm able to establish a WebSocket connection.
I am not a super Websocket expert, but I understand WebSocket connections will be terminated by the API Gateway and cannot be used as a connection pass-through. You can forward web socket events using AWS_PROXY integration to a graphQL server backend, BUT it's not a maintained direct connection - API Gateway terminates and events towards the backend integration and will not return the integration response to the WebSocket since it is event-driven and not a connection-oriented service - hence the “error: Server sent no subprotocol” you are seeing.
So to use API GW as the WebSocket layer, you would need to build out connection management functionality somewhere to manage the event-based nature of the APIGW and send out data to the APIGW connections or adjust the integration mechanism within the graphql server to utilise the #connection functionality to send responses/notifications to WebSocket consumers.
Integrating Backend Service documentation:
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api-routes-integrations.html
Sending responses to a connected client:
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-how-to-call-websocket-api-connections.html
API GW Websockets are great for building custom solutions but take some effort since you will be configuring the setup for the events.
For a GraphQL API on AWS - I would recommend taking a look at AppSync, which is an AWS Managed GraphQL service - it handles GraphQL subscriptions via WebSockets natively and with zero additional code and its highly scalable out of the box and would simplify the GraphQL hosting burden of an ECS based solution.
I suspect there may be a lot of other reasons for the need to build out using existing GraphQL on ECS, so understand it's not always possible to pivot to something like AppSync. I feel the NLB solution you tried is okay within the existing ECS backend landscape and, as you have noted, is connection-oriented (via NLB), so will achieve the outcome you are after.
I am aware that Azure application gateway supports websockets. However, I cant figure out from the samples and documentation how websocket access is reflected in the Access Logs.
I have been going over Azure gateway documentation for Access logs over here
https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#for-application-gateway-and-waf-v2-sku
There is no protocol field - which usually carry ws or wss to indicate websocket access.
Thanks for help in advance
There is no user-configurable setting to selectively enable or disable WebSocket support in Application gateway. WebSocket protocols are designed to work over traditional HTTP ports of 80 and 443. You can continue using a standard HTTP listener on port 80 or 443 to receive WebSocket traffic. WebSocket traffic is then directed to the WebSocket enabled backend server using the appropriate backend pool as specified in application gateway rules.
Here is a clear documentation explaining about Websocket workflow in Application Gateway.
We are hosting an application in the preprod azure PCF environment which exposes websocket endpoints for client devices to connect to. Is there a prescribed methodology to secure the said websocket endpoint using TLS/SSL when hosted on PCF and running behind the PCF HAProxy?
I am having trouble interpreting this information, as in, are we supposed to expose port 4443 on the server and PCF shall by default pick it up to be a secure port that ensures unsecured connections cannot be established? Or does it require some configuration to be done on HAProxy?
Is there a prescribed methodology to secure the said websocket endpoint using TLS/SSL when hosted on PCF and running behind the PCF HAProxy?
A few things:
You don't need to configure certs or anything like that when deploying your app to PCF. The platform takes care of all that. In your case, it'll likely be handled by HAProxy, but it could be some other load balancer or even Gorouter depending on your platform operations team installed PCF. The net result is that TLS is first terminated before it hits your app, so you don't need to worry about it.
Your app should always force users to HTTPS. How you do this depends on the language/framework you're using, but most have some functionality for this.
This process generally works by checking to see if the incoming request was over HTTP or HTTPS. If it's HTTP, then you issue a redirect to the same URL, but over HTTPS. This is important for all apps, not just ones using WebSockets. Encrypt all the things.
Do keep in mind that you are behind one or more reverse proxies, so if you are doing this manually, you'll need to consider what's in x-forwarded-proto or x-forwarded-port, not just the upstream connection which would be Gorouter, not your client's browser.
https://docs.pivotal.io/platform/application-service/2-7/concepts/http-routing.html#http-headers
If you are forcing your user's to HTTPS (#1 above), then your users will be unable to initiate an insecure WebSocket connection to your app. Browsers like Chrome & Firefox have restrictions to prevent an insecure WebSocket connection from being made when the site was loaded over HTTPS.
You'll get a message like The operation is insecure in Firefox or Cannot connect: SecurityError: Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS. in Chrome.
I am having trouble interpreting this information, as in, are we supposed to expose port 4443 on the server and PCF shall by default pick it up to be a secure port that ensures unsecured connections cannot be established? Or does it require some configuration to be done on HAProxy?
From the application perspective, you don't do anything different. Your app is supposed to start and listen on the assigned port, i.e. what's in $PORT. This is the same for HTTP, HTTP, WS & WSS traffic. In short, as an app developer you don't need to think about this when deploying to PCF.
The only exception would be if your platform operations team uses a load balancer that does not natively support WebSockets. In this case, to work around the issue they need to separate traffic. HTTP and HTTPS go on the traditional ports 80 and 443, and they will route WebSockets on a different port. The PCF docs recommend 4443, which is where you're probably seeing that port. I can't tell you if your platform is set up this way, but if you know that you're using HAproxy, it is probably not.
https://docs.pivotal.io/platform/application-service/2-8/adminguide/supporting-websockets.html
At any rate, if you don't know just push an app and try to initiate a secure WebSocket connection over port 443 and see if it works. If it fails, try 4443 and see if that works. That or ask your platform operations team.
For what it's worth, even if your need to use port 4443 there is no difference to your application that runs on PCF. The only difference would be in your JavaScript code that initiates the WebSocket connection. It would need to know to use port 4443 instead of the default 443.
I establish a server using WCF based on Https protocol, now i want to create a client using WSO2 WSF/C++(and the client code is generated by Code Generator Tool), however, i can't communicate with server, because the client is based on Http, not Https. And i do not know how to code my client, also there is little sample or code for reference, almost of them are based on http. Now, how can i code my client to implement Https protocol.Does any one know how should i use the API? Thanks!
Since WSF/C++ uses Axis2/C, the HTTPS configurations are same for both WSF/C++ and Axis2/C. You can find HTTPS configuration here.
A Web Socket detects the presence of a proxy server and automatically sets up a tunnel to pass through the proxy. The tunnel is established by issuing an HTTP CONNECT statement to the proxy server, which requests for the proxy server to open a TCP/IP connection to a specific host and port. Once the tunnel is set up, communication can flow unimpeded through the proxy. Since HTTP/S works in a similar fashion, secure Web Sockets over SSL can leverage the same HTTP CONNECT technique. [1]
OK, sounds useful! But, in the client implementations I've seen thus far (Go [2], Java [3]) I do not see anything related to proxy detection.
Am I missing something or are these implementations just young? I know WebSockets is extremely new and client implementations may be equally young and immature. I just want to know if I'm missing something about proxy detection and handling.
[1] http://www.kaazing.org/confluence/display/KAAZING/What+is+an+HTML+5+WebSocket
[2] http://golang.org/src/pkg/websocket/client.go
[3] http://github.com/adamac/Java-WebSocket-client/raw/master/src/com/sixfire/websocket/WebSocket.java
Let me try to explain the different success rates you may have encountered. While the HTML5 Web Socket protocol itself is unaware of proxy servers and firewalls, it features an HTTP-compatible handshake so that HTTP servers can share their default HTTP and HTTPS ports (80 and 443) with a Web Sockets gateway or server.
The Web Socket protocol defines a ws:// and wss:// prefix to indicate a WebSocket and a WebSocket Secure connection, respectively. Both schemes use an HTTP upgrade mechanism to upgrade to the Web Socket protocol. Some proxy servers are harmless and work fine with Web Sockets; others will prevent Web Sockets from working correctly, causing the connection to fail. In some cases additional proxy server configuration may be required, and certain proxy servers may need to be upgraded to support Web Sockets.
If unencrypted WebSocket traffic flows through an explicit or a transparent proxy server on its way the WebSocket server, then, whether or not the proxy server behaves as it should, the connection is almost certainly bound to fail today (in the future, proxy servers may become Web Socket aware). Therefore, unencrypted WebSocket connections should be used only in the simplest topologies.
If encrypted WebSocket connection is used, then the use of Transport Layer Security (TLS) in the Web Sockets Secure connection ensures that an HTTP CONNECT command is issued when the browser is configured to use an explicit proxy server. This sets up a tunnel, which provides low-level end-to-end TCP communication through the HTTP proxy, between the Web Sockets Secure client and the WebSocket server. In the case of transparent proxy servers, the browser is unaware of the proxy server, so no HTTP CONNECT is sent. However, since the wire traffic is encrypted, intermediate transparent proxy servers may simply allow the encrypted traffic through, so there is a much better chance that the WebSocket connection will succeed if Web Sockets Secure is used. Using encryption, of course, is not free, but often provides the highest success rate.
One way to see it in action is to download and install the Kaazing WebSocket Gateway--a highly optimized, proxy-aware WebSocket gateway, which provides native WebSocket support as well as a full emulation of the standard for older browsers.
The answer is that these clients simply do not support proxies.
-Occam
The communication channel is already established by the time the WebSocket protocol enters the scene. The WebSocket is built on top of TCP and HTTP so you don't have to care about the things already done by these protocols, including proxies.
When a WebSocket connection is established it always starts with a HTTP/TCP connection which is later "upgraded" during the "handshake" phase of WebSocket. At this time the tunnel is established so the proxies are transparent, there's no need to care about them.
Regarding websocket clients and transparent proxies,
I think websocket client connections will fail most of the time for the following reasons (not tested):
If the connection is in clear, since the client does not know it is communicating with a http proxy server, it won't send the "CONNECT TO" instruction that turns the http proxy into a tcp proxy (needed for the client after the websocket handshake). It could work if the proxy supports natively websocket and handles the URL with the ws scheme differently than http.
If the connection is in SSL, the transparent proxy cannot know to which server it should connect to since it has decrypt the host name in the https request. It could by either generating a self-signed certificate on the fly (like for SSLStrip) or providing its own static certificate and decrypt the communication but if the client validates the server certificate it will fail (see https://serverfault.com/questions/369829/setting-up-a-transparent-ssl-proxy).
You mentioned Java proxies, and to respond to that I wanted to mention that Java-Websocket now supports proxies.
You can see the information about that here: http://github.com/TooTallNate/Java-WebSocket/issues/88
websocket-client, a Python package, supports proxies, at the very least over secure scheme wss:// as in that case proxy need no be aware of the traffic it forwards.
https://github.com/liris/websocket-client/commit/9f4cdb9ec982bfedb9270e883adab2e028bbd8e9