I have configured spring with web sockets, including rabbit mq on the back end and I can confirm that I can send push messages to the browser.
And using SockJS on the front end.
Up until now I have been using the classic load balancer.
I am trying to get web sockets to work on AWS. I have upgraded to the Application Load Balancer but I still get Bad Request response when I try to make the web socket connection to:
ws://XXXX.eu-west-1.elasticbeanstalk.com/spring/hello/870/sbmdv5tn/websocket
That call still gives 400 Bad Request response...
And I see
Handshake failed due to invalid Upgrade header: null
Errors on the back end...
It has to do the fact that the a connection upgrade is requested and these upgrade requests occur "per hop".
In my scenario I am running with apache in front of tomcat and in order for tomcat to receive these upgrade headers I need to enable web socket tunnelling on the apache proxy such that apache will simply pass through the upgrade request.
UPDATE:
Although a better solution is to bypass apache altogether and go straight to tomcat - that is configure the load balancer to route to port 8080 and not port 80. I suspect the reason elastic beanstalk does not do this by default because it then requires a load balancer - and if you only want single instance you don't need a load balancer.
Related
We are hosting an application in the preprod azure PCF environment which exposes websocket endpoints for client devices to connect to. Is there a prescribed methodology to secure the said websocket endpoint using TLS/SSL when hosted on PCF and running behind the PCF HAProxy?
I am having trouble interpreting this information, as in, are we supposed to expose port 4443 on the server and PCF shall by default pick it up to be a secure port that ensures unsecured connections cannot be established? Or does it require some configuration to be done on HAProxy?
Is there a prescribed methodology to secure the said websocket endpoint using TLS/SSL when hosted on PCF and running behind the PCF HAProxy?
A few things:
You don't need to configure certs or anything like that when deploying your app to PCF. The platform takes care of all that. In your case, it'll likely be handled by HAProxy, but it could be some other load balancer or even Gorouter depending on your platform operations team installed PCF. The net result is that TLS is first terminated before it hits your app, so you don't need to worry about it.
Your app should always force users to HTTPS. How you do this depends on the language/framework you're using, but most have some functionality for this.
This process generally works by checking to see if the incoming request was over HTTP or HTTPS. If it's HTTP, then you issue a redirect to the same URL, but over HTTPS. This is important for all apps, not just ones using WebSockets. Encrypt all the things.
Do keep in mind that you are behind one or more reverse proxies, so if you are doing this manually, you'll need to consider what's in x-forwarded-proto or x-forwarded-port, not just the upstream connection which would be Gorouter, not your client's browser.
https://docs.pivotal.io/platform/application-service/2-7/concepts/http-routing.html#http-headers
If you are forcing your user's to HTTPS (#1 above), then your users will be unable to initiate an insecure WebSocket connection to your app. Browsers like Chrome & Firefox have restrictions to prevent an insecure WebSocket connection from being made when the site was loaded over HTTPS.
You'll get a message like The operation is insecure in Firefox or Cannot connect: SecurityError: Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS. in Chrome.
I am having trouble interpreting this information, as in, are we supposed to expose port 4443 on the server and PCF shall by default pick it up to be a secure port that ensures unsecured connections cannot be established? Or does it require some configuration to be done on HAProxy?
From the application perspective, you don't do anything different. Your app is supposed to start and listen on the assigned port, i.e. what's in $PORT. This is the same for HTTP, HTTP, WS & WSS traffic. In short, as an app developer you don't need to think about this when deploying to PCF.
The only exception would be if your platform operations team uses a load balancer that does not natively support WebSockets. In this case, to work around the issue they need to separate traffic. HTTP and HTTPS go on the traditional ports 80 and 443, and they will route WebSockets on a different port. The PCF docs recommend 4443, which is where you're probably seeing that port. I can't tell you if your platform is set up this way, but if you know that you're using HAproxy, it is probably not.
https://docs.pivotal.io/platform/application-service/2-8/adminguide/supporting-websockets.html
At any rate, if you don't know just push an app and try to initiate a secure WebSocket connection over port 443 and see if it works. If it fails, try 4443 and see if that works. That or ask your platform operations team.
For what it's worth, even if your need to use port 4443 there is no difference to your application that runs on PCF. The only difference would be in your JavaScript code that initiates the WebSocket connection. It would need to know to use port 4443 instead of the default 443.
Hasura GraphQL Engine is deployed on a Cloudfoundry instance backed by AWS, it is exposed at a subdomain via an AWS ELB. The console is exposed at https://hasura.cloud.domain.com/console and the GraphQL API accepts queries at https://hasura.cloud.domain.com/v1alpha1/graphql.
But when a subscription is executed from console, an error happens with the following log on JS Console:
vendor.js:1 WebSocket connection to 'wss://hasura.cloud.domain.com/v1alpha1/graphql' failed: Error during WebSocket handshake: Unexpected response code: 200
Analyzing the websocket frames on Chrome indicates an error with (Opcode -1).
Basically, the client is unable to open a websocket connection.
Some load balancers do not support passing WebSocket handshake requests containing the Upgrade header to the CF router. For instance, the Amazon Web Services (AWS) Elastic Load Balancer (ELB) does not support this behavior. In this scenario, you must configure your load balancer to forward TCP traffic to your CF router to support WebSockets.
ref: https://docs.cloudfoundry.org/adminguide/supporting-websockets.html#config
Basically, there is some configuration required with AWS ELB and CF Router to get websockets working. This is typically done by setting up a non-standard port to forward all TCP connections to the CF Router. We have learned from our clients that this port is typically 4443.
So, to get websocket connections to work, choose the endpoint as wss://hasura.cloud.domain.com:4443/v1alpha1/graphql for websocket connections and thus subscriptions.
The console can be opened at https://hasura.cloud.domain.com:4443 as well.
Utilizing socket.io, my WebSocket connections proxied through the GCLB are closed at the interval defined in the backend service configuration:
Obviously as pictured, the workaround is an impractically long timeout period, but for the sake of correctness, what is the proper way to configure the WebSocket connection through socket.io (or elsewhere if it is not a socket.io's problem) to avoid these closures?
Reason for closure according to GCLB log is "backend_connection_closed_after_partial_response_sent".
As stated in the Cloud Load Balancer documentation it would seem that we need to send a proper Upgrade response in order for the load balancer to keep the connection alive.
When HTTP(S) Load Balancing recognizes a WebSocket Upgrade request from an HTTP(S) client and the request is followed by a successful Upgrade response from the backend instance, the load balancer proxies bidirectional traffic for the duration of the current connection. If the backend does not return a successful Upgrade response, the load balancer closes the connection.
https://cloud.google.com/load-balancing/docs/https/#websocket_proxy_support (emphasis mine)
How can this be done?
I have gotten the picture that if the google load balancer has been configured to handle HTTPS (by adding SSL certificate) that I don't need to have a ssl certificate on my compute engine instances. From my understanding the load balancer gets the secure request and then just forwards with http to an instance.
Now the frontend for the load balancer is configured for two ports. 8080 for regular HTTP protocol and 443 for HTTPS protocol. If I only want to handle HTTPS is setting the spring-boot application to listen to port 443 the only thing I have to do to make it work? Simply adding the following this to application.properties.
server.port = 443
Or is there more configuration needed from the spring part? I'm genuinely interested in learning this and have researched and tried reading up on this but I can't seem to find any good resources doing something similar. I get the feeling that a lot of the knowledge around these kind of problems is gotten through practical experiences.
If you want the Google load balancer to terminate HTTPS and forward HTTP to your backend services, simply configure the load balancer with a HTTP backend. If you're using a HTTPS backend you'll have to listen to and handle HTTPS traffic in your app.
The difference is if the traffic between the load balancer and your backend (inside GCP) is encrypted or not. usually HTTPS termination at the load balancer level is enough.
Our Spring application is running on several different servers. For one of those servers POST requests do not seem to be working. All site functionality that uses GET requests works completely fine; however, as soon as I hit something that uses a POST request (ex. form submit) the site just hangs permanently. The server won't give any response. We can see the requests in Tomcat Manager but they don't time out.
Has anyone ever seen this?
We have found the problem. Our DBA accidentally deleted the MySQL database files on that particular server (/sigh). In our Spring application we use GET requests for record retrieval and the records we were trying to retrieve must have been cached by MySQL. This made it seem as if GET requests were working. When trying to add new data to the database, which we use POST requests to do, Tomcat would wait for a response, which never came, from MySQL.
In my experience if you're getting a timeout error it's almost always due to not having correct ports open for your application. For example, go into your virtual machine's rules and insure port 8080, 8443 or 80, 443 are open for http and https traffic.
In google cloud platform: its under VPC networking -> firewall rules. Azure and AWS are similar.