I was reading the the official RFC for the Web Sockets protocol in order to implement it for learning purposes, I wanted to make things a bit different somehow but I was not sure what to make different. While I was reading the document, I came across this:
The Web Socket Protocol attempts to address the goals of existing bidirectional HTTP technologies in the context of the existing HTTP infrastructure; as such, it is designed to work over HTTP ports 80 and 443 as well as to support HTTP proxies and intermediaries, even if this implies some complexity specific to then current environment. However, the design does not limit Web Socket to HTTP, and future implementations could use a simpler handshake over a dedicated port without reinventing the entire protocol.
Does this imply a custom web socket protocol can be implemented with a non http based handshake?.
If so, does it mean a the regular Java Script Web Socket client will not work with this and I would need to implement a custom client to communicate using this protocol?
Related
What was a reason for registering special URI schemes: "ws" and "wss"?
I briefly read specs, and didn't find any reason about this change.
https://www.rfc-editor.org/rfc/rfc6455#section-11.1
WebSocket client sends special headers (Connection: Upgrade, Upgrade: websocket) which lets relatively easy decide whether this is normal HTTP requests (GET,POST ...etc) or this is WebSocket connection request.
Why WebSockets can't work using "http" and "https" schemes?
I wasn't on the board or anything, but here's any 2¢...:
The protocol's handshake currently uses HTTP(S) for connection establishment, but this is as far as the HTTP involvement goes.
It's highly possible that future versions might extend the handshake possibilities, using technologies such as ALPN.
By using the URI scheme ws://... and wss://..., the handshake is decoupled from the URI, allowing future changes to occur with less disturbance.
They use it for sake of architectural clarity, looks like there is no technical limitation why you could not handle your websocket connections using purely http.
Please notice javascript client libraries for handling websocket connections like Paho, are handling websockets purely over http protocol, and even fail if you are explicitly providing host prefixed with ws protocol instead - A piece of simple example from showcase code:
// Create a client instance
client = new Paho.MQTT.Client(
location.hostname, Number(location.port), "clientId"
);
I'd like to compare somehow capabilities of grpc vs. zeromq & its patterns: and I'd like to create some comparsion (feature set) - somehow - 0mq is "better" sockets - but anyways - if I apply 0mq patterns - I get comparable 'frameworks' I think - and here 0mq seems to be much more flexible ...
The main requirements are:
async req / res communication (inproc or remote) between nodes
flexible messages routing
loadbalancing support
well documented
any ideas?
thanks!
async req / res communication (inproc or remote) between nodes
Both libraries allow for synchronous or asynchronous communication depending on how to implement the communication. See this page for gRPC: http://www.grpc.io/docs/guides/concepts.html. Basically gRPC allow for typical HTTP synchronous request/response or a 'websocket-like' bidirectional streaming. For 0mq you can setup a simple REQ-REP connection which is basically a synchronous communication path, or you can create async ROUTER-DEALER type topologies.
flexible messages routing
'Routing' essentially means that a message gets from A to B via some broker. This is trivially done in 0mq and there are a number of topologies that support stuff like this (http://zguide.zeromq.org/page:all#Basic-Reliable-Queuing-Simple-Pirate-Pattern). In gRPC the same sort of topology could be created with a 'pub-sub' like connection over a stream. gRPC supports putting metadata in the message (https://github.com/grpc/grpc-go/blob/master/Documentation/grpc-metadata.md) which will allow you to 'route' a message into a queue that a 'pub-sub' connection could pull from.
loadbalancing support
gRPC has a health check support (https://github.com/grpc/grpc/blob/master/doc/health-checking.md) but because it's HTTP/2 you'd have to have a HTTP/2 load balancer to support the health check. This isn't a huge big deal, however, because you can tie the health check to a HTTP/1.1 service which the load balancer calls. 0mq is a tcp connection which means that a load balancer would likely check a 'socket connect' in tcpmode to verify the connection. This works but it's not as nice. Again you could get slick and front-end the 0mq service with a HTTP/1.1 webserver that the load balancer reads from.
well documented
both are well documented. 0mq's documentation must be read to throughly understand the technology and is more of a higher lift.
Here's the big differences:
0mq is a tcp protocol whereas gRPC is HTTP with a binary payload.
0mq requries you design a framing protocol (frame 1 = verison, frame 2 = payload, etc.), whereas much of this work is done for you in gRPC
gRPC is transparently coverted to REST (https://github.com/grpc-ecosystem/grpc-gateway) whereas 0mq requires a middleware application if you want to talk REST to it.
gRPC uses standard tls x509 certificates (think websites) whereas 0mq uses a custom encryption/authentication protocol (http://curvezmq.org/). Prior to 4.x there was no encryption support in 0mq and if you really wanted it you had to dive into this crap: https://wiki.openssl.org/index.php/BIO. (trust me don't do it)
0mq can create some pretty sick topologies (https://github.com/zeromq/majordomo) (https://rfc.zeromq.org/spec:7/MDP/) whereas gRPC is basically client/server
0mq requires much more time to build and get running whereas gRPC is basically compiling a protobuf messages and importing the service into your code.
Not quite the same. gRPC is primarily for heterogeneous service interoperability, ZeroMQ (ZMQ/0MQ/ØMQ) is a lower level messaging framework. ØMQ doesn't specify payload serialization beyond passing binary blobs whereas gRPC chooses Protocol Buffers by default. ØMQ is pretty much stuck on the same machine or machines between datacenters/clouds, whereas gRPC could potentially work on a real client too (ie mobile or web, it already supports iOS). gRPC using ØMQ could be significantly faster and more efficient for in-cloud/-datacenter services than the overhead, latency and complexity of http2 request/response chain. I'm not sure how (or even if) gRPC TLS security is adequate for public cloud and mobile/web usage, but one could always inject end-to-end security requirements (ie libsodium) at the router/controller level of the app/app framework and run in plaintext mode (which would also remove OpenSSL fork BoringSSL from causing maintenance headaches because of upstream flaws).
For very high latency / low bandwidth services (ie mission to mars), one would think about RPC using a transport like SMTP (ie Active Directory alternate replication) or MQTT (ie Facebook Messenger, ZigBee, SCADA)
Bonus (off-topic): It would be nice if gRPC had alternate pluggable transports like ØMQ (which also itself supports UNIX sockets, TCP, PGM and inproc), because HTTP/2 isn't stable in all languages yet and it's slower than ØMQ. And, it's worth looking at nanomsg (especially in the HFT world) because it could be extended with RDMA/SDP/MPI and made crazy low-latency/zero-copy/Infiniband-ready.
I'm trying to get a better understanding of how the server-side architecture works for WebSockets with the goal of implementing it in an embedded application. It seems that there are 3 different server-side software components in play here: 1) the web server to serve static HTTP pages and handle upgrade request, 2) a WebSockets library such as libwebsockets to handle the "nuts and bolts" of WebSockets communications, and 3) my custom application to actually figure out what to do with incoming data. How do all these fit together? Is it common to have a separate web server and WebSocket handling piece, aka a WebSocket server/daemon?
How does my application communicate with the web server and/or WebSockets library to send/receive data? For example, with CGI, the web server uses environmental variables to send info to the custom application, and stdout to receive responses. What is the equivalent communication system here? Or do you typically link in a WebSocket library into the customer application? But then how would communication with the web server to the WebSocket library + custom application work? Or all 3 combined into a single component?
Here's why I am asking. I'm using the boa web server on a uClinux/no MMU platform on a Blackfin processor with limited memory. There is no native WebSocket support in boa, only CGI. I'm trying to figure out how I can add WebSockets support to that. I would prefer to use a compiled solution as opposed to something interpreted such as JavaScript, Python or PHP. My current application using long polling over CGI, which does not provide adequate performance for planned enhancements.
First off, it's important to understand how a webSocket connection is established because that plays into an important relationship between webSocket connections and your web server.
Every webSocket connection starts with an HTTP request. The browser sends an HTTP request to the host/port that the webSocket connection is requested on. That request might look something like this:
GET /chat HTTP/1.1
Host: example.com:8000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
What distinguishes this request from any other HTTP request to that server is the Upgrade: websocket header in the request. This tells the HTTP server that this particular request is actually a request to initiate a webSocket connection. This header also allows the web server to tell the difference between a regular HTTP request and a request to open a webSocket connection. This allows something very important in the architecture and it was done this way entirely on purpose. This allows the exact same server and port to be used for both serving your web requests and for webSocket connections. All that is needed is a component on your web server that looks for this Upgrade header on all incoming HTTP connections and, if found, it takes over the connection and turns it into a webSocket connection.
Once the server recognizes this upgrade header, it responds with a legal HTTP response, but one that signals the client that the upgrade to the webSocket protocol has been accepted that looks like this:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
At that point, both client and server keep that socket from the original HTTP request open and both switch to the webSocket protocol.
Now, to your specific questions:
How does my application communicate with the web server and/or
WebSockets library to send/receive data?
Your application may use the built-in webSocket support in modern browsers and can initiate a webSocket connection like this:
var socket = new WebSocket("ws://www.example.com");
This will instruct the browser to initiate a webSocket connection to www.example.com use the same port that the current web page was connected with. Because of the built-in webSocket support in the browser, the above HTTP request and upgrade protocol is handled for you automatically from the client.
On the server-side of things, you need to make sure you are using a web server that has incoming webSocket support and that the support is enabled and configured. Because a webSocket connection is a continuous connection once established, it does not really follow the CGI model at all. There must be at least one long-running process handling live webSocket connections. In server models (like CGI), you would need some sort of webServer add-on that supports this long-running process for your webSocket connections. In a server environment like node.js which is already a long running process, the addition of webSockets is no change at all architecturally - but rather just an additional library to support the webSocket protocol.
I'd suggest you may find this article interesting as it discussions this transition from CGI-style single request handling to the continuous socket connections of webSocket:
Web Evolution: from CGI to Websockets (and how it will help you better monitor your cloud infrastructure)
If you really want to stick with the stdin/stdout model, there are libraries that model that for your for webSockets. Here's one such library. Their tagline is "It's like CGI, twenty years later, for WebSockets".
I'm trying to figure out how I can add WebSockets support to that. I
would prefer to use a compiled solution as opposed to something
interpreted such as JavaScript, Python or PHP.
Sorry, but I'm not familiar with that particular server environment. It will likely take some in-depth searching to find out what your options are. Since a webSocket connection is a continuous connection, then you will need a process that is running continuously that can be the server-side part of the webSocket connection. This can either be something built into your webServer or it can be an additional process that the webServer starts up and forwards incoming connections to.
FYI, I have a custom application at home here built on a Raspberry Pi that uses webSockets for real-time communication with browser web pages and it works just fine. I happen to be using node.js for the server environment and the socket.io library that runs on top of webSockets to give me a higher level interface on top of webSockets. My server code checks several hardware sensors on a regular interval and then whenever there is new/changed data to report, it sends messages down any open webSockets so the connected browsers get real-time updates on the sensor readings.
You would likely need some long-running application that incoming webSocket connections were passed from the web server to your long running process or you'd need to make the webSocket connections on a different port than your web server (so they could be fielded by a completely different server process) in which case you'd have a whole separate server to handle your webSocket requests and sockets (this server would also have to support CORS to enable browsers to connect to it since it would be a different port than your web pages).
Shall I use WebSocket on non-80 ports? Does it ruin the whole purpose of using existing web/HTTP infrastructures? And I think it no longer fits the name WebSocket on non-80 ports.
If I use WebSocket over other ports, why not just use TCP directly? Or is there any special benefits in the WebSocket protocol itself?
And since current WebSocket handshake is in the form of a HTTP UPGRADE request, does it mean I have to enable HTTP protocol on the port so that WebSocket handshake can be accomplished?
Shall I use WebSocket on non-80 ports? Does it ruin the whole purpose
of using existing web/HTTP infrastructures? And I think it no longer
fits the name WebSocket on non-80 ports.
You can run a webSocket server on any port that your host OS allows and that your client will be allowed to connect to.
However, there are a number of advantages to running it on port 80 (or 443).
Networking infrastructure is generally already deployed and open on port 80 for outbound connections from the places that clients live (like desktop computers, mobile devices, etc...) to the places that servers live (like data centers). So, new holes in the firewall or router configurations, etc... are usually not required in order to deploy a webSocket app on port 80. Configuration changes may be required to run on different ports. For example, many large corporate networks are very picky about what ports outbound connections can be made on and are configured only for certain standard and expected behaviors. Picking a non-standard port for a webSocket connection may not be allowed from some corporate networks. This is the BIG reason to use port 80 (maximum interoperability from private networks that have locked down configurations).
Many webSocket apps running from the browser wish to leverage existing security/login/auth infrastructure already being used on port 80 for the host web page. Using that exact same infrastructure to check authentication of a webSocket connection may be simpler if everything is on the same port.
Some server infrastructures for webSockets (such as socket.io in node.js) use a combined server infrastructure (single process, one listener) to support both HTTP requests and webSockets. This is simpler if both are on the same port.
If I use WebSocket over other ports, why not just use TCP directly? Or
is there any special benefits in the WebSocket protocol itself?
The webSocket protocol was originally defined to work from a browser to a server. There is no generic TCP access from a browser so if you want a persistent socket without custom browser add-ons, then a webSocket is what is offered. As compared to a plain TCP connection, the webSocket protocol offers the ability to leverage HTTP authentication and cookies, a standard way of doing app-level and end-to-end keep-alive ping/pong (TCP offers hop-level keep-alive, but not end-to-end), a built in framing protocol (you'd have to design your own packet formats in TCP) and a lot of libraries that support these higher level features. Basically, webSocket works at a higher level than TCP (using TCP under the covers) and offers more built-in features that most people find useful. For example, if using TCP, one of the first things you have to do is get or design a protocol (a means of expressing your data). This is already built-in with webSocket.
And since current WebSocket handshake is in the form of a HTTP UPGRADE
request, does it mean I have to enable HTTP protocol on the port so
that WebSocket handshake can be accomplished?
You MUST have an HTTP server running on the port that you wish to use webSocket on because all webSocket requests start with an HTTP request. It wouldn't have to be heavily featured HTTP server, but it does have to handle the initial HTTP request.
Yes - Use 443 (ie, the HTTPS port) instead.
There's little reason these days to use port 80 (HTTP) for anything other than a redirection to port 443 (HTTPS), as certification (via services like LetsEncrypt) are easy and free to set up.
The only possible exceptions to this rule are local development, and non-internet facing services.
Should I use a non-standard port?
I suspect this is the intent of your question. To this, I'd argue that doing so adds an unnecessary layer of complication with no obvious benefits. It doesn't add security, and it doesn't make anything easier.
But it does mean that specific firewall exceptions need to be made to host and connect to your websocket server. This means that people accessing your services from a corporate/school/locked down environment are probably not going to be able to use it, unless they can somehow convince management that it is mandatory. I doubt there are many good reasons to exclude your userbase in this way.
But there's nothing stopping you from doing it either...
In my opinion, yes you can. 80 is the default port, but you can change it to any as you like.
I've read about WebSockets but they don't seem to be pure "sockets", because there is an application layer protocol over them. "ws:"
Is there any way of doing a pure socket connection from a web browser, to enliven webpages?
Here are my random stabs in the dark
Applets sockets provided by Java (need java installed)
Flash sockets provided by Flash (need flash installed)
But about HTML5, Why are they called WebSockets if they aren't Sockets?
Is the websocket protocol so simple to implement that it is "almost"-sockets?
I've read about WebSockets but they don't seem to be pure "sockets", because there is an application layer protocol over them.
[Is the] websocket protocol so simple to implement that [it is] "almost"-sockets?
Allowing regular socket connections directly from the browser is never going to happen because it opens up a huge risk. WebSockets is about as close to raw sockets from the browser as you are going to get. The initial WebSockets handshake is similar to an HTTP handshake (allowing web servers to proxy/bridge it) and adds CORS type security. In addition, WebSockets is a message based transport (rather than streaming as raw TCP) and this is done using a two byte header on each message frame.
Even flash is not able to quite make raw TCP connections. Flash sockets also add CORS security, but instead of an in-band handshake, flash socket connections make a connection to port 843 on the target server to request a security policy file.
Is there any way of doing a pure socket connection from a web browser, to enliven webpages?
Yes, you can use my websockify bridge/proxy which allows a WebSockets enabled browser to connect directly to a TCP socket via websockify.
But about HTML5, Why are they called WebSockets if they aren't Sockets?
WebSockets are a transport built on TCP sockets. After the handshake there is very minimal overhead (typically just a two byte header).
I can't improve on Kanaka's answers to your secondary questions, and I know this question is a year old. But for the main question, Is there any way of doing a pure socket connection from a web browser, to enliven webpages? There is a project called the Java / JavaScript Socket Bridge that might be what you (or anyone coming across this page from a Google search) are looking for. The advantage of this method over what others have mentioned is that it does not require either a client-side or a server-side service to be run. So, for instance, if you wanted to implement an IRC client purely in JavaScript but your web host does not allow you sufficient rights to proxy the connection, this Java applet would be the way to go. The only concern is making sure the client has Java installed and allowed.
You can just send data between a client and a server with WebSockets. Simply speaking, the only difference that WebSockets introduces is that the client:
adds some header bytes, like the type of data and the length
adds masks and encodes the data using them
The server also has to add header bytes, but does not need to encode the data.
If you implement the protocol correctly (server side, that is, since the browser already has an implementation), you can use it with ease to send text and binary data. (Although browser support is narrow, especially for the latter.)
The benefit of WebSocket is that it is HTTP based. You can use it also in environments there http proxies are used. Thus Websocket has a higher infrastructure compatibility as plain tcp.
Additionally http/WebSocket is providing you some features which you otherwise have to specify on your own:
Redirect
NAT keepalive
Multiplexing via URI
Framing
If you are asking for some data to be pushed from server it is widely termed as COMET or Reverse Ajax.
Web sockets is still not very popular as there are inherent firewall issues and minimal support yet from popular browsers.
You can take a look at http://www.ape-project.org/ as this is one of the most popular implementations (but native to unix/linux only for now. For windows they suggest using a virtual box or vmware based implementation)