Is there a summary anywhere of the protocol difference between the various WebSockets drafts?
The browser support levels are still all over the place, so it is not sufficient just to consider the RFC.
Obviously the Sec-WebSocket-Version changes, and I know the early format was pretty radically different. However, I mean the more subtle changes in the protocol. For example, hybi-10 (v8), in framing, suggests the extended payload length is stored as 16/63, rather than 16/64 in RFC 6455 (v13).
So: is there a summary of changes anywhere?
Alternatively (if we ignore the very early drafts, and the version numbers), is it the case that the protocol is essentially the same, and that the drafts are mainly corrections to the specification text?
The Wikipedia WebSocket lists which browsers support which protocol.
Also, the IETF provides an diff tool that can be used to compare any two RFC draft specifications. For example, to compare WebSocket draft 15 and 17 go here:
http://tools.ietf.org/rfcdiff?url1=draft-ietf-hybi-thewebsocketprotocol-15.txt&url2=draft-ietf-hybi-thewebsocketprotocol-17.txt
Adjust the url1 and url2 addresses to get a diff for arbitrary versions. Note that this will shows you textual differences to the spec and large changes to the spec often happen without corresponding differences on the wire. I suggest searching the diffs for the "Protocol Overview" section and the "Base Framing Protocol" section which show the header summary and the framing diagram respectively.
The biggest difference in the wire protocol occured between Hixie-76/HyBi-00 (HyBi-00 was just a copy of Hixie-76 for starting the new series) and the rest of the HyBi series starting with HyBi-04 (HyBi-17 became IETF RFC 6455). Some of the major changes from the Hixie series to the HyBi series:
In the Hixie-76 protocol, there was a peculiar hash handshake that happened after the handshake headers but before the actual data frames.
In Hixie-76, the frames were prefixed with 0x00 and suffixed with 0xff. There was no way to determine the length of the frame except by receiving/buffering all the way to the end of the frame. In the HyBi series (after HyBi-00) the frame length is part of the prefix/header and there is no suffix.
The HyBi series supports both UTF-8 text and binary data in the payload (Hixie only supported UTF-8). This is indicated by and opcode in the frame header.
To add a specific change; in Sec-WebSocketVersion <= 8, the origin is in Sec-WebSocket-Origin; however, in 13 this changes to the Origin header. This changes specifically between hybi-10 and hybi-11, which are both version "8" implementations. Also note that it is Origin in hixie-76/hybi-00, so it looks like it went from Origin to Sec-WebSocket-Origin and then back to Origin.
I'm not aware of many of the protocol versions being in current use. I have a websocket server which supports Hixie-76 and hybi-10 through 17 (just changes in Sec-WebSocket-Version) which works against Safari (desktop + iOS), Firefox and Chrome.
(The older) Hixie-76 is useful for talking to iOS devices at least.
hybi-10 onwards are essentially the same. I'd assumed that your example of the extended payload being advertised as 63 bits in hybi-10 was a typo and was one of the many small corrections made when the drafts moved rapidly from 10 to 17.
Later: edited to show that some Safari versions actually use Hixie-76
Related
My question is in the title, this provides context to help you understand my confusion. Everything is sent over https.
My understanding of base 64 encoding is that it is a way of representing binary data as text, such that the text is safe to transmit across networks or the internet because it avoids anything that might be interpreted as a control code by the various possible protocols that might be involved at some point.
Given this understanding, I am confused why everything sent to over the internet is not base 64 encoded. When is it safe not to base 64 encode something before sending it? I understand that not everything understands or expects to receive things in base 64, but my question is why doesn't everything expect and work with this if it is the only way to send data without the possibility it could be interpreted as control codes?
I am designing an Android app and server API such that the app can use the API to send data to the server. There are some potentially large SQLite database files the client will be sending to the server (I know this sounds strange, yes it needs to send the entire database files). They are being gzipped prior to uploading. I know there is also a header that can be used to indicate this: Content-Encoding: gzip. Would it be safe to compress the data and send it with this header without base 64 encoding it? If not, why does such a header exist if it is not safe to use? I mean, if you base 64 encode it first and then compress it, you undo the point of base 64 encoding and it is not at that point base 64 encoded. If you compress it first and then base 64 encode it, that header would no longer be valid as it is not in the compressed format at that point. We actually don't want to use the header because we want to save the files in a compressed state, and using the header will cause the server to decompress it prior to our API code running. I'm only asking this to further clarify why I am confused about whether it is safe to send gzip compressed data without base 64 encoding it.
My best guess is that it depends on if what you are sending is binary data or not. If you are sending binary data, it should be base 64 encoded as the final step before uploading it. But if you are sending text data, you may not need to do this. However it still seems to my logic, this might still depends on the character encoding used. Perhaps some character encodings can result in sending data that could be interpreted as a control code? If this is true, which character encodings are safe to send without base 64 encoding them as the final step prior to sending it? If I am correct about this, it implies you should only use the that gzip header if you are sending compressed text that has not been base 64 encoded. Does compressing it create the possibility of something that could be interpreted as a control code?
I realize this was rather long, so I will repeat my primary questions (the title) here: Is either Gzip compressed binary data or uncompressed text safe to transmit, or should it be base 64 encoded as the final step before sending it? Okay I lied there is one more question involved in this. Would sending gzip compressed text always be safe to send without base 64 encoding it at the end, no matter which character encoding it had prior to compression?
My understanding of base 64 encoding is that it is a way of representing binary data as text,
Specifically, as text consisting of characters drawn from a 64-character set, plus a couple of additional characters serving special purposes.
such that the text is safe to transmit across networks or the internet because it avoids anything that might be interpreted as a control code by the various possible protocols that might be involved at some point.
That's a bit of an overstatement. For two endpoints to communicate with each other, they need to agree on one protocol. If another protocol becomes involved along the way, then it is the responsibility of the endpoints for that transmission to handle any needed encoding considerations for it.
What bytes and byte combinations can successfully be conveyed is a matter of the protocol in use, and there are plenty that handle binary data just fine.
At one time there was also an issue that some networks were not 8-bit clean, so that bytes with numeric values greater than 127 could not be conveyed across those networks, but that is not a practical concern today.
Given this understanding, I am confused why everything sent to over the internet is not base 64 encoded.
Given that the understanding you expressed is seriously flawed, it is not surprising that you are confused.
When is it safe not to base 64 encode something before sending it?
It is not only safe but essential to avoid base 64 encoding when the recipient of the transmission expects something different. The two or more parties to a given transmission must agree about the protocol to be used. That establishes the acceptable parameters of the communication. Although Base 64 is an available option for part or all of a message, it is by no means the only one, nor is it necessarily the best one for binary data, much less for data that are textual to begin with.
I understand that not everything understands or expects to receive things in base 64, but my question is why doesn't everything expect and work with this if it is the only way to send data without the possibility it could be interpreted as control codes?
Because it is not by any means the only way to avoid data being misinterpreted.
They are being gzipped prior to uploading. I know there is also a header that can be used to indicate this: Content-Encoding: gzip. Would it be safe to compress the data and send it with this header without base 64 encoding it?
It would be expected to transfer such data without base-64 encoding it. HTTP(S) handles binary data just fine. The Content-Encoding header tells the recipient how to interpret the message body, and if it specifies a binary content type (such as gzip) then binary data conforming to that content type are what the recipient will expect.
My best guess is that it depends on if what you are sending is binary data or not.
No. These days, for all practical intents and purposes, it depends only on what application-layer protocol you are using for the transmission. If it specifies that some or all of the message is to be base-64 encoded (according to a particular base-64 scheme, as there are more than one) then that's what the sender must do and how the receiver will interpret the message. If the protocol does not specify that, then the sender must not perform base-64 encoding. Some protocols afford the sender the option to make this choice, but those also provide a way for the sender to indicate inside the transmission what choice has been made.
Is either Gzip compressed binary data or uncompressed text safe to transmit, or should it be base 64 encoded as the final step before sending it?
Neither is inherently unsafe to transmit on today's networks. Whether data are base-64 encoded for transmission is a question of agreement between sender and receiver.
Okay I lied there is one more question involved in this. Would sending gzip compressed text always be safe to send without base 64 encoding it at the end, no matter which character encoding it had prior to compression?
The character encoding of the uncompressed text is not a factor in whether a gzipped version can be safely and successfully conveyed. But it probably matters for the receiver or anyone to whom they forward that data to understand the uncompressed text correctly. If you intend to accommodate multiple character encodings then you will want to provide a way to indicate which applies to each text.
I am implementing a WebSocket server in C and was wondering what's the purpose of the text/binary frame indicators (opcode 1 and 2). Why they are there? In the end in both cases the payload contains bits. And when there is a protocol using websocket or so then I know what expect in the data. Is it because when it's a text message I can be sure that payload only contains UTF-8 valid data?
I will start my answer by pointing out that WebSockets are often implemented with a Javascript client in mind (i.e., a browser).
When you're using C, the different opcode might be used in different ways, but when using Javascript, this difference controls the type of the data in the event (Blob vs. String).
As you point out in the question, a string is always a valid UTF-8 stream of bytes, whereas a blob isn't.
This affects some data transport schemes (such as JSON parsing, which requires a UTF-8 valid stream).
Obviously, in C, this opcode could be used in different ways, but it would be better to use the opcode in the same manner as a potential javascript client.
P.S.
There are a number of Websocket C libraries and frameworks out there (I'm the author of facil.io).
Unless this is a study project, I would consider using one of the established frameworks / libraries.
I have been starting to see uggc/uggcf (rot-13 encoded http/https) links show up in our system.
Are these worth supporting, is there actually a demand for it? The IETF document (link) has not been touched since 2001 and I cannot find much information on them at all.
Is there an area of the world where this is more common? I've only noticed them since we went world-wide.
The document describes it as a method to 'secure' the url as well as the data. What is the value of rot-13 encoding the data if it can be reversed without a key? HTTPS handles all of this, except for the domain itself.
I know this is an old answer but this is an interesting topic so I'll try to answer your question:
The "Encrypted Hypertext Transfer Protocol -- UGGC/1.0" specification, is an April Fools RFC. The IETF releases these almost yearly on 1st April, and ROT13 "encryption" would be pointless, since by knowning the encryption algortihm, you would be able to "decrypt" the message, in this case the URL.
So no, it's not worth supporting, and it does not provide any "serious" protection. The only usage I've seen is in some CTFs or hacking/crypto challenges.
Is there exist rfc ftp specifcation fresher than RFC 959?
There were several extensions to FTP presented in later RFCs (rfc2228, rfc2389, rfc3659, rfc4217 and maybe some more) and they are enough for all current FTP-related needs. The only things not adopted as a standard but relatively wide-spread are implicit FTP on port 990 and Mode Z / compression.
You can see the current status of RFC 959 in the status section at the top. If some later edition of the standard had replaced RFC 959, then there would be an entry "Obsoleted by" like there is an "Obsoletes" line for RFC 765, the previous version of the standard. However, other documents have since "Updated" the document (you could think of it as patches) and the documents are listed in the status section as "Updated by". There have also been a number of independent extensions that did not need to "update" the standard, like RFC 2428. There is no reliably maintained index of such extension specifications besides the general RFC Index maintained by the RFC Editor.
I am building an AJAX application that uses both HTTP Content and HTTP Header to send and receive data. Is there a point where the data received from the HTTP Header won't be read by the browser because it is too big ? If yes, what is the limit and is it the same behaviour in all the browser ?
I know that theoretically there is no limit to the size of HTTP headers, but in practice what is the point that past that, I could have problem under certain platform, browsers or with certain software installed on the client computer or machine. I am more looking into a guide-line for safe practice of using HTTP headers. In other word, up to what extend can HTTP headers be used for transmitting additional data without having potential problem coming into the line ?
Thanks, for all the input about this question, it was very appreciated and interesting. Thomas answer got the bounty, but Jon Hanna's answer brought up a very good point about the proxy.
Short answers:
Same behaviour: No
Lowest limit found in popular browsers:
10KB per header
256 KB for all headers in one response.
Test results from MacBook running Mac OS X 10.6.4:
Biggest response successfully loaded, all data in one header:
Opera 10: 150MB
Safari 5: 20MB
IE 6 via Wine: 10MB
Chrome 5: 250KB
Firefox 3.6: 10KB
Note
Those outrageous big headers in Opera, Safari and IE took minutes to load.
Note to Chrome:
Actual limit seems to be 256KB for the whole HTTP header.
Error message appears: "Error 325 (net::ERR_RESPONSE_HEADERS_TOO_BIG): Unknown error."
Note to Firefox:
When sending data through multiple headers 100MB worked fine, just split up over 10'000 headers.
My Conclusion:
If you want to support all popular browsers 10KB per header seems to be the limit and 256KB for all headers together.
My PHP Code used to generate those responses:
<?php
ini_set('memory_limit', '1024M');
set_time_limit(90);
$header = "";
$bytes = 256000;
for($i=0;$i<$bytes;$i++) {
$header .= "1";
}
header("MyData: ".$header);
/* Firfox multiple headers
for($i=1;$i<1000;$i++) {
header("MyData".$i.": ".$header);
}*/
echo "Length of header: ".($bytes / 1024).' kilobytes';
?>
In practice, while there are rules prohibitting proxies from not passing certain headers (indeed, quite clear rules on which can be modified and even on how to inform a proxy on whether it can modify a new header added by a later standard), this only applies to "transparent" proxies, and not all proxies are transparent. In particular, some wipe headers they don't understand as a deliberate security practice.
Also, in practice some do misbehave (though things are much better than they were).
So, beyond the obvious core headers, the amount of header information you can depend on being passed from server to client is zero.
This is just one of the reasons why you should never depend on headers being used well (e.g., be prepared for the client to repeat a request for something it should have cached, or for the server to send the whole entity when you request a range), barring the obvious case of authentication headers (under the fail-to-secure principle).
Two things.
First of all, why not just run a test that gives the browser progressively larger and larger headers and wait till it hits a number that doesn't work? Just run it once in each browser. That's the most surefire way to figure this out. Even if it's not entirely comprehensive, you at least have some practical numbers to go off of, and those numbers will likely cover a huge majority of your users.
Second, I agree with everyone saying that this is a bad idea. It should not be hard to find a different solution if you are really that concerned about hitting the limit. Even if you do test on every browser, there are still firewalls, etc to worry about, and there is absolutely no way you will be able to test every combination (and I'm almost positive that no one else has done this before you). You will not be able to get a hard limit for every case.
Though in theory, this should all work out fine, there might later be that one edge case that bites you in the butt if you decide to do this.
TL;DR: This is a bad idea. Save yourself the trouble and find a real solution instead of a workaround.
Edit: Since you mention that the requests can come from several types of sources, why not just specify the source in the request header and have the data contained entirely in the body? Have some kind of Source or ClientType field in the header that specifies where the request is coming from. If it's coming from a browser, include the HTML in the body; if it's coming from a PHP application, put some PHP-specific stuff in there; etc etc. If the field is empty, don't add any extra data at all.
The RFC for HTTP/1.1 clearly does not limit the length of the headers or the body.
According to this page modern browsers (Firefox, Safari, Opera), with the exception of IE can handle very long URIs: https://web.archive.org/web/20191019132547/https://boutell.com/newfaq/misc/urllength.html. I know it is different from receiving headers, but at least shows that they can create and send huge HTTP requests (possibly unlimited length).
If there's any limit in the browsers it would be something like the size of the available memory or limit of a variable type, etc.
Theoretically, there's no limit to the amount of data that can be sent in the browser. It's almost like saying there's a limit to the amount of content that can be in the body of a web page.
If possible, try to transmit the data through the body of the document. To be on the safe side, consider splitting the data up, so that there are multiple passes for loading.