Decode a websocket frame - websocket

Question

I am trying to decode a websocket frame, but I'm not successful when it comes to decoding the extended payload. Here what I did achieve so far:
char *in = data;
char *buffer;
unsigned int i;
unsigned char mask[4];
unsigned int packet_length = 0;
int rc;
/* Expect a finished text frame. */
assert(in[0] == '\x81');
packet_length = ((unsigned char) in[1]) & 0x7f;
mask[0] = in[2];
mask[1] = in[3];
mask[2] = in[4];
mask[3] = in[5];
if (packet_length <= 125) { **// This decoding works**
/* Unmask the payload. */
for (i = 0; i < packet_length; i++)
in[6 + i] ^= mask[i % 4];
rc = asprintf(&buffer, "%.*s", packet_length, in + 6);
} else
if (packet_length == 126) { **//This decosing does NOT work**
/* Unmask the payload. */
for (i = 0; i < packet_length; i++)
in[8 + i] ^= mask[i % 4];
rc = asprintf(&buffer, "%.*s", packet_length, in + 8);
}
What am I doing wrong? How do I encode the extended payload?

Answer 1

The sticking point is at > 125 bytes payload.
The format is pretty simple, lets say you send ten a's in JavaScript:
ws.send("a".repeat(10))
Then the server will receive:
bytes[16]=818a8258a610e339c771e339c771e339
byte 0: The 0x81 is just an indicator that a message received
byte 1: the 0x8a is the length, substract 0x80 from it, 0x0A == 10
byte 2, 3, 4, 5: the 4 byte xor key to decrypt the payload
the rest: payload
But now lets say you send 126 a's in JavaScript:
ws.send("a".repeat(126))
Then the server will receive:
bytes[134]=81fe007ee415f1e5857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574
If the length of the payload is > 125, the byte 1 will have the value 0xfe, the format changes then to:
byte 0: The 0x81 is just an indicator that a message received
byte 1: will be 0xfe
byte 2, 3: the length of the payload as a uint16 number
byte 4, 5, 6, 7: the 4 byte xor key to decrypt the payload
the rest: payload
Example code in C#:
List<byte[]> decodeWebsocketFrame(Byte[] bytes)
{
List<Byte[]> ret = new List<Byte[]>();
int offset = 0;
while (offset + 6 < bytes.Length)
{
// format: 0==ascii/binary 1=length-0x80, byte 2,3,4,5=key, 6+len=message, repeat with offset for next...
int len = bytes[offset + 1] - 0x80;
if (len <= 125)
{
//String data = Encoding.UTF8.GetString(bytes);
//Debug.Log("len=" + len + "bytes[" + bytes.Length + "]=" + ByteArrayToString(bytes) + " data[" + data.Length + "]=" + data);
Debug.Log("len=" + len + " offset=" + offset);
Byte[] key = new Byte[] { bytes[offset + 2], bytes[offset + 3], bytes[offset + 4], bytes[offset + 5] };
Byte[] decoded = new Byte[len];
for (int i = 0; i < len; i++)
{
int realPos = offset + 6 + i;
decoded[i] = (Byte)(bytes[realPos] ^ key[i % 4]);
}
offset += 6 + len;
ret.Add(decoded);
} else
{
int a = bytes[offset + 2];
int b = bytes[offset + 3];
len = (a << 8) + b;
//Debug.Log("Length of ws: " + len);
Byte[] key = new Byte[] { bytes[offset + 4], bytes[offset + 5], bytes[offset + 6], bytes[offset + 7] };
Byte[] decoded = new Byte[len];
for (int i = 0; i < len; i++)
{
int realPos = offset + 8 + i;
decoded[i] = (Byte)(bytes[realPos] ^ key[i % 4]);
}
offset += 8 + len;
ret.Add(decoded);
}
}
return ret;
}

Answer 2

If packet_length is 126, the following 2 bytes give the length of data to be read.
If packet_length is 127, the following 8 bytes give the length of data to be read.
The mask is contained in the following 4 bytes (after the length).
The message to be decoded follows this.
The data framing section of the spec has a useful illustration of this.
If you re-order your code to something like
Read packet_length
Check for packet_length of 126 or 127. Reassign packet_length to value of following 2/4 bytes if required.
Read mask (the 4 bytes after packet_length, including any additional 2 or 8 bytes read for the step above).
Decode message (everything after the mask).
then things should work.