After reading this, https://httpwg.org/specs/rfc7541.html#integer.representation
I am confused about quite a few things, although I seem to have the overall gist of the idea.
For one, What are the 'prefixes' exactly/what is their purpose?
For two:
C.1.1. Example 1: Encoding 10 Using a 5-Bit Prefix
The value 10 is to be encoded with a 5-bit prefix.
10 is less than 31 (2^5 - 1) and is represented using the 5-bit prefix.
0 1 2 3 4 5 6 7
+---+---+---+---+---+---+---+---+
| X | X | X | 0 | 1 | 0 | 1 | 0 | 10 stored on 5 bits
+---+---+---+---+---+---+---+---+
What are the leading Xs? What is the starting 0 for?
>>> bin(10)
'0b1010'
>>>
Typing this in the python IDE, you see almost the same output... Why does it differ?
This is when the number fits within the number of prefix bits though, making it seemingly simple.
C.1.2. Example 2: Encoding 1337 Using a 5-Bit Prefix
The value I=1337 is to be encoded with a 5-bit prefix.
1337 is greater than 31 (25 - 1).
The 5-bit prefix is filled with its max value (31).
I = 1337 - (25 - 1) = 1306.
I (1306) is greater than or equal to 128, so the while loop body executes:
I % 128 == 26
26 + 128 == 154
154 is encoded in 8 bits as: 10011010
I is set to 10 (1306 / 128 == 10)
I is no longer greater than or equal to 128, so the while loop terminates.
I, now 10, is encoded in 8 bits as: 00001010.
The process ends.
0 1 2 3 4 5 6 7
+---+---+---+---+---+---+---+---+
| X | X | X | 1 | 1 | 1 | 1 | 1 | Prefix = 31, I = 1306
| 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 1306>=128, encode(154), I=1306/128
| 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 10<128, encode(10), done
+---+---+---+---+---+---+---+---+
The octet-like diagram shows three different numbers being produced... Since the numbers are produced throughout the loop, how do you replicate this octet-like diagram within an integer? What is the actual final result? The diagram or "I" being 10, or 00001010.
def f(a, b):
if a < 2**b - 1:
print(a)
else:
c = 2**b - 1
remain = a - c
print(c)
if remain >= 128:
while 1:
e = remain % 128
g = e + 128
remain = remain / 128
if remain >= 128:
continue
else:
print(remain)
c+=int(remain)
print(c)
break
As im trying to figure this out, I wrote a quick python implementation of it, It seems that i am left with a few useless variables, one being g which in the documentation is the 26 + 128 == 154.
Lastly, where does 128 come from? I can't find any relation between the numbers besides the fact 2 raised to the 7th power is 128, but why is that significant? Is this because the first bit is reserved as a continuation flag? and an octet contains 8 bits so 8 - 1 = 7?
For one, What are the 'prefixes' exactly/what is their purpose?
Integers are used in a few places in HPACK messages and often they have leading bits that cannot be used to for the actual integer. Therefore, there will often be a few leading digits that will be unavailable to use for the integer itself. They are represented by the X. For the purposes of this calculation it doesn't make what those Xs are: could be 000, or 111, or 010 or...etc. Also, there will not always be 3 Xs - that is just an example. There could only be one leading X, or two, or four...etc.
For example, to look up a previous HPACK decoded header, we use 6.1. Indexed Header Field Representation which starts with a leading 1, followed by the table index value. Therefore that 1 is the X in the previous example. We have 7-bits (instead of only 5-bits in the original example in your question). If the table index value is 127 or less we can represent it using those 7-bits. If it's >= 127 then we need to do some extra work (we'll come back to this).
If it's a new value we want to add to the table (to reuse in future requests), but we already have that header name in the table (so it's just a new value for that name we want as a new entry) then we use 6.2.1. Literal Header Field with Incremental Indexing. This has 2 bits at the beginning (01 - which are the Xs), and we only have 6-bits this time to represent the index of the name we want to reuse. So in this case there are two Xs.
So don't worry about there being 3 Xs - that's just an example. In the above examples there was one X (as first bit had to be 1), and two Xs (as first two bits had to be 01) respectively. The Integer Representation section is telling you how to handle any prefixed integer, whether prefixed by 1, 2, 3... etc unusable "X" bits.
What are the leading Xs? What is the starting 0 for?
The leading Xs are discussed above. The starting 0 is just because, in this example we have 5-bits to represent the integers and only need 4-bits. So we pad it with 0. If the value to encode was 20 it would be 10100. If the value was 40, we couldn't fit it in 5-bits so need to do something else.
Typing this in the python IDE, you see almost the same output... Why does it differ?
Python uses 0b to show it's a binary number. It doesn't bother showing any leading zeros. So 0b1010 is the same as 0b01010 and also the same as 0b00001010.
This is when the number fits within the number of prefix bits though, making it seemingly simple.
Exactly. If you need more than the number of bits you have, you don't have space for it. You can't just use more bits as HPACK will not know whether you are intending to use more bits (so should look at next byte) or if it's just a straight number (so only look at this one byte). It needs a signal to know that. That signal is using all 1s.
So to encode 40 in 5 bits, we need to use 11111 to say "it's not big enough", overflow to next byte. 11111 in binary is 31, so we know it's bigger than that, so we'll not waste that, and instead use it, and subtract it from the 40 to give 9 left to encode in the next byte. A new additional byte gives us 8 new bits to play with (well actually only 7 as we'll soon discover, as the first bit is used to signal a further overflow). This is enough so we can use 00001001 to encode our 9. So our complex number is represented in two bytes: XXX11111 and 00001001.
If we want to encode a value bigger than can fix in the first prefixed bit, AND the left over is bigger than 127 that would fit into the available 7 bits of the second byte, then we can't use this overflow mechanism using two bytes. Instead we use another "overflow, overflow" mechanism using three bytes:
For this "overflow, overflow" mechanism, we set the first byte bits to 1s as usual for an overflow (XXX11111) and then set the first bit of the second byte to 1. This leaves 7 bits available to encode the value, plus the next 8 bits in the third byte we're going to have to use (actually only 7 bits of the third byte, because again it uses the first bit to indicate another overflow).
There's various ways they could go have gone about this using the second and third bytes. What they decided to do was encode this as two numbers: the 128 mod, and the 128 multiplier.
1337 = 31 + (128 * 10) + 26
So that means the frist byte is set to 31 as per pervious example, the second byte is set to 26 (which is 11010) plus the leading 1 to show we're using the overflow overflow method (so 100011010), and the third byte is set to 10 (or 00001010).
So 1337 is encoded in three bytes: XXX11111 100011010 00001010 (including setting X to whatever those values were).
Using 128 mod and multiplier is quite efficient and means this large number (and in fact any number up to 16,383) can be represented in three bytes which is, not uncoincidentally, also the max integer that can be represented in 7 + 7 = 14 bits). But it does take a bit of getting your head around!
If it's bigger than 16,383 then we need to do another round of overflow in a similar manner.
All this seems horrendously complex but is actually relatively simply, and efficiently, coded up. Computers can do this pretty easily and quickly.
It seems that i am left with a few useless variables, one being g
You are not print this value in the if statement. Only the left over value in the else. You need to print both.
which in the documentation is the 26 + 128 == 154.
Lastly, where does 128 come from? I can't find any relation between the numbers besides the fact 2 raised to the 7th power is 128, but why is that significant? Is this because the first bit is reserved as a continuation flag? and an octet contains 8 bits so 8 - 1 = 7?
Exactly, it's because the first bit (value 128) needs to be set as per explanation above, to show we are continuing/overflowing into needing a third byte.
I'm just trying to solve this problem. Maybe you can help me.
_________________________________________________
|15|14|13|12|11|10|9 |8 |7 |6 |5 |4 |3 |2 |1 |0 |
|Tag |Index. |Word Off|Byte Offs
–––––––––––––––––––––––––––––––––––––––––––––––––
Tag= 4Bits
Index= 6Bits
Word Offset= 3Bits
Byte Offset= 3Bits
Totaly= 16Bits
For this I have the following tasks:
a) What is the data word width in bytes and bits? I think 16Bits(2 Byte)
b) What is the number of lines in the cache? I think this is 2^index bits, so the result should be 64 cache-line
c) How large is each cache line in bytes? This should be 2^offsetbits, so I get 1 Byte (8Bit) as result.
d) Which byte amount of pure data (without tags and flags) can the cache store?Here I have calculated 2^Tagbits, and as a result 8 Byte received.
e)What is the total size of the cache in bytes including one valid bit (per line) and the TAG bits?each cache-line is 1 byte (8bit) in size, there are 64 cache-lines in total, 1 valid bit is added to each cache-line and 4 tag bits. So I have 13 * 64 = 832 bits = 104 bytes
I'm not sure this is all right. Maybe you can help me.
Thank you already
salute
tobmes
I have this question in an Operating System test:
Given a disk of 1GB with 16KB blocks:
(1) Calculate the size of the File Allocation Table:
My Answer: since there are 2^16 blocks in the disk, we have a table with 2^16 entry, and every entry needs to store 16 bit (since there are 2^16 different blocks, we need 16 bit to identify each of them). So the size is 2^16 times 16 bit = 2^16 x 2^4 = 2^20 bit = 2^17 byte = 128Kb.
(2) Given the following table, indicate in which block are stored the following byte:
-byte 131080 of FileA starting at block 4.
-byte 62230 of FileB starting at block 3.
Entry Content
0 10
1 2
2 0
3 6
4 1
5 8
6 7
7 11
8 12
So FileA is (4) -> (1) -> (2) but the problem is: since every block is 16Kb = 2^4 x 2^10 byte = 2^14 byte = 16384 byte, block 4 contains from 1 to 16384, block 1 contains from 16385 to 32768, and block 2 from 32769 to 49152, where am I supposed to find the byte 131080???
Where is this wrong??
The golang blog states :
"A slice can also be formed by "slicing" an existing slice or array. Slicing is done by specifying a half-open range with two indices separated by a colon. For example, the expression b[1:4] creates a slice including elements 1 through 3 of b (the indices of the resulting slice will be 0 through 2)."
Can someone please explain to me the logic in the above. IE. Why doesn't b[1:4] reference elements 1 through 4? Is this consistent with other array referencing?
Indexes point to the "start" of the element. This is shared by all languages using zero-based indexing:
| 0 | first | 1 | second | 2 | third | 3 | fourth | 4 | fifth | 5 |
[0] = ^
[0:1] = ^ --------> ^
[1:4] = ^-------------------------------------> ^
[0:5] = ^ ----------------------------------------------------------> ^
It's also common to support negative indexing, although Go doesn't allow this:
|-6 | |-5 | |-4 | |-3 | |-2 | |-1 |
| 0 | first | 1 | second | 2 | third | 3 | fourth | 4 | fifth | 5 |
The reason is given in the Go Language Specification section on Slices.
For a string, array, or slice a, the
primary expression
a[low : high]
constructs a substring or slice. The
index expressions low and high select
which elements appear in the result.
The result has indexes starting at 0
and length equal to high - low.
For convenience, any of the index
expressions may be omitted. A missing
low index defaults to zero; a missing
high index defaults to the length of
the sliced operand.
It's easy and efficient to calculate the length of the slice as high - low.
Half-open intervals make sense for many reasons, when you get down to it. For instance, with a half-open interval like this, the number of elements is:
n = end - start
which is a pretty nice and easy formula. For a closed interval, it would be:
n = (end - start) + 1
which is (not a lot, but still) more complicated.
It also means that for e.g. a string, the entire string is [1, len(s)] which also seems intuitive. If the interval was closed, to get the entire string you would need [1, len(s) + 1].
Go uses half-open intervals for slices like many other languages. In a more mathematical notation, the slice b[1:4] is the interval [1,4) which excludes the upper endpoint.
This is from a homework assignment:
Assume that each page (disk block) has 16K bytes and each KVP has 8 bytes. Thus
we decide to use a B-tree of minsize (16000/8)/2 = 1000. Let T be such a B-tree and
suppose that height of T is 3. What is the minimum and maximum number of keys
that can be stored in T? Briefy justify your answer.
Note the following due to the properties of B-trees:
Each node has at most 2000 keys
Each node has at least 1000 keys (except for the root node)
I am having trouble understanding how the memory is limiting the number of keys.
It seems to me that since each page has 16000 bytes of space and each key takes up 8 bytes, then each page can store 2000 keys which is the max number of keys that can be stored at each level anyways.
The following are my calculations:
Minimum number of keys = 1000(1001)(2) + 1 = 2002001 keys at minimum
(Since the root is not constrained to having at least 1000 keys)
Maximum number of keys = 2000(2001)(2001) = 8008002000 keys at maximum
I feel I am missing something vital as the question cannot be this simple.
Somewhat blatant hint: Each non-leaf node has a right and a left child. Plus, there are pointers to key/value pairs, however they might be stored. (1000 seems like a lot...) Think about how you're going to store those 1000+ data points.
+--------------+
| Root |
| Left Right |
+---+------+---+
| |
| +---+----------+
| | Level 2 +---Data: List, hash table, whatever
| | Left Right |
| +---+------+---+
| | |
| Etc Etc
|
+---+----------+
| Level 2 +---Data: List, hash table, whatever
| Left Right |
+---+------+---+
| |
Etc Etc