health warning: total beginner
why is this the standard method to reset a pin:
REGISTER_NAME &= ~(1 << PINXX);
rather than the straightforward:
REGISTER_NAME &= (0 << PINXX);
thanks in advance for time, answers & bibliographical hints
It need to be understood, what each sign means in
REGISTER_NAME &= ~(1 << PINXX);
1) << - this is logical shift left. This means that number on left side is shifted left by number of binary position, which is in right side.
For example. 1 << 5 means 0b00000001 shifted left 5 times (while zeroes appears at right). i.e. result will be 0b00100000 == 32.
In other words, 1 << X means "give a number, in which only bit #X is set, and other bits are zeros"
Ok. At this point you can see that 0 << anything gives always zero,
2) ~ - this is bitwise negation. It is inverts value of every bit in every position in binary representation of the number.
I.e. ~0b00100000 (or ~(1 << 5)) == 0b11011111.
Concluding, ~(1 << X) means "give a number, in which all bits are set, except of bit #X, which is to became zero"
3) &= - this is assignment combined with bitwise AND. I.e. X &= Y is the same that X = X & Y;
4) & - this is bitwise AND. result of this operation is number, in which only that bits are set, which are set at same position in both operands, and zero otherwise.
That means, if particular bit in one of operands is set, then same bit in result will be equal to bit at same position of another operand. If this bit is zero in either operand, then same bit in result will be zero.
You can rephrase X &= Y in this way: "if a bit in Y is one, leave the bit at the same position in X unchanged (whatever it is - one or zero). If the bit in Y is zero, then clear the same bit in X."
As was said above, ~(1 << N) gives a number where every bit is set, except of N. So X &= ~(1 << N) means "Clear bit #N, and remain all other bits unchanged".
Now you can see, that
REGISTER_NAME &= (0 << PINXX);
is nonsense, because it means "Clear all the bits". It is the same as REGISTER_NAME = 0;
By the way, if you are using assignment in this form:
REGISTER_NAME |= (0 << PINXX) | (1 << PINYY);
this doesn't mean "set bit PINYY, and clear bit PINXX", it means "set bit PINYY and do nothing with bit PINXX (as with all other bits)"
first instruction will reset pin XX, second instruction will reset all the pins associated to that register
Why would shifting a zero be more "straightforward"? That makes no sense at all, it's just 0 which when bitwise-anded would clear the entire register. Not at all what you want.
Remember that this:
REGISTER &= ~(1 << PIN);
means
REGISTER = REGISTER & ~(1 << PIN);
I.e. it will bitwise-and the register with a constant that has 1s in all positions except bit number PIN. In particular the ~ operator applies to the shifted value, not just to the 1.
This has the effect of setting just that bit to 0, and not touching any others. Just what you want, which of course is why it's the code you always see used. :)
I think the confusion happens because you've been given an inaccurate description of what's going on in the "standard" way.
I suspect you've been told that when you do this:
REGISTER_NAME &= ~(1 << PINXX);
it means "logically AND the pin at position PINXX with ~1, that is, with 0".
But it doesn't mean that. What it actually means depends on how big the register is. I'm going to assume that it's 16 bits, and PINXX is 8.
Then what the standard way actually says is:
REGISTER_NAME &= ~(1 << 8)
or (assuming you have a compiler that accepts 0b for binary numbers)
REGISTER_NAME &= ~(0b0000000100000000)
or
REGISTER_NAME &= 0b1111111011111111
That is, the command isn't operating on just one pin - it's operating on all of them. (Just the operation on everything but the pin you want to change is "logically AND with 1", which means it is left alone). The expression ~(1 << PINXX) is just a way to construct a number that's all 1s in the places you don't want to touch, and 0 in the spot you do.
Your proposed "simpler" version would be the equivalent of:
REGISTER_NAME &= 0b0000000000000000
Which would reset all the pins controlled by that register.
To set a bit, you variable |= (1 << position). This will generate something like 0000100, assuming position = 2.
To unset a bit, you do the reverse: you AND instead of OR, and the mask is the opposite of the other.
Assuming position = 2,
1 << position: 00000100 as before.
Reverse it with bitwise not (~): 11111011.
Apply bitwise AND. If it's `11011100, for example, you get:
11011100 &
11111011 =
-----------
11011000
_____^____
For the sake of purpose I assumed 8 bit values but the procedure has no changes with longer widths. It could seem a little hard the first times but, if you do the calculus yourself, the logic behind it should show up; it's math, after all.
Related
TL;DR:
How can I quickly find the smallest integer that satisfies int > floor && ((int ^ bits) & mask) == 0, where floor, bits and mask are known? (Ideally a non-looping, non-branching solution, probably with some serious bit-twiddling involved.)
Background
Suppose I have a list of key-values sorted by key, where each key is an integer. I want to quickly gather each value whose key has certain flags set or clear, i.e. matches the filter (key & mask) == (bits & mask) or equivalently ((key ^ bits) & mask) == 0.
In a naïve implementation, I could iterate over every key-value and gather those matching the filter. However, if my matches are sparse, I'd like to take advantage of my list's sortedness to quickly find new matches.
So my current approach is an algorithm that switches back and forth between binary-searching and scanning the list, maintaining the smallest possible future key based on the last key read in. If a search for the key succeeds, it scans until the the read-in key no longer matches the filter. Either way, the search begins anew with a newly computed "smallest possible future key" based on the last key we read in (aka floor, which doesn't match the filter.)
My testing has shown the algorithm to be correct (thankfully), but I'm using a placeholder iterative method to compute the "next" key instead of a bit-twiddling method like I was hoping to have come up with... which brings us back to the TL;DR!
Here is one approach, though it does not quite meet the strictest version of your criteria. It's not too bad though, a few branches is all.
The condition ((key ^ bits) & mask) == 0 (henceforth "The Condition") requires some bits to take on particular values, I will call those bits the "fixed bits" and other bits the "non-fixed bits". WLOG I will assume that (bits & mask) == bits.
Consider floor. How would it have to change to fit The Condition? Some bits may have to be set to match the fixed bits, some bits may have to be reset to match the fixed bits. Sometimes no bits would have to be changed. Let's consider some different cases:
floor already satisfies The Condition. But floor isn't larger than floor, so we have to increment it, but without violating The Condition. Incrementing a number of which some bits stay fixed has a known solution: (floor | mask) + 1 & ~mask | bits, which could be redundantly parenthesized to (((floor | mask) + 1) & ~mask) | bits
The highest "wrong" bit in floor, at index i, needs to be set. In that case, setting that bit would enable the lower non-fixed bits to become zero without going below floor. So we reset all the bits below i, and then bitwise-OR with bits to fill in the fixed bits.
The highest "wrong" bit in floor, at index i, needs to be reset. Resetting it would decrease the value below floor, so the non-fixed bits above i need to be incremented. We can use a variant of the trick seen in the first case, but instead of adding 1, we add 1 << i. Then again we reset all the bits below that and OR with bits.
The conditional addition of 1 << i if bit i of x is set, could be implemented as adding x & (1 << i).
There are different ways to actually implement that, for example (somewhat tested):
uint32_t next_above_with_pattern(uint32_t floor, uint32_t mask, uint32_t bits)
{
uint32_t x = floor;
if (((x ^ bits) & mask) == 0)
{
x = (x | mask) + 1 & ~mask | bits;
}
else
{
int highestIndexOfViolation = 31 - (int)_lzcnt_u32((x ^ bits) & mask);
x = (x | mask) + (x & (1UL << highestIndexOfViolation)) & ~mask | bits;
x &= ~((1UL << highestIndexOfViolation) - 1);
x |= bits;
}
return x;
}
With some changes as suggested by Ben
uint32_t next_above_with_patternD(uint32_t floor, uint32_t mask, uint32_t bits)
{
uint32_t x = floor;
if (((x ^ bits) & mask) == 0)
{
x = (((x | mask) + 1) & ~mask) | bits;
}
else
{
uint32_t maskOfHighestViolation = 0x80000000U >> _lzcnt_u32((x ^ bits) & mask);
x = (((x | mask) + (x & maskOfHighestViolation)) & ~mask) | bits;
x &= ~(maskOfHighestViolation - 1); // could be -maskOfHighestViolation
x |= bits;
}
return x;
}
If you had a bit swizzling operation, this would be really easy:
Starting with floor, swizzle away the bits controlled by mask leaving only the variable bits. Let this be v(floor)
Compute 1+v(floor)
For the results of both step 1 and 2, swizzle the mask-controlled bits back in. Compare to floor, picking the lower feasible value.
However, bit interleaving ("swizzling") is not one of the fundamental operations on a CPU (With an FPGA it is trivial)
Context: I'm writing an external SRAM tester for a microcontroller-based embedded system. No security, no cryptography involved. For reproducible access to "as-non-consecutive-as-possible" memory locations, I'm looking for an implementation of
y = shuffle(x), taking and returning an integer between 0 and a fixed N = 2^16 - 1
It may not use a lot of RAM itself, as e.g. a naive shuffled array of addresses would. On the upside, it is allowed to be slow. No strict definition of non-consecutive - it's about bumping address lines up and down, hunting for soldering and other faults of the printed circuit board. Suggestions?
I found so far Knuth shuffle a.k.a. Fisher-Yates shuffle.
A late EDIT: I think I'm looking to maximize the Hamming distance. "Anti-Grey code"?
I agree with Jim Mischel that xorshift is a good candidate for a fast non-crypto PRNG. Coefficients need to be carefully chosen to achieve full cycle, which includes all values except 0.
Since you wired the problem to 16 bits in your question, here's a 16 bit implementation written in Ruby, but easily ported to anything else you like:
# << is shift left, >> is shift right, ^ is bitwise XOR, & is bitwise AND
MASK_16 = (1 << 16) - 1
def xorshift(x)
x ^= x << 7 & MASK_16
x ^= x >> 9
x ^= x << 8 & MASK_16
return x
end
counter = 0
x = 1
y = 1
# Floyd's "Tortoise and Hare" cycle-finding algorithm shows
# the generator to be full cycle for 16 bits, excluding zero.
loop do
counter += 1
x = xorshift(x)
y = xorshift(xorshift(y))
break if x == y
end
puts counter # => 65535
I'd suggest implementing Xorshift, or something similar. They are fast, require little memory, can be constructed to have a long period, and satisfy many tests of randomness.
Another way to do it would be to uniquely map every number in the range 0..(n-1) to another number within that range. That's easy enough to do using a modular multiplicative inverse, as I describe in this answer.
I'm working on this question and come up with a solution (May be one or two condition needs to be added) but not sure if this is the right way to do it and find it cumbersome to use two loops and not sure if this is the efficient way of doing it. It would be great if anyone has some nice trick to do it or any better approach would be welcome :). (Language is not a barrier)
My Algorithm:
First find the first '0' lsb bit in the number
then find the next set bit which is next to this '0' bit
Change the one you find '0' bit to 1 & '1' bit to '0'
The number you'll get is next smaller
if all the bit is set then you don't have any number which is next smaller with the same number of '1' bits.
void nextSmaller(int number) {
int firstZeroBitHelper = 1, nextOneBitHelper;
while (firstZeroBitHelper < number) {
// when we find first lsb zero bit we'll stop
bool bit = number & firstZeroBitHelper;
if (bit == false)
break;
firstZeroBitHelper = firstZeroBitHelper << 1;
}
if (firstZeroBitHelper >= number) {
cout << "No minimum number exists" << endl;
return;
}
nextOneBitHelper = firstZeroBitHelper;
nextOneBitHelper = nextOneBitHelper << 1;
while (nextOneBitHelper < number) {
// when we get '1' after the previous zero we stop
bool bit = number & nextOneBitHelper;
if (bit == true)
break;
nextOneBitHelper = nextOneBitHelper << 1;
}
// change the first zero to 1
number = number | firstZeroBitHelper;
// change the next set bit to zero
number = number & ~nextOneBitHelper;
cout << number << endl;
}
Continuing from my comment..
Well, I found it, and pretty quickly too. See The Art of Computer Programming chapter 7.1.3 (in volume 4A), answer to question 21: "the reverse of Gosper's hack".
It looks like this:
t = y + 1;
u = t ^ y;
v = t & y;
x = v - (v & -v) / (u + 1);
Where y is the input and x the result. The same optimizations as in Gosper's hack apply to that division.
Going upwards:
Find the rightmost occurrence of "01" in the number and make it "10".
Justify all following 1-bits as far to the right as possible.
Going downwards:
Find the rightmost occurrence of "10" in the number and make it "01".
Left-justify all following 1-bits (i.e. don't do anything if the bit you just set is already followed by a 1).
An example to make the downwards case clear:
225 = 0b11100001
Swap: 0b11010001
Left-justify: 0b11011000 = 216
I'll explain the case of going upwards first, because it feels less tricky to me. We want to find the least-significant position where we can move a 1-bit one position left (in other words, the rightmost 0 that has a 1 to its right). It should be clear that this is the rightmost bit that we can possibly set, since we need to clear a bit somewhere else for every bit we set, and we need to clear a bit somewhere to the right of the bit we set, or else the number will get smaller instead of larger.
Now that we've set this bit, we want to clear one bit (to restore the total number of set bits), and reshuffle the remaining bits so that the number is as small as possible (this makes it the next greatest number with the same number of set bits). We can clear the bit to the right of the one we just set, and we can push any remaining 1-bits as far right as possible without fear of going below our original number, since all the less-significant bits together still add up to less than the single bit we just set.
Finding the next lower number instead of the next higher is basically the same, except that we're looking for the rightmost position where we can move a set bit one position right, and after doing that we want to move all less-significant bits as far left as possible.
It looks like others have got the bit-twiddling versions of this well in hand, but I wanted to see if I could give a good explanation of the logical/mathematical side of the algorithm.
anatolyg covered your algorithm pretty well, but there's a more efficient solution.
You can use Gosper's hack with a clever realization that if you flip the bits around, then Gosper's produces the values in descending order.
Something like this pseudocode would work:
let k := number
let n := num bits in k (log base 2)
k = k ^ ((1 << n) - 1)
k = gosper(k)
k = k ^ ((1 << n) - 1)
return k
This gives you a nice O(1) (or O(log n) if you consider xor to be linear time) algorithm. :)
There are some cases you have to consider, like if k=2^x-1 for some x, but that's pretty easy to catch.
The algorithm you described is not quite correct; it does everything right except one detail. Any binary number has the following form, found in the middle of your algorithm:
xxxxx...10000...1111...
---n---// f //
Here xxx... are arbitrary bits, and the numbers of consecutive zeros and ones are determined by firstZeroBitHelper and nextOneBitHelper (f and n).
Now you have to decrease this number leaving the same number of set bits, which necessarily turns the most significant 1 to 0:
xxxxx...0????...????...
-----n+f------
Note that any value for bits ??? makes the new number less than the original one, and you really want to choose these bits such that the resulting number has maximal value:
xxxxx...011111...0000...
---f+1--//n-1//
So you have to flip not just 2 bits, but f+2 bits (one bit from 1 to 0, and f+1 others from 0 to 1).
One way to do that is as follows.
First turn off all relevant bits:
number &= ~nextOneBitHelper;
number &= ~(nextOneBitHelper - 1);
Now turn on the needed bits, starting from MSB:
nextOneBitHelper >>= 1;
while (firstZeroBitHelper != 0)
{
number |= nextOneBitHelper;
nextOneBitHelper >>= 1;
firstZeroBitHelper >>= 1;
}
It is possible to implement the bit twiddling described above without loops; for that you would need to calculate n and f. Having done that:
unsigned mask = (1 << (f + 1)) - 1; // has f+1 bits set to 1
mask <<= n - 1; // now has these bits at correct positions
number |= mask; // now the number has these bits set
#include <iostream>
bool AlmostdivBy2(int num) {
return (-~num & (num)) == 0;
}
void toggleright(int &num) {
int t;
for (t = -1; num & 1; t++)
num >>= 1;
++num = (num << t) | ~-(1 << t);
}
void toggleleft(int &num) {
while (~num & 1)
num >>= 1; //Simply keep chopping off zeros
//~num & 1 checks if the number is even
//Even numbers have a zero at bit at the rightmost spot
}
int main() {
int value;
std::cin >> value;
if (!AlmostdivBy2(value)) {
(~value & 1) ? toggleleft(value) : toggleright(value);
}
std::cout << value << "\n";
return 0;
}
I think I might have over thought this one, but here is my explanation:
If the number is close to being a power of 2 i.e values like 1, 3, 7, 15, 31, ..., then there is no value smaller than it that could have the same number of ones in their binary representation. Therefore we don't worry about these numbers.
if the number is even, that is another easy fix, we simply keep chopping off zeros from the end until the number is odd
Odd numbers presented the most challange which is why it is recursive. First you had to find the first zero bit starting from the right of the number. When this is found, you add one to that number which will turn that last bit to a 1. As the recursion unwinds you keep shifting the bits to the left and adding one. When this is done, you have the next smallest.
Hope I didn't confuse you
EDIT
Worked on it more and here is a non recursive version of toggleright
void toggleright(int &num) {
int t = 1;
while ( (num >>= 1) & 1 && t++ );
num = (-~num << ~-t) | ~-(1 << t);
}
On Page 140 of Programming Pearls, 2nd Edition, Jon proposed an implementation of sets with bit vectors.
We'll turn now to two final structures that exploit the fact that our sets represent integers. Bit vectors are an old friend from Column 1. Here are their private data and functions:
enum { BITSPERWORD = 32, SHIFT = 5, MASK = 0x1F };
int n, hi, *x;
void set(int i) { x[i>>SHIFT] |= (1<<(i & MASK)); }
void clr(int i) { x[i>>SHIFT] &= ~(1<<(i & MASK)); }
int test(int i) { return x[i>>SHIFT] &= (1<<(i & MASK)); }
As I gathered, the central idea of a bit vector to represent an integer set, as described in Column 1, is that the i-th bit is turned on if and only if the integer i is in the set.
But I am really at a loss at the algorithms involved in the above three functions. And the book doesn't give an explanation.
I can only get that i & MASK is to get the lower 5 bits of i, while i>>SHIFT is to move i 5 bits toward the right.
Anybody would elaborate more on these algorithms? Bit operations always seem a myth to me, :(
Bit Fields and You
I'll use a simple example to explain the basics. Say you have an unsigned integer with four bits:
[0][0][0][0] = 0
You can represent any number here from 0 to 15 by converting it to base 2. Say we have the right end be the smallest:
[0][1][0][1] = 5
So the first bit adds 1 to the total, the second adds 2, the third adds 4, and the fourth adds 8. For example, here's 8:
[1][0][0][0] = 8
So What?
Say you want to represent a binary state in an application-- if some option is enabled, if you should draw some element, and so on. You probably don't want to use an entire integer for each one of these- it'd be using a 32 bit integer to store one bit of information. Or, to continue our example in four bits:
[0][0][0][1] = 1 = ON
[0][0][0][0] = 0 = OFF //what a huge waste of space!
(Of course, the problem is more pronounced in real life since 32-bit integers look like this:
[0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0][0] = 0
The answer to this is to use a bit field. We have a collection of properties (usually related ones) which we will flip on and off using bit operations. So, say, you might have 4 different lights on a piece of hardware that you want to be on or off.
3 2 1 0
[0][0][0][0] = 0
(Why do we start with light 0? I'll explain this in a second.)
Note that this is an integer, and is stored as an integer, but is used to represent multiple states for multiple objects. Crazy! Say we turn lights 2 and 1 on:
3 2 1 0
[0][1][1][0] = 6
The important thing you should note here: There's probably no obvious reason why lights 2 and 1 being on should equal six, and it may not be obvious how we would do anything with this scheme of information storage. It doesn't look more obvious if you add more bits:
3 2 1 0
[1][1][1][0] = 0xE \\what?
Why do we care about this? Do we have exactly one state for each number between 0 and 15?How are we going to manage this without some insane series of switch statements? Ugh...
The Light at the End
So if you've worked with binary arithmetic a bit before, you might realize that the relationship between the numbers on the left and the numbers on the right is, of course, base 2. That is:
1*(23) + 1*(22) + 1*(21) +0 *(20) = 0xE
So each light is present in the exponent of each term of the equation. If the light is on, there is a 1 next to its term- if the light is off, there is a zero. Take the time to convince yourself that there is exactly one integer between 0 and 15 that corresponds to each state in this numbering scheme.
Bit operators
Now that we have this done, let's take a second to see what bitshifting does to integers in this setup.
[0][0][0][1] = 1
When you shift bits to the left or the right in an integer, it literally moves the bits left and right. (Note: I 100% disavow this explanation for negative numbers! There be dragons!)
1<<2 = 4
[0][1][0][0] = 4
4>>1 = 2
[0][0][1][0] = 2
You will encounter similar behavior when shifting numbers represented with more than one bit. Also, it shouldn't be hard to convince yourself that x>>0 or x<<0 is just x. Doesn't shift anywhere.
This probably explains the naming scheme of the Shift operators to anyone who wasn't familiar with them.
Bitwise operations
This representation of numbers in binary can also be used to shed some light on the operations of bitwise operators on integers. Each bit in the first number is xor-ed, and-ed, or or-ed with its fellow number. Take a second to venture to wikipedia and familiarize yourself with the function of these Boolean operators - I'll explain how they function on numbers but I don't want to rehash the general idea in great detail.
...
Welcome back! Let's start by examining the effect of the OR (|) operator on two integers, stored in four bit.
OR OPERATOR ON:
[1][0][0][1] = 0x9
[1][1][0][0] = 0xC
________________
[1][1][0][1] = 0xD
Tough! This is a close analogue to the truth table for the boolean OR operator. Notice that each column ignores the adjacent columns and simply fills in the result column with the result of the first bit and the second bit OR'd together. Note also that the value of anything or'd with 1 is 1 in that particular column. Anything or'd with zero remains the same.
The table for AND (&) is interesting, though somewhat inverted:
AND OPERATOR ON:
[1][0][0][1] = 0x9
[1][1][0][0] = 0xC
________________
[1][0][0][0] = 0x8
In this case we do the same thing- we perform the AND operation with each bit in a column and put the result in that bit. No column cares about any other column.
Important lesson about this, which I invite you to verify by using the diagram above: anything AND-ed with zero is zero. Also, equally important- nothing happens to numbers that are AND-ed with one. They stay the same.
The final table, XOR, has behavior which I hope you all find predictable by now.
XOR OPERATOR ON:
[1][0][0][1] = 0x9
[1][1][0][0] = 0xC
________________
[0][1][0][1] = 0x5
Each bit is being XOR'd with its column, yadda yadda, and so on. But look closely at the first row and the second row. Which bits changed? (Half of them.) Which bits stayed the same? (No points for answering this one.)
The bit in the first row is being changed in the result if (and only if) the bit in the second row is 1!
The one lightbulb example!
So now we have an interesting set of tools we can use to flip individual bits. Let's go back to the lightbulb example and focus only on the first lightbulb.
0
[?] \\We don't know if it's one or zero while coding
We know that we have an operation that can always make this bit equal to one- the OR 1 operator.
0|1 = 1
1|1 = 1
So, ignoring the rest of the bulbs, we could do this
4_bit_lightbulb_integer |= 1;
and know for sure that we did nothing but set the first lightbulb to ON.
3 2 1 0
[0][0][0][?] = 0 or 1? \\4_bit_lightbulb_integer
[0][0][0][1] = 1
________________
[0][0][0][1] = 0x1
Similarly, we can AND the number with zero. Well- not quite zero- we don't want to affect the state of the other bits, so we will fill them in with ones.
I'll use the unary (one-argument) operator for bit negation. The ~ (NOT) bitwise operator flips all of the bits in its argument. ~(0X1):
[0][0][0][1] = 0x1
________________
[1][1][1][0] = 0xE
We will use this in conjunction with the AND bit below.
Let's do 4_bit_lightbulb_integer & 0xE
3 2 1 0
[0][1][0][?] = 4 or 5? \\4_bit_lightbulb_integer
[1][1][1][0] = 0xE
________________
[0][1][0][0] = 0x4
We're seeing a lot of integers on the right-hand-side which don't have any immediate relevance. You should get used to this if you deal with bit fields a lot. Look at the left-hand side. The bit on the right is always zero and the other bits are unchanged. We can turn off light 0 and ignore everything else!
Finally, you can use the XOR bit to flip the first bit selectively!
3 2 1 0
[0][1][0][?] = 4 or 5? \\4_bit_lightbulb_integer
[0][0][0][1] = 0x1
________________
[0][1][0][*] = 4 or 5?
We don't actually know what the value of * is now- just that flipped from whatever ? was.
Combining Bit Shifting and Bitwise operations
The interesting fact about these two operations is when taken together they allow you to manipulate selective bits.
[0][0][0][1] = 1 = 1<<0
[0][0][1][0] = 2 = 1<<1
[0][1][0][0] = 4 = 1<<2
[1][0][0][0] = 8 = 1<<3
Hmm. Interesting. I'll mention the negation operator here (~) as it's used in a similar way to produce the needed bit values for ANDing stuff in bit fields.
[1][1][1][0] = 0xE = ~(1<<0)
[1][1][0][1] = 0xD = ~(1<<1)
[1][0][1][1] = 0xB = ~(1<<2)
[0][1][1][1] = 0X7 = ~(1<<3)
Are you seeing an interesting relationship between the shift value and the corresponding lightbulb position of the shifted bit?
The canonical bitshift operators
As alluded to above, we have an interesting, generic method for turning on and off specific lights with the bit-shifters above.
To turn on a bulb, we generate the 1 in the right position using bit shifting, and then OR it with the current lightbulb positions. Say we want to turn on light 3, and ignore everything else. We need to get a bit shifting operation that ORs
3 2 1 0
[?][?][?][?] \\all we know about these values at compile time is where they are!
and 0x8
[1][0][0][0] = 0x8
Which is easy, thanks to bitshifting! We'll pick the number of the light and switch the value over:
1<<3 = 0x8
and then:
4_bit_lightbulb_integer |= 0x8;
3 2 1 0
[1][?][?][?] \\the ? marks have not changed!
And we can guarantee that the bit for the 3rd lightbulb is set to 1 and that nothing else has changed.
Clearing a bit works similarly- we'll use the negated bits table above to, say, clear light 2.
~(1<<2) = 0xB = [1][0][1][1]
4_bit_lightbulb_integer & 0xB:
3 2 1 0
[?][?][?][?]
[1][0][1][1]
____________
[?][0][?][?]
The XOR method of flipping bits is the same idea as the OR one.
So the canonical methods of bit switching are this:
Turn on the light i:
4_bit_lightbulb_integer|=(1<<i)
Turn off light i:
4_bit_lightbulb_integer&=~(1<<i)
Flip light i:
4_bit_lightbulb_integer^=(1<<i)
Wait, how do I read these?
In order to check a bit we can simply zero out all of the bits except for the one we care about. We'll then check to see if the resulting value is greater than zero- since this is the only value that could possibly be nonzero, it will make the entire integer nonzero if and only if it is nonzero. For example, to check bit 2:
1<<2:
[0][1][0][0]
4_bit_lightbulb_integer:
[?][?][?][?]
1<<2 & 4_bit_lightbulb_integer:
[0][?][0][0]
Remember from the previous examples that the value of ? didn't change. Remember also that anything AND 0 is 0. So, we can say for sure that if this value is greater than zero, the switch at position 2 is true and the lightbulb is zero. Similarly, if the value is off, the value of the entire thing will be zero.
(You can alternately shift the entire value of 4_bit_lightbulb_integer over by i bits and AND it with 1. I don't remember off the top of my head if one is faster than the other but I doubt it.)
So the canonical checking function:
Check if bit i is on:
if (4_bit_lightbulb_integer & 1<<i) {
\\do whatever
}
The specifics
Now that we have a complete set of tools for bitwise operations, we can look at the specific example here. This is basically the same idea- except a much more concise and powerful way of executing it. Let's look at this function:
void set(int i) { x[i>>SHIFT] |= (1<<(i & MASK)); }
From the canonical implementation I'm going to make a guess that this is trying to set some bits to 1! Let's take an integer and look at what's going on here if i feed the value 0x32 (50 in decimal) into i:
x[0x32>>5] |= (1<<(0x32 & 0x1f))
Well, that's a mess.. let's dissect this operation on the right. For convenience, pretend there are 24 more irrelevant zeros, since these are both 32 bit integers.
...[0][0][0][1][1][1][1][1] = 0x1F
...[0][0][1][1][0][0][1][0] = 0x32
________________________
...[0][0][0][1][0][0][1][0] = 0x12
It looks like everything is being cut off at the boundary on top where 1s turn into zeros. This technique is called Bit Masking. Interestingly, the boundary here restricts the resulting values to be between 0 and 31... Which is exactly the number of bit positions we have for a 32 bit integer!
x[0x32>>5] |= (1<<(0x12))
Let's look at the other half.
...[0][0][1][1][0][0][1][0] = 0x32
Shift five bits to the right:
...[0][0][0][0][0][0][0][1] = 0x01
Note that this transformation exactly destroyed all information from the first part of the function- we have 32-5 = 27 remaining bits which could be nonzero. This indicates which of 227 integers in the array of integers are selected. So the simplified equation is now:
x[1] |= (1<<0x12)
This just looks like the canonical bit-setting operation! We've just chosen
So the idea is to use the first 27 bits to pick an integer to shift and the last five bits indicate which bit of the 32 in that integer to shift.
The key to understanding what's going on is to recognize that BITSPERWORD = 2SHIFT. Thus, x[i>>SHIFT] finds which 32-bit element of the array x has the bit corresponding to i. (By shifting i 5 bits to the right, you're simply dividing by 32.) Once you have located the correct element of x, the lower 5 bits of i can then be used to find which particular bit of x[i>>SHIFT] corresponds to i. That's what i & MASK does; by shifting 1 by that number of bits, you move the bit corresponding to 1 to the exact position within x[i>>SHIFT] that corresponds to the ith bit in x.
Here's a bit more of an explanation:
Imagine that we want capacity for N bits in our bit vector. Since each int holds 32 bits, we will need (N + 31) / 32 int values for our storage (that is, N/32 rounded up). Within each int value, we will adopt the convention that bits are ordered from least significant to most significant. We will also adopt the convention that the first 32 bits of our vector are in x[0], the next 32 bits are in x[1], and so forth. Here's the memory layout we are using (showing the bit index in our bit vector corresponding to each bit of memory):
+----+----+-------+----+----+----+
x[0]: | 31 | 30 | . . . | 02 | 01 | 00 |
+----+----+-------+----+----+----+
x[1]: | 63 | 62 | . . . | 34 | 33 | 32 |
+----+----+-------+----+----+----+
etc.
Our first step is to allocate the necessary storage capacity:
x = new int[(N + BITSPERWORD - 1) >> SHIFT]
(We could make provision for dynamically expanding this storage, but that would just add complexity to the explanation.)
Now suppose we want to access bit i (either to set it, clear it, or just to know its current value). We need to first figure out which element of x to use. Since there are 32 bits per int value, this is easy:
subscript for x = i / 32
Making use of the enum constants, the x element we want is:
x[i >> SHIFT]
(Think of this as a 32-bit-wide window into our N-bit vector.) Now we have to find the specific bit corresponding to i. Looking at the memory layout, it's not hard to figure out that the first (rightmost) bit in the window corresponds to bit index 32 * (i >> SHIFT). (The window starts afteri >> SHIFT slots in x, and each slot has 32 bits.) Since that's the first bit in the window (position 0), then the bit we're interested in is is at position
i - (32 * (i >> SHIFT))
in the windows. With a little experimenting, you can convince yourself that this expression is always equal to i % 32 (actually, that's one definition of the mod operator) which, in turn, is always equal to i & MASK. Since this last expression is the fastest way to calculate what we want, that's what we'll use.
From here, the rest is pretty simple. We start with a single bit in the least-significant position of the window (that is, the constant 1), and move it to the left by i & MASK bits to get it to the position in the window corresponding to bit i in the bit vector. This is where the expression
1 << (i & MASK)
comes from. With the bit now moved to where we want it, we can use this as a mask to set, clear, or query the value of the bit at that position in x[i>>SHIFT] and we know that we're actually setting, clearing, or querying the value of bit i in our bit vector.
If you store your bits in an array of n words you can imagine them to be layed out as a matrix with n rows and 32 columns (BITSPERWORD):
3 0
1 0
0 xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx
1 xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx
2 xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx
....
n xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx
To get the k-th bit you divide k by 32. The (integer) result will give you the row (word) the bit is in, the reminder will give you which bit is within the word.
Dividing by 2^p can be done simply by shifting p postions to the right. The reminder can be obtained by getting the p rightmost bits (i.e the bitwise AND with (2^p - 1)).
In C terms:
#define div32(k) ((k) >> 5)
#define mod32(k) ((k) & 31)
#define word_the_bit_is_in(k) div32(k)
#define bit_within_word(k) mod32(k)
Hope it helps.
From "Signed Types" on Encoding - Protocol Buffers - Google Code:
ZigZag encoding maps signed integers to unsigned integers so that numbers with a small absolute value (for instance, -1) have a small varint encoded value too. It does this in a way that "zig-zags" back and forth through the positive and negative integers, so that -1 is encoded as 1, 1 is encoded as 2, -2 is encoded as 3, and so on, as you can see in the following table:
Signed Original Encoded As
0 0
-1 1
1 2
-2 3
2147483647 4294967294
-2147483648 4294967295
In other words, each value n is encoded using
(n << 1) ^ (n >> 31)
for sint32s, or
(n << 1) ^ (n >> 63)
for the 64-bit version.
How does (n << 1) ^ (n >> 31) equal whats in the table? I understand that would work for positives, but how does that work for say, -1? Wouldn't -1 be 1111 1111, and (n << 1) be 1111 1110? (Is bit-shifting on negatives well formed in any language?)
Nonetheless, using the fomula and doing (-1 << 1) ^ (-1 >> 31), assuming a 32-bit int, I get 1111 1111, which is 4 billion, whereas the table thinks I should have 1.
Shifting a negative signed integer to the right copies the sign bit, so that
(-1 >> 31) == -1
Then,
(-1 << 1) ^ (-1 >> 31) = -2 ^ -1
= 1
This might be easier to visualise in binary (8 bit here):
(-1 << 1) ^ (-1 >> 7) = 11111110 ^ 11111111
= 00000001
Another way to think about zig zag mapping is that it is a slight twist on a sign and magnitude representation.
In zig zag mapping, the least significant bit (lsb) of the mapping indicates the sign of the value: if it's 0, then the original value is non-negative, if it's 1, then the original value is negative.
Non-negative values are simply left shifted one bit to make room for the sign bit in the lsb.
For negative values, you could do the same one bit left shift for the absolute value (magnitude) of the number and simply have the lsb indicate the sign. For example, -1 could map to 0x03 or 0b00000011, where the lsb indicates that it is negative and the magnitude of 1 is left shifted by 1 bit.
The ugly thing about this sign and magnitude representation is "negative zero," mapped as 0x01 or 0b00000001. This variant of zero "uses up" one of our values and shifts the range of integers we can represent by one. We probably want to special case map negative zero to -2^63, so that we can represent the full 64b 2's complement range of [-2^63, 2^63). That means we've used one of our valuable single byte encodings to represent a value that will very, very, very rarely be used in an encoding optimized for small magnitude numbers and we've introduced a special case, which is bad.
This is where zig zag's twist on this sign and magnitude representation happens. The sign bit is still in the lsb, but for negative numbers, we subtract one from the magnitude rather than special casing negative zero. Now, -1 maps to 0x01 and -2^63 has a non-special case representation too (i.e. - magnitude 2^63 - 1, left shifted one bit, with lsb / sign bit set, which is all bits set to 1s).
So, another way to think about zig zag encoding is that it is a smarter sign and magnitude representation: the magnitude is left shifted one bit, the sign bit is stored in the lsb, and 1 is subtracted from the magnitude of negative numbers.
It is faster to implement these transformations using the unconditional bit-wise operators that you posted rather than explicitly testing the sign, special case manipulating negative values (e.g. - negate and subtract 1, or bitwise not), shifting the magnitude, and then explicitly setting the lsb sign bit. However, they are equivalent in effect and this more explicit sign and magnitude series of steps might be easier to understand what and why we are doing these things.
I will warn you though that bit shifting signed values in C / C++ is not portable and should be avoided. Left shifting a negative value has undefined behavior and right shifting a negative value has implementation defined behavior. Even left shifting a positive integer can have undefined behavior (e.g. - if you shift into the sign bit it might cause a trap or something worse). So, in general, don't bit shift signed types in C / C++. "Just say no."
Cast first to the unsigned version of the type to have safe, well-defined results according to the standards. This does mean that you then won't have arithmetic shift of negative values (i.e. - dragging the sign bit to the right) -- only logical shift, so you need to adjust the logic to account for that.
Here are the safe and portable versions of the zig zag mappings for 2's complement 64b integers in C:
#include <stdint.h>
uint64_t zz_map( int64_t x )
{
return ( ( uint64_t ) x << 1 ) ^ -( ( uint64_t ) x >> 63 );
}
int64_t zz_unmap( uint64_t y )
{
return ( int64_t ) ( ( y >> 1 ) ^ -( y & 0x1 ) );
}
Note the arithmetic negation of the sign bit in the right hand term of the XORs. That yields either 0 for non-negatives or all 1's for negatives -- just like arithmetic shift of the sign bit from msb to lsb would do. The XOR then effectively "undoes" / "redoes" the 2's complementation minus 1 (i.e. - 1's complementation or logical negation) for negative values without any conditional logic or further math.
Let me add my two cents to the discussion. As other answers noted, the zig-zag encoding can be thought as a sign-magnitude twist. This fact can be used to implement conversion functions which work for arbitrary-sized integers.
For example, I use the following code in one on my Python projects:
def zigzag(x: int) -> int:
return x << 1 if x >= 0 else (-x - 1) << 1 | 1
def zagzig(x: int) -> int:
assert x >= 0
sign = x & 1
return -(x >> 1) - 1 if sign else x >> 1
These functions work despite Python's int has no fixed bitwidth; instead, it extends dynamically. However, this approach may be inefficient in compiled languages since it requires conditional branching.