I have some c# code that performs some bitwise operations on a byte. I am trying to do the same in golang but am having difficulties.
Example in c#
byte a, c;
byte[] data;
int j;
c = data[j];
c = (byte)(c + j);
c ^= a;
c ^= 0xFF;
c += 0x48;
I have read that golang cannot perform bitwise operations on the byte type. Therefore will I have to modify my code to a type uint8 to perform these operations? If so is there a clean and correct/standard way to implement this?
Go certainly can do bitwise operations on the byte type, which is simply an alias of uint8. The only changes I had to make to your code were:
Syntax of the variable declarations
Convert j to byte before adding it to c, since Go lacks (by design) integer promotion conversions when doing arithmetic.
Removing the semicolons.
Here you go
var a, c byte
var data []byte
var j int
c = data[j]
c = c + byte(j)
c ^= a
c ^= 0xFF
c += 0x48
If you're planning to do bitwise-not in Go, note that the operator for that is ^, not the ~ that is used in most other contemporary programming languages. This is the same operator that is used for xor, but the two are not ambiguous, since the compiler can tell which is which by determining whether the ^ is used as a unary or binary operator.
Related
I'm computing CRC32 in a rolling fashion on the contents of a file. If the file has 3 blocks ABC, CRC32 is computed linearly CRC(CRC(CRC(A, 0xffffffff), B), C). This is done with code that looks like:
uint32_t crc32(unsigned char const *buf, uint32_t buf_size, uint32_t crc32) {
for (int i = 0; i < buf_size; i++)
crc32 = (crc32 >> 8) ^ table[(crc32 ^ buf[i]) & 0xff];
return crc32;
}
Even though I write entire content ABC at once, computing the CRC as above(which gets verified at the server), read is normally done on a specific block. So, I would like to track CRC32 of each individual block as it is written.
Based on my limited understanding of how CRC32 polynomial works,
A mod G = CRC1
AB mod G = CRC2
If I want CRC32 of B, I'm thinking following should do the trick:
(CRC2 - CRC1) mod G
or
(CRC2 ^ CRC1) mod G
Of course, following code doesn't work.
uint32_t
crc32sw_diff(uint32_t crc1, uint32_t crc2)
{
uint32_t delta = crc1 ^ crc2;
return crc32(&delta, 4, 0xffffffff);
}
Other option is probably to compute CRC32 of individual blocks and combine it with something like zlib's crc32_combine() to get CRC32 of entire file.
See this answer for how CRC combination works. CRC(A) ^ CRC(B) is not equal to CRC(AB). However (for pure CRCs) using the notation that AB is the concatenated message of A followed by B, and 0 meaning an equal length message will all zeros, then CRC(A0) ^ CRC(0B) is equal to CRC(AB).
This also means that CRC(A0) ^ CRC(AB) == CRC(0B). Since CRC(0B) == CRC(B) (feeding zeros doesn't change a pure CRC), you can find it using crc32_combine() from zlib.
So, crc32_combine(crca, crcab, lenb) will return crcb.
I tried the following code as a naive attempt to implement swapping of R and B bytes in an ABGR word
#include <stdio.h>
#include <stdint.h>
uint32_t ABGR_to_ARGB(uint32_t abgr)
{
return ((abgr ^= (abgr >> 16) & 0xFF) ^= (abgr & 0xFF) << 16) ^= (abgr >> 16) & 0xFF;
}
int main()
{
uint32_t tmp = 0x11223344;
printf("%x %x\n", tmp, ABGR_to_ARGB(tmp));
}
To my surprise this code "worked" in GCC in C++17 mode - the bytes were swapped
http://coliru.stacked-crooked.com/a/43d0fc47f5539746
But it is not supposed to swap bytes! C++17 clearly states that the RHS of assignment is supposed to be [fully] sequenced before the LHS, which applies to compound assignment as well. This means that in the above expression each RHS of each ^= is supposed to use the original value of abgr. Hence the ultimate result in abgr should simply have B byte XORed by R byte. This is what Clang appears to produce (amusingly, with a sequencing warning)
http://coliru.stacked-crooked.com/a/eb9bdc8ced1b5f13
A quick look at GCC assembly
https://godbolt.org/g/1hsW5a
reveals that it seems to sequence it backwards: LHS before RHS. Is this a bug? Or is this some sort of conscious decision on GCC's part? Or am I misunderstanding something?
The exact same behavior is exhibited by int a = 1; (a += a) += a;, for which GCC calculates a == 4 afterwards and clang a == 3.
The underlying ambiguity arises from this part of the standard (from working draft N4762):
[expr.ass]: 7.6.18 Assignment and compound assignment operators
Paragraph 1: The assignment operator (=) and the compound assignment operators all group right-to-left. All require a
modifiable lvalue as their left operand; their result is an lvalue referring to the left operand. The result in all
cases is a bit-field if the left operand is a bit-field. In all cases, the assignment is sequenced after the value
computation of the right and left operands, and before the value computation of the assignment expression.
The right operand is sequenced before the left operand. With respect to an indeterminately-sequenced
function call, the operation of a compound assignment is a single evaluation.
Paragraph 7: The behavior of an expression of the form E1 op = E2 is equivalent to E1 = E1 op E2 except that E1 is
evaluated only once. In += and -=, E1 shall either have arithmetic type or be a pointer to a possibly
cv-qualified completely-defined object type. In all other cases, E1 shall have arithmetic type.
GCC seems to be using this rule to internally transfrom (a += a) += a to (a = a + a) += a to a = (a = a + a) + a (since a = a + a has to be evaluated only once) - and for this expression the sequencing rules are correctly applied.
Clang however seems to do that last transformation step differently: auto temp = a + a; temp = temp + a; a = temp;
Both compilers give a warning about this, though (from the original code):
GCC: warning: operation on 'abgr' may be undefined [-Wsequence-point]
clang: warning: unsequenced modification and access to 'abgr' [-Wunsequenced]
So the compiler writers know about this ambiguity and decided to prioritize differently (GCC: Paragraph 7 > Paragraph 1; clang: Paragraph 1 > Paragraph 7).
This seems to be a defect in the standard.
Do not make things more complicated than necessary. You can swap the 2 components in a fairly straightforward way without painting yourself into dark corners of the language:
uint32_t ABGR_to_ARGB(uint32_t abgr) {
constexpr uint32_t mask = 0xff00ff00;
uint32_t grab = abgr >> 16 | abgr << 16;
return (abgr & mask) | (grab & ~mask);
}
It also generates much better assembly than the original version. On x86 it uses single rol instruction for the 3 bitwise operators to produce grab:
ABGR_to_ARGB(unsigned int):
mov eax, edi
and edi, -16711936
rol eax, 16
and eax, 16711935
or eax, edi
ret
Note: This question is different from Fastest way to calculate a 128-bit integer modulo a 64-bit integer.
Here's a C# fiddle:
https://dotnetfiddle.net/QbLowb
Given the pseudocode:
UInt64 a = 9228496132430806238;
UInt32 d = 585741;
How do i calculate
UInt32 r = a % d?
The catch, of course, is that i am not in a compiler that supports the UInt64 data type.1 But i do have access to the Windows ULARGE_INTEGER union:
typedef struct ULARGE_INTEGER {
DWORD LowPart;
DWORD HighPart;
};
Which means really that i can turn my code above into:
//9228496132430806238 = 0x80123456789ABCDE
UInt32 a = 0x80123456; //high part
UInt32 b = 0x789ABCDE; //low part
UInt32 r = 585741;
How to do it
But now comes how to do the actual calculation. I can start with the pencil-and-paper long division:
________________________
585741 ) 0x80123456 0x789ABCDE
To make it simpler, we can work in variables:
Now we are working entirely with 32-bit unsigned types, which my compiler does support.
u1 = a / r; //integer truncation math
v1 = a % r; //modulus
But now i've brought myself to a standstill. Because now i have to calculate:
v1||b / r
In other words, I have to perform division of a 64-bit value, which is what i was unable to perform in the first place!
This must be a solved problem already. But the only questions i can find on Stackoverflow are people trying to calculate:
a^b mod n
or other cryptographically large multi-precision operations, or approximate floating point.
Bonus Reading
Microsoft Research: Division and Modulus for Computer Scientists
https://stackoverflow.com/questions/36684771/calculating-large-mods-by-hand
Fastest way to calculate a 128-bit integer modulo a 64-bit integer (unrelated question; i hate you people)
1But it does support Int64, but i don't think that helps me
Working with Int64 support
I was hoping for the generic solution to the performing modulus against a ULARGE_INTEGER (and even LARGE_INTEGER), in a compiler without native 64-bit support. That would be the correct, good, perfect, and ideal answer, which other people will be able to use when they need.
But there is also the reality of the problem i have. And it can lead to an answer that is generally not useful to anyone else:
cheating by calling one of the Win32 large integer functions (although there is none for modulus)
cheating by using 64-bit support for signed integers
I can check if a is positive. If it is, i know my compiler's built-in support for Int64 will handle:
UInt32 r = a % d; //for a >= 0
Then there's there's how to handle the other case: a is negative
UInt32 ModU64(ULARGE_INTEGER a, UInt32 d)
{
//Hack: Our compiler does support Int64, just not UInt64.
//Use that Int64 support if the high bit in a isn't set.
Int64 sa = (Int64)a.QuadPart;
if (sa >= 0)
return (sa % d);
//sa is negative. What to do...what to do.
//If we want to continue to work with 64-bit integers,
//we could now treat our number as two 64-bit signed values:
// a == (aHigh + aLow)
// aHigh = 0x8000000000000000
// aLow = 0x0fffffffffffffff
//
// a mod d = (aHigh + aLow) % d
// = ((aHigh % d) + (aLow % d)) % d //<--Is this even true!?
Int64 aLow = sa && 0x0fffffffffffffff;
Int64 aHigh = 0x8000000000000000;
UInt32 rLow = aLow % d; //remainder from low portion
UInt32 rHigh = aHigh % d; //this doesn't work, because it's "-1 mod d"
Int64 r = (rHigh + rLow) % d;
return d;
}
Answer
It took a while, but i finally got an answer. I would post it as an answer; but Z29kIGZ1Y2tpbmcgZGFtbiBzcGVybSBidXJwaW5nIGNvY2tzdWNraW5nIHR3YXR3YWZmbGVz people mistakenly decided that my unique question was an exact duplicate.
UInt32 ModU64(ULARGE_INTEGER a, UInt32 d)
{
//I have no idea if this overflows some intermediate calculations
UInt32 Al = a.LowPart;
UInt32 Ah = a.HighPart;
UInt32 remainder = (((Ah mod d) * ((0xFFFFFFFF - d) mod d)) + (Al mod d)) mod d;
return remainder;
}
Fiddle
I just updated my ALU32 class code in this related QA:
Cant make value propagate through carry
As CPU assembly independent code for mul,div was requested. The divider is solving all your problems. However it is using Binary long division so its a bit slover than stacking up 32 bit mul/mod/div operations. Here the relevant part of code:
void ALU32::div(DWORD &c,DWORD &d,DWORD ah,DWORD al,DWORD b)
{
DWORD ch,cl,bh,bl,h,l,mh,ml;
int e;
// edge cases
if (!b ){ c=0xFFFFFFFF; d=0xFFFFFFFF; cy=1; return; }
if (!ah){ c=al/b; d=al%b; cy=0; return; }
// align a,b for binary long division m is the shifted mask of b lsb
for (bl=b,bh=0,mh=0,ml=1;bh<0x80000000;)
{
e=0; if (ah>bh) e=+1; // e = cmp a,b {-1,0,+1}
else if (ah<bh) e=-1;
else if (al>bl) e=+1;
else if (al<bl) e=-1;
if (e<=0) break; // a<=b ?
shl(bl); rcl(bh); // b<<=1
shl(ml); rcl(mh); // m<<=1
}
// binary long division
for (ch=0,cl=0;;)
{
sub(l,al,bl); // a-b
sbc(h,ah,bh);
if (cy) // a<b ?
{
if (ml==1) break;
shr(mh); rcr(ml); // m>>=1
shr(bh); rcr(bl); // b>>=1
continue;
}
al=l; ah=h; // a>=b ?
add(cl,cl,ml); // c+=m
adc(ch,ch,mh);
}
cy=0; c=cl; d=al;
if ((ch)||(ah)) cy=1; // overflow
}
Look the linked QA for description of the class and used subfunctions. The idea behind a/b is simple:
definition
lets assume that we got 64/64 bit division (modulus will be a partial product) and want to use 32 bit arithmetics so:
(ah,al) / (bh,bl) = (ch,cl)
each 64bit QWORD will be defined as high and low 32bit DWORD.
align a,b
exactly like computing division on paper we must align b so it divides a so find sh that:
(bh,bl)<<sh <= (ah,al)
(bh,bl)<<(sh+1) > (ah,al)
and compute m so
(mh,ml) = 1<<sh
beware that in case bh>=0x80000000 stop the shifting or we would overflow ...
divide
set result c = 0 and then simply substract b from a while b>=a. For each substraction add m to c. Once b>a shift both b,m right to align again. Stop if m==0 or a==0.
result
c will hold 64bit result of division so use cl and similarly a holds the remainder so use al as your modulus result. You can check if ch,ah are zero if not overflow occurs (as result is bigger than 32 bit). The same goes for edge cases like division by zero...
Now as you want 64bit/32bit simply set bh=0 ... To do this I needed 64bit operations (+,-,<<,>>) which I did by stacking up 32bit operations with Carry (that is the reason why my ALU32 class was created in the first place) for more info see the link above.
we are sending some data over a serial line, and i can do pretty much everything via a bash script (instead of code), except for the crc16 calculation. if i can do it all in scripts versus code, it would make configuration a heckofalot easier (especially while in the field).
i'm alright with commands, but i lose all ability when we get to the tricky stuff.
so my question is, can someone do a rewrite of this CRC16 for me, but within bash?
here is the algorithm grabbed from wikipedia, and it is the one in our code:
uint16_t Encoder::checksum(std::string thestring)
{
uint8_t d, e, f;
uint16_t c, r, crccalc;
c = 0xffff;
for (unsigned int i = 0; i < thestring.length(); i++)
{
d = thestring[i];
e = c ^ d;
f = e ^ (e << 4);
r = (c >> 8) ^ (f << 8) ^ (f << 3) ^ (f >> 4);
c = r;
}
c ^= 0xffff;
crccalc = c;
return crccalc;
}
i can easily create an executable out of the C++ code, and just feed it stdin, but i think it would be really neat to be able to have this within the bash.
the other thing i don't know is how to ensure that my variable sizes are correct. how can i ensure that i am getting a 16 bit integer?
any help would be great. i found a little script online, but i didn't trust it. thought it would be really cool to have answered here.
Bash have:
xor ($((5^2)) will be 7);
left shift ($(3<<2) will be 12);
right shift ($(8>>2) will be 2);
hexademical numbers support ($((0xFF)) will be 255).
Nothing comes to mind to convert from 32 (64) to 16 bit integer in pure Bash but you can do it with awk:
$ echo 65536 | awk '{printf("%hu\n",$1)}'
0
This should be enough to rewrite algorithm in Bash.
How can we take care of the overflow happening during swapping of two variables without using a third variable. I believe the XOR solution can be used only for integers. what about other variable types?
This isn't an answer but it doesn't fit in a comment.
Under what circumstances would you be running so close to the edge of your available stack storage that the additional use of a temporary variable for the swap is going to cause you difficulties?
I could see some embedded scenarios, but I'm hard pressed to imagine a scenario where you'd be so tight on stack space that this would matter (where you're not writing code in assembly language).
XOR will work for anything you can get your XOR operator to process; it's a property of binary data, not of binary data used to represent integers.
By not doing it at all. The XOR swap algorithm is cool hack. It shouldn't be used in production code.
The XOR solution works with any type that can be bitwise copied, not just integers. However, do not XOR a variable with itself: i.e.
int x = 10;
int *p1 = &x;
int *p2 = p1;
*p1 = *p1 ^ *p2;
*p2 = *p1 ^ *p2;
*p1 = *p1 ^ *p2;
/* now x == 0 :( */
What's wrong with XCHG? No stack needed, no overflow(carry flag)? set either :).
a = a + b;
b = a - b;
a = a - b;
This will work for integers and float.