How do you write an integer to LEB128 format in Go? I'm trying to encode an int32 to a Minecraft VarInt, so far I've tried importing the example on the wiki to Go. I get the wrong results when testing though, wiki says -1 should equal [255 255 255 255 15], but I get [255 255 255 255 255] instead. What I'm I doing wrong here?
func WriteVarInt2(v int32) []byte{
var out []byte
c := 0
for{
currentByte := byte(v & 0b01111111)
v >>= 7
if v != 0 {
currentByte |= 0b10000000
}
out = append(out, currentByte)
c++
if c >= 5 || v == 0{
return out
}
}
}
The problem is with the shifting operation.
>> is arithmetic shift right, >>> is logical shift right. The difference is that >> brings in the sign bit (on the left), while >>> brings in zeros (whatever the sign bit was).
The algorithm of LEB128's Varint uses logical shift, and Go's >> is arithmetic shift.
There is no distinct logical shift in Go, but if you treat the number as unsigned, you'll get exactly that:
func WriteVarInt2(v_ int32) []byte {
v := uint32(v_)
// rest of your function unchanged
// ...
}
Testing it:
fmt.Println(WriteVarInt2(-1))
Output is as expected (try it on the Go Playground):
[255 255 255 255 15]
Related
If I'm right in C++ the LSB is the last bit and determines the sign of the integer. So for example in case of a 8 bit 1111 1111 will be -127 and 1111 1110 will be 127. Please correct me if I'm wrong, but it's not related.
I would check fir the sign of the integer in Go, so I wrote the following:
func signCheck(n int8) {
var sign int8 = 1<<7
if n&sign == 1 {
fmt.Printf("%d is odd - %b\n", n, n)
} else {
fmt.Printf("%d is even - %b\n", n, n)
}
}
This will print out "constant 128 overflows int8", but that make sense because because there is only 7 bit used for determine a number. So I modified as follow:
func signCheck(n int8) {
if n&1<<7 == 1 {
fmt.Printf("%d is odd - %b\n", n, n)
} else {
fmt.Printf("%d is even - %b\n", n, n)
}
}
In this case I don't have to say it's an int8 but I tested it with -127 and 127 and got the following prints:
-127 is even - -1111111
127 is even - 1111111
So in that case how should I check for the sign?
go version go1.13.1 linux/amd64
To represent negative integers, Go (and I believe C++ too like most languages) uses the 2's complement format and standard.
In 2's complement the highest bit (MSB) will be 1 if the number is negative, and 0 otherwise.
In Go you can't use a 0x80 typed constant having int8 as its type, because that's outside of its valid range.
You may however convert int8 to uint8 "losslessly", and then you may do so.
Also, when you're masking the 0x80 bit, you have to compare the result to 0x80, because x & 0x80 can never result in 1, only in 0 or 0x80.
func signCheck(n int8) {
if uint8(n)&0x80 == 0x80 {
fmt.Printf("%d is negative - %b\n", n, n)
} else {
fmt.Printf("%d is non-negative - %b\n", n, n)
}
}
Another option is to compare the masking result to 0:
if uint8(n)&0x80 != 0 { }
Both solutions give the output (try them on the Go Playground):
-127 is negative - -1111111
127 is non-negative - 1111111
I'm working on Tormenta (https://github.com/jpincas/tormenta) which is backed by BadgerDB (https://github.com/dgraph-io/badger). BadgerDB stores keys (slices of bytes) in byte order. I am creating keys which contain floats which need to be stored in order so I can user Badger's key iteration properly. I don't have a solid CS background so I'm a little out of my depth.
I encode the floats like this: binary.Write(buf, binary.BigEndian, myFloat). This works fine for positive floats - the key order is what you'd expect, but the byte ordering breaks down for negative floats.
As an aside, ints present the same problem, but I was able to fix that relatively easily by flipping the sign bit on the int with b[0] ^= 1 << 7 (where b is the []byte holding the result of encoding the int), and then flipping back when retrieving the key.
Although b[0] ^= 1 << 7 DOES also flip the sign bit on floats and thus places all the negative floats before the positive ones, the negative ones are incorrectly (backwards) ordered. It is necessary to flip the sign bit and reverse the order of the negative floats.
A similar question was asked on StackOverflow here: Sorting floating-point values using their byte-representation, and the solution was agreed to be:
XOR all positive numbers with 0x8000... and negative numbers with 0xffff.... This should flip the sign bit on both (so negative numbers go first), and then reverse the ordering on negative numbers.
However, that's way above my bit-flipping-skills level, so I was hoping a Go bit-ninja could help me translate that into some Go code.
Using math.Float64bits()
You could use math.Float64bits() which returns an uint64 value having the same bytes / bits as the float64 value passed to it.
Once you have an uint64, performing bitwise operations on it is trivial:
f := 1.0 // Some float64 value
bits := math.Float64bits(f)
if f >= 0 {
bits ^= 0x8000000000000000
} else {
bits ^= 0xffffffffffffffff
}
Then serialize the bits value instead of the f float64 value, and you're done.
Let's see this in action. Let's create a wrapper type holding a float64 number and its bytes:
type num struct {
f float64
data [8]byte
}
Let's create a slice of these nums:
nums := []*num{
{f: 1.0},
{f: 2.0},
{f: 0.0},
{f: -1.0},
{f: -2.0},
{f: math.Pi},
}
Serializing them:
for _, n := range nums {
bits := math.Float64bits(n.f)
if n.f >= 0 {
bits ^= 0x8000000000000000
} else {
bits ^= 0xffffffffffffffff
}
if err := binary.Write(bytes.NewBuffer(n.data[:0]), binary.BigEndian, bits); err != nil {
panic(err)
}
}
This is how we can sort them byte-wise:
sort.Slice(nums, func(i int, j int) bool {
ni, nj := nums[i], nums[j]
for k := range ni.data {
if bi, bj := ni.data[k], nj.data[k]; bi < bj {
return true // We're certain it's less
} else if bi > bj {
return false // We're certain it's not less
} // We have to check the next byte
}
return false // If we got this far, they are equal (=> not less)
})
And now let's see the order after byte-wise sorting:
fmt.Println("Final order byte-wise:")
for _, n := range nums {
fmt.Printf("% .7f %3v\n", n.f, n.data)
}
Output will be (try it on the Go Playground):
Final order byte-wise:
-2.0000000 [ 63 255 255 255 255 255 255 255]
-1.0000000 [ 64 15 255 255 255 255 255 255]
0.0000000 [128 0 0 0 0 0 0 0]
1.0000000 [191 240 0 0 0 0 0 0]
2.0000000 [192 0 0 0 0 0 0 0]
3.1415927 [192 9 33 251 84 68 45 24]
Without math.Float64bits()
Another option would be to first serialize the float64 value, and then perform XOR operations on the bytes.
If the number is positive (or zero), XOR the first byte with 0x80, and the rest with 0x00, which is basically do nothing with them.
If the number is negative, XOR all the bytes with 0xff, which is basically a bitwise negation.
In action: the only part that is different is the serialization and XOR operations:
for _, n := range nums {
if err := binary.Write(bytes.NewBuffer(n.data[:0]), binary.BigEndian, n.f); err != nil {
panic(err)
}
if n.f >= 0 {
n.data[0] ^= 0x80
} else {
for i, b := range n.data {
n.data[i] = ^b
}
}
}
The rest is the same. The output will also be the same. Try this one on the Go Playground.
I have a file containing two bytes, in Big Endian order, hexdump gives me:
81 50
which is 1000 0001 0101 0000 in binary. However, I want the most significant bit to be a flag, so in golang I have to load the file content, clear the most significant bit, and only then read the value.
So:
valueBuf := make([]byte, 2)
_, err := f.Read(valueBuf) // now printing valueBuf gives me [129 80] in decimal
value := int16(binary.BigEndian.Uint16(valueBuf[0:2])) // now value is -32432
Ok, I have tried to use something like:
func clearBit(n int16, pos uint) int16 {
mask := ^(1 << pos)
n &= mask
return n
}
But it apparently doesn't work as expected. The output value should be 336 in decimal, as normal int, and I cannot get it. How should I do this?
for n &= mask to work, n and mask have to be matching types. So you should write
mask := int16(^(1 << pos))
then, value = clearBit(value, 15) works fine.
Or, since constants are untyped, you can eliminate mask, and also eliminate the assignment to n since it's just returned on the following line, and shorten clearBit to
func clearBit(n int16, pos uint) int16 {
return n & ^(1 << pos)
}
So computers use Two's complement to internally represent signed integers. I.e., -5 is represented as ^5 + 1 = "1111 1011".
However, trying to print the binary representation, e.g. the following code:
var i int8 = -5
fmt.Printf("%b", i)
Outputs -101. Not quite what I'd expect. Is the formatting different or is it not using Two's complement after all?
Interestingly, converting to an unsigned int results in the "correct" bit pattern:
var u uint8 = uint(i)
fmt.Printf("%b", u)
Output is 11111011 - exactly the 2s complement of -5.
So it seems to me the value is internally the really using Two's complement, but the formatting is printing the unsigned 5 and prepending a -.
Can somebody clarify this?
I believe the answer lies in how the fmt module formats binary numbers, rather than the internal format.
If you take a look at fmt.integer, one of the very first actions that the function does is to convert the negative signed integer to a positive one:
165 negative := signedness == signed && a < 0
166 if negative {
167 a = -a
168 }
There's then logic to append - in front of the string that's output here.
IOW -101 really is - appended to 5 in binary.
Note: fmt.integer is called from pp.fmtInt64 in print.go, itself called from pp.printArg in the same function.
Here is a method without using unsafe:
package main
import (
"fmt"
"math/bits"
)
func unsigned8(x uint8) []byte {
b := make([]byte, 8)
for i := range b {
if bits.LeadingZeros8(x) == 0 {
b[i] = 1
}
x = bits.RotateLeft8(x, 1)
}
return b
}
func signed8(x int8) []byte {
return unsigned8(uint8(x))
}
func main() {
b := signed8(-5)
fmt.Println(b) // [1 1 1 1 1 0 1 1]
}
In this case you could also use [8]byte, but the above is better if you have
a positive integer, and want to trim the leading zeros.
https://golang.org/pkg/math/bits#RotateLeft
Unsafe pointers must be used to correctly represent negative numbers in binary format.
package main
import (
"fmt"
"strconv"
"unsafe"
)
func bInt8(n int8) string {
return strconv.FormatUint(uint64(*(*uint8)(unsafe.Pointer(&n))), 2)
}
func main() {
fmt.Println(bInt8(-5))
}
Output
11111011
I want to convert from int to hex in Golang.
In strconv, there is a method that converts strings to hex. Is there a similar method to get a hex string from an int?
Since hex is a Integer literal, you can ask the fmt package for a string representation of that integer, using fmt.Sprintf(), and the %x or %X format.
See playground
i := 255
h := fmt.Sprintf("%x", i)
fmt.Printf("Hex conv of '%d' is '%s'\n", i, h)
h = fmt.Sprintf("%X", i)
fmt.Printf("HEX conv of '%d' is '%s'\n", i, h)
Output:
Hex conv of '255' is 'ff'
HEX conv of '255' is 'FF'
"Hex" isn't a real thing. You can use a hexadecimal representation of a number, but there's no difference between 0xFF and 255. More info on that can be found in the docs which point out you can use 0xff to define an integer constant 255! As you mention, if you're trying to find the hexadecimal representation of an integer you could use strconv
package main
import (
"fmt"
"strconv"
)
func main() {
fmt.Println(strconv.FormatInt(255, 16))
// gives "ff"
}
Try it in the playground
If formatting some bytes, hex needs a 2 digits representation, with leading 0.
For exemple: 1 => '01', 15 => '0f', etc.
It is possible to force Sprintf to respect this :
h:= fmt.Sprintf("%02x", 14)
fmt.Println(h) // 0e
h2:= fmt.Sprintf("%02x", 231)
fmt.Println(h2) // e7
The pattern "%02x" means:
'0' force using zeros
'2' set the output size as two charactes
'x' to convert in hexadecimal
i := 4357640193405743614
h := fmt.Sprintf("%016x",i)
fmt.Printf("Decimal: %d,\nHexa: %s", i, h)
# Result
Decimal..: 4357640193405743614,
Hexa.....: 3c7972ab0ae9f1fe
Playground: https://play.golang.org/p/ndlMyBdQjmT
Sprintf is more versatile but FormatInt is faster. Choose what is better for you
func Benchmark_sprintf(b *testing.B) { // 83.8 ns/op
for n := 0; n < b.N; n++ {
_ = fmt.Sprintf("%x", n)
}
}
func Benchmark_formatint(b *testing.B) { // 28.5 ns/op
bn := int64(b.N)
for n := int64(0); n < bn; n++ {
_ = strconv.FormatInt(n, 16)
}
}
E.g. if its uint32, you can convert it to HEX as seen below =>
var p uint32
p = 4278190335
r := p >> 24 & 0xFF
g := p >> 16 & 0xFF
b := p >> 8 & 0xFF
fmt.Println(r, g, b)//255 0 0
DEMO
you can also check this online tool for ref. https://cryptii.com/pipes/integer-encoder