I generated a hash value using sha3 and I need to convert it to a big.Int value. Is it possible ? or is there a method to get the integervalue of the hash ?
the following code throws an error that cannot convert type hash.Hash to type int64 :
package main
import (
"math/big"
"golang.org/x/crypto/sha3"
"fmt"
)
func main(){
chall := "hello word"
b := byte[](chall)
h := sha3.New244()
h.Write(chall)
h.Write(b)
d := make([]byte, 16)
h.Sum(d)
val := big.NewInt(int64(h))
fmt.Println(val)
}
TL;DR;
sha3.New224() cannot be represented in uint64 type.
There are many hash types - and of differing sizes. Go standard library picks a very generic interface to cover all type of hashes: https://golang.org/pkg/hash/#Hash
type Hash interface {
io.Writer
Sum(b []byte) []byte
Reset()
Size() int
BlockSize() int
}
Having said that some Go hash implementations optionally include extra methods like hash.Hash64:
type Hash64 interface {
Hash
Sum64() uint64
}
others may implement encoding.BinaryMarshaler:
type BinaryMarshaler interface {
MarshalBinary() (data []byte, err error)
}
which one can use to preserve a hash state.
sha3.New224() does not implement the above 2 interfaces, but crc64 hash does.
To do a runtime check:
h64, ok := h.(hash.Hash64)
if ok {
fmt.Printf("64-bit: %d\n", h64.Sum64())
}
Working example: https://play.golang.org/p/uLUfw0gMZka
(See Peter's comment for the simpler version of this.)
Interpreting a series of bytes as a big.Int is the same as interpreting a series of decimal digits as an arbitrarily large number. For example, to convert the digits 1234 into a "number", you'd do this:
Start with 0
Multiply by 10 = 0
Add 1 = 1
Multiply by 10 = 10
Add 2 = 12
Multiply by 10 = 120
Add 3 = 123
Multiply by 10 = 1230
Add 4 = 1234
The same applies to bytes. The "digits" are just base-256 rather than base-10:
val := big.NewInt(0)
for i := 0; i < h.Size(); i++ {
val.Lsh(val, 8)
val.Add(val, big.NewInt(int64(d[i])))
}
(Lsh is a left-shift. Left shifting by 8 bits is the same as multiplying by 256.)
Playground
Related
If I have a slice of bytes in Go, similar to this:
numBytes := []byte { 0xFF, 0x10 }
How would I convert it to it's uint16 value (0xFF10, 65296)?
you may use binary.BigEndian.Uint16(numBytes)
like this working sample code (with commented output):
package main
import (
"encoding/binary"
"fmt"
)
func main() {
numBytes := []byte{0xFF, 0x10}
u := binary.BigEndian.Uint16(numBytes)
fmt.Printf("%#X %[1]v\n", u) // 0XFF10 65296
}
and see inside binary.BigEndian.Uint16(b []byte):
func (bigEndian) Uint16(b []byte) uint16 {
_ = b[1] // bounds check hint to compiler; see golang.org/issue/14808
return uint16(b[1]) | uint16(b[0])<<8
}
I hope this helps.
To combine two bytes into uint16
x := uint16(numBytes[i])<<8 | uint16(numBytes[i+1])
where i is the starting position of the uint16. So if your array is always only two items it would be x := uint16(numBytes[0])<<8 | uint16(numBytes[1])
Firstly you have a slice not an array - an array has a fixed size and would be declared like this [2]byte.
If you just have a 2 bytes slice, I wouldn't do anything fancy, I'd just do
numBytes := []byte{0xFF, 0x10}
n := int(numBytes[0])<<8 + int(numBytes[1])
fmt.Printf("n =0x%04X = %d\n", n, n)
Playground
EDIT: Just noticed you wanted uint16 - replace int with that in the above!
You can use the following unsafe conversion:
*(*uint16)(unsafe.Pointer(&numBytes[0])
So computers use Two's complement to internally represent signed integers. I.e., -5 is represented as ^5 + 1 = "1111 1011".
However, trying to print the binary representation, e.g. the following code:
var i int8 = -5
fmt.Printf("%b", i)
Outputs -101. Not quite what I'd expect. Is the formatting different or is it not using Two's complement after all?
Interestingly, converting to an unsigned int results in the "correct" bit pattern:
var u uint8 = uint(i)
fmt.Printf("%b", u)
Output is 11111011 - exactly the 2s complement of -5.
So it seems to me the value is internally the really using Two's complement, but the formatting is printing the unsigned 5 and prepending a -.
Can somebody clarify this?
I believe the answer lies in how the fmt module formats binary numbers, rather than the internal format.
If you take a look at fmt.integer, one of the very first actions that the function does is to convert the negative signed integer to a positive one:
165 negative := signedness == signed && a < 0
166 if negative {
167 a = -a
168 }
There's then logic to append - in front of the string that's output here.
IOW -101 really is - appended to 5 in binary.
Note: fmt.integer is called from pp.fmtInt64 in print.go, itself called from pp.printArg in the same function.
Here is a method without using unsafe:
package main
import (
"fmt"
"math/bits"
)
func unsigned8(x uint8) []byte {
b := make([]byte, 8)
for i := range b {
if bits.LeadingZeros8(x) == 0 {
b[i] = 1
}
x = bits.RotateLeft8(x, 1)
}
return b
}
func signed8(x int8) []byte {
return unsigned8(uint8(x))
}
func main() {
b := signed8(-5)
fmt.Println(b) // [1 1 1 1 1 0 1 1]
}
In this case you could also use [8]byte, but the above is better if you have
a positive integer, and want to trim the leading zeros.
https://golang.org/pkg/math/bits#RotateLeft
Unsafe pointers must be used to correctly represent negative numbers in binary format.
package main
import (
"fmt"
"strconv"
"unsafe"
)
func bInt8(n int8) string {
return strconv.FormatUint(uint64(*(*uint8)(unsafe.Pointer(&n))), 2)
}
func main() {
fmt.Println(bInt8(-5))
}
Output
11111011
I was playing around with Go and was wondering what the best way is to perform idiomatic type conversions in Go. Basically my problem lays within automatic type conversions between uint8, uint64, and float64. From my experience with other languages a multiplication of a uint8 with a uint64 will yield a uint64 value, but not so in go.
Here is an example that I build and I ask if this is the idiomatic way of writing this code or if I'm missing an important language construct.
package main
import ("math";"fmt")
const(Width=64)
func main() {
var index uint32
var bits uint8
index = 100
bits = 3
var c uint64
// This is the line of interest vvvv
c = uint64(math.Ceil(float64(index * uint32(bits))/float64(Width)))
fmt.Println("Test: %v\n", c)
}
From my point of view the calculation of the ceiling value seems unnecessary complex because of all the explicit type conversions.
Thanks!
There are no implicit type conversions for non-constant values.
You can write
var x float64
x = 1
But you cannot write
var x float64
var y int
y = 1
x = y
See the spec for reference.
There's a good reason, to not allow automatic/implicit type conversions, as they can
become very messy and one has to learn many rules to circumvent the various caveats
that may occur. Take the Integer Conversion Rules in C for example.
For example,
package main
import "fmt"
func CeilUint(a, b uint64) uint64 {
return (a + (b - 1)) / b
}
func main() {
const Width = 64
var index uint32 = 100
var bits uint8 = 3
var c uint64 = CeilUint(uint64(index)*uint64(bits), Width)
fmt.Println("Test:", c)
}
Output:
Test: 5
To add to #nemo terrific answer. The convenience of automatic conversion between numeric types in C is outweighed by the confusion it causes. See https://Golang.org/doc/faq#conversions. Thats why you can't even convert from int to int32 implicitly. See https://stackoverflow.com/a/13852456/12817546.
package main
import (
. "fmt"
. "strconv"
)
func main() {
i := 71
c := []interface{}{byte(i), []byte(string(i)), float64(i), i, rune(i), Itoa(i), i != 0}
checkType(c)
}
func checkType(s []interface{}) {
for k, _ := range s {
Printf("%T %v\n", s[k], s[k])
}
}
byte(i) creates a uint8 with a value of 71, []byte(string(i)) a []uint8 with [71], float64(i) float64 71, i int 71, rune(i) int32 71, Itoa(i) string 71 and i != 0 a bool with a value of true.
Since Go won't convert numeric types automatically for you (See https://stackoverflow.com/a/13851553/12817546) you have to convert between types manually. See https://stackoverflow.com/a/41419962/12817546. Note, Itoa(i) sets an "Integer to an ASCII". See comment in https://stackoverflow.com/a/10105983/12817546.
I want to assign string to bytes array:
var arr [20]byte
str := "abc"
for k, v := range []byte(str) {
arr[k] = byte(v)
}
Have another method?
Safe and simple:
[]byte("Here is a string....")
For converting from a string to a byte slice, string -> []byte:
[]byte(str)
For converting an array to a slice, [20]byte -> []byte:
arr[:]
For copying a string to an array, string -> [20]byte:
copy(arr[:], str)
Same as above, but explicitly converting the string to a slice first:
copy(arr[:], []byte(str))
The built-in copy function only copies to a slice, from a slice.
Arrays are "the underlying data", while slices are "a viewport into underlying data".
Using [:] makes an array qualify as a slice.
A string does not qualify as a slice that can be copied to, but it qualifies as a slice that can be copied from (strings are immutable).
If the string is too long, copy will only copy the part of the string that fits (and multi-byte runes may then be copied only partly, which will corrupt the last rune of the resulting string).
This code:
var arr [20]byte
copy(arr[:], "abc")
fmt.Printf("array: %v (%T)\n", arr, arr)
...gives the following output:
array: [97 98 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ([20]uint8)
I also made it available at the Go Playground
For example,
package main
import "fmt"
func main() {
s := "abc"
var a [20]byte
copy(a[:], s)
fmt.Println("s:", []byte(s), "a:", a)
}
Output:
s: [97 98 99] a: [97 98 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
Piece of cake:
arr := []byte("That's all folks!!")
I think it's better..
package main
import "fmt"
func main() {
str := "abc"
mySlice := []byte(str)
fmt.Printf("%v -> '%s'",mySlice,mySlice )
}
Check here: http://play.golang.org/p/vpnAWHZZk7
Go, convert a string to a bytes slice
You need a fast way to convert a []string to []byte type. To use in situations such as storing text data into a random access file or other type of data manipulation that requires the input data to be in []byte type.
package main
func main() {
var s string
//...
b := []byte(s)
//...
}
which is useful when using ioutil.WriteFile, which accepts a bytes slice as its data parameter:
WriteFile func(filename string, data []byte, perm os.FileMode) error
Another example
package main
import (
"fmt"
"strings"
)
func main() {
stringSlice := []string{"hello", "world"}
stringByte := strings.Join(stringSlice, " ")
// Byte array value
fmt.Println([]byte(stringByte))
// Corresponding string value
fmt.Println(string([]byte(stringByte)))
}
Output:
[104 101 108 108 111 32 119 111 114 108 100] hello world
Please check the link playground
Besides the methods mentioned above, you can also do a trick as
s := "hello"
b := *(*[]byte)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&s))))
Go Play: http://play.golang.org/p/xASsiSpQmC
You should never use this :-)
Ended up creating array specific methods to do this. Much like the encoding/binary package with specific methods for each int type. For example binary.BigEndian.PutUint16([]byte, uint16).
func byte16PutString(s string) [16]byte {
var a [16]byte
if len(s) > 16 {
copy(a[:], s)
} else {
copy(a[16-len(s):], s)
}
return a
}
var b [16]byte
b = byte16PutString("abc")
fmt.Printf("%v\n", b)
Output:
[0 0 0 0 0 0 0 0 0 0 0 0 0 97 98 99]
Notice how I wanted padding on the left, not the right.
http://play.golang.org/p/7tNumnJaiN
Arrays are values... slices are more like pointers. That is [n]type is not compatible with []type as they are fundamentally two different things. You can get a slice that points to an array by using arr[:] which returns a slice that has arr as it's backing storage.
One way to convert a slice of for example []byte to [20]byte is to actually allocate a [20]byte which you can do by using var [20]byte (as it's a value... no make needed) and then copy data into it:
buf := make([]byte, 10)
var arr [10]byte
copy(arr[:], buf)
Essentially what a lot of other answers get wrong is that []type is NOT an array.
[n]T and []T are completely different things!
When using reflect []T is not of kind Array but of kind Slice and [n]T is of kind Array.
You also can't use map[[]byte]T but you can use map[[n]byte]T.
This can sometimes be cumbersome because a lot of functions operate for example on []byte whereas some functions return [n]byte (most notably the hash functions in crypto/*).
A sha256 hash for example is [32]byte and not []byte so when beginners try to write it to a file for example:
sum := sha256.Sum256(data)
w.Write(sum)
they will get an error. The correct way of is to use
w.Write(sum[:])
However, what is it that you want? Just accessing the string bytewise? You can easily convert a string to []byte using:
bytes := []byte(str)
but this isn't an array, it's a slice. Also, byte != rune. In case you want to operate on "characters" you need to use rune... not byte.
If someone is looking for a quick consider use unsafe conversion between slices, you can refer to the following comparison.
package demo_test
import (
"testing"
"unsafe"
)
var testStr = "hello world"
var testBytes = []byte("hello world")
// Avoid copying the data.
func UnsafeStrToBytes(s string) []byte {
return *(*[]byte)(unsafe.Pointer(&s))
}
// Avoid copying the data.
func UnsafeBytesToStr(b []byte) string {
return *(*string)(unsafe.Pointer(&b))
}
func Benchmark_UnsafeStrToBytes(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = UnsafeStrToBytes(testStr)
}
}
func Benchmark_SafeStrToBytes(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = []byte(testStr)
}
}
func Benchmark_UnSafeBytesToStr(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = UnsafeBytesToStr(testBytes)
}
}
func Benchmark_SafeBytesToStr(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = string(testBytes)
}
}
go test -v -bench="^Benchmark" -run=none
output
cpu: Intel(R) Core(TM) i7-8565U CPU # 1.80GHz
Benchmark_UnsafeStrToBytes
Benchmark_UnsafeStrToBytes-8 1000000000 0.2465 ns/op
Benchmark_SafeStrToBytes
Benchmark_SafeStrToBytes-8 289119562 4.181 ns/op
Benchmark_UnSafeBytesToStr
Benchmark_UnSafeBytesToStr-8 1000000000 0.2530 ns/op
Benchmark_SafeBytesToStr
Benchmark_SafeBytesToStr-8 342842938 3.623 ns/op
PASS
I'm trying to encode a large number to a list of bytes(uint8 in Go).
The number of bytes is unknown, so I'd like to use vector.
But Go doesn't provide vector of byte, what can I do?
And is it possible to get a slice of such a byte vector?
I intends to implement data compression.
Instead of store small and large number with the same number of bytes,
I'm implements a variable bytes that uses less bytes with small number
and more bytes with large number.
My code can not compile, invalid type assertion:
1 package main
2
3 import (
4 //"fmt"
5 "container/vector"
6 )
7
8 func vbEncodeNumber(n uint) []byte{
9 bytes := new(vector.Vector)
10 for {
11 bytes.Push(n % 128)
12 if n < 128 {
13 break
14 }
15 n /= 128
16 }
17 bytes.Set(bytes.Len()-1, bytes.Last().(byte)+byte(128))
18 return bytes.Data().([]byte) // <-
19 }
20
21 func main() { vbEncodeNumber(10000) }
I wish to writes a lot of such code into binary file,
so I wish the func can return byte array.
I haven't find a code example on vector.
Since you're trying to represent large numbers, you might see if the big package serves your purposes.
The general Vector struct can be used to store bytes. It accepts an empty interface as its type, and any other type satisfies that interface. You can retrieve a slice of interfaces through the Data method, but there's no way to convert that to a slice of bytes without copying it. You can't use type assertion to turn a slice of interface{} into a slice of something else. You'd have to do something like the following at the end of your function: (I haven't tried compiling this code because I can't right now)
byteSlice = make([]byte, bytes.Len())
for i, _ := range byteSlice {
byteSlice[i] = bytes.At(i).(byte)
}
return byteSlice
Take a look at the bytes package and the Buffer type there. You can write your ints as bytes into the buffer and then you can use the Bytes() method to access byte slices of the buffer.
I've found the vectors to be a lot less useful since the generic append and copy were added to the language. Here's how I'd do it in one shot with less copying:
package main
import "fmt"
func vbEncodeNumber(n uint) []byte {
bytes := make([]byte, 0, 4)
for n > 0 {
bytes = append(bytes, byte(n%256))
n >>= 8
}
return bytes
}
func main() {
bytes := vbEncodeNumber(10000)
for i := len(bytes)-1; i >= 0 ; i-- {
fmt.Printf("%02x ", bytes[i])
}
fmt.Println("")
}