Convert an integer to a byte array - go

I have a function which receives a []byte but what I have is an int, what is the best way to go about this conversion ?
err = a.Write([]byte(myInt))
I guess I could go the long way and get it into a string and put that into bytes, but it sounds ugly and I guess there are better ways to do it.

I agree with Brainstorm's approach: assuming that you're passing a machine-friendly binary representation, use the encoding/binary library. The OP suggests that binary.Write() might have some overhead. Looking at the source for the implementation of Write(), I see that it does some runtime decisions for maximum flexibility.
func Write(w io.Writer, order ByteOrder, data interface{}) error {
// Fast path for basic types.
var b [8]byte
var bs []byte
switch v := data.(type) {
case *int8:
bs = b[:1]
b[0] = byte(*v)
case int8:
bs = b[:1]
b[0] = byte(v)
case *uint8:
bs = b[:1]
b[0] = *v
...
Right? Write() takes in a very generic data third argument, and that's imposing some overhead as the Go runtime then is forced into encoding type information. Since Write() is doing some runtime decisions here that you simply don't need in your situation, maybe you can just directly call the encoding functions and see if it performs better.
Something like this:
package main
import (
"encoding/binary"
"fmt"
)
func main() {
bs := make([]byte, 4)
binary.LittleEndian.PutUint32(bs, 31415926)
fmt.Println(bs)
}
Let us know how this performs.
Otherwise, if you're just trying to get an ASCII representation of the integer, you can get the string representation (probably with strconv.Itoa) and cast that string to the []byte type.
package main
import (
"fmt"
"strconv"
)
func main() {
bs := []byte(strconv.Itoa(31415926))
fmt.Println(bs)
}

Check out the "encoding/binary" package. Particularly the Read and Write functions:
binary.Write(a, binary.LittleEndian, myInt)

Sorry, this might be a bit late. But I think I found a better implementation on the go docs.
buf := new(bytes.Buffer)
var num uint16 = 1234
err := binary.Write(buf, binary.LittleEndian, num)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x", buf.Bytes())

i thought int type has any method for getting int hash to bytes, but first i find math / big method for this
https://golang.org/pkg/math/big/
var f int = 52452356235; // int
var s = big.NewInt(int64(f)) // int to big Int
var b = s.Bytes() // big Int to bytes
// b - byte slise
var r = big.NewInt(0).SetBytes(b) // bytes to big Int
var i int = int(r.Int64()) // big Int to int
https://play.golang.org/p/VAKSGw8XNQq
However, this method uses an absolute value.
If you spend 1 byte more, you can transfer the sign
func IntToBytes(i int) []byte{
if i > 0 {
return append(big.NewInt(int64(i)).Bytes(), byte(1))
}
return append(big.NewInt(int64(i)).Bytes(), byte(0))
}
func BytesToInt(b []byte) int{
if b[len(b)-1]==0 {
return -int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
return int(big.NewInt(0).SetBytes(b[:len(b)-1]).Int64())
}
https://play.golang.org/p/mR5Sp5hu4jk
or new(https://play.golang.org/p/7ZAK4QL96FO)
(The package also provides functions for fill into an existing slice)
https://golang.org/pkg/math/big/#Int.FillBytes

Adding this option for dealing with basic uint8 to byte[] conversion
foo := 255 // 1 - 255
ufoo := uint16(foo)
far := []byte{0,0}
binary.LittleEndian.PutUint16(far, ufoo)
bar := int(far[0]) // back to int
fmt.Println("foo, far, bar : ",foo,far,bar)
output :
foo, far, bar : 255 [255 0] 255

Here is another option, based on the Go source code [1]:
package main
import (
"encoding/binary"
"fmt"
"math/bits"
)
func encodeUint(x uint64) []byte {
buf := make([]byte, 8)
binary.BigEndian.PutUint64(buf, x)
return buf[bits.LeadingZeros64(x) >> 3:]
}
func main() {
for x := 0; x <= 64; x += 8 {
buf := encodeUint(1<<x-1)
fmt.Println(buf)
}
}
Result:
[]
[255]
[255 255]
[255 255 255]
[255 255 255 255]
[255 255 255 255 255]
[255 255 255 255 255 255]
[255 255 255 255 255 255 255]
[255 255 255 255 255 255 255 255]
Much faster than math/big:
BenchmarkBig-12 28348621 40.62 ns/op
BenchmarkBit-12 731601145 1.641 ns/op
https://github.com/golang/go/blob/go1.16.5/src/encoding/gob/encode.go#L113-L117

You can try musgo_int. All you need to do is to cast your variable:
package main
import (
"github.com/ymz-ncnk/musgo_int"
)
func main() {
var myInt int = 1234
// from int to []byte
buf := make([]byte, musgo_int.Int(myInt).SizeMUS())
musgo_int.Int(myInt).MarshalMUS(buf)
// from []byte to int
_, err := (*musgo_int.Int)(&myInt).UnmarshalMUS(buf)
if err != nil {
panic(err)
}
}

Convert Integer to byte slice.
import (
"bytes"
"encoding/binary"
"log"
)
func IntToBytes(num int64) []byte {
buff := new(bytes.Buffer)
bigOrLittleEndian := binary.BigEndian
err := binary.Write(buff, bigOrLittleEndian, num)
if err != nil {
log.Panic(err)
}
return buff.Bytes()
}

Maybe the simple way is using protobuf, see the Protocol Buffer Basics: Go
define message like
message MyData {
int32 id = 1;
}
get more in Defining your protocol format
// Write
out, err := proto.Marshal(mydata)
read more in Writing a Message

Try math/big package to convert bytes array to int and to convert int to bytes array.
package main
import (
"fmt"
"math/big"
)
func main() {
// Convert int to []byte
var int_to_encode int64 = 65535
var bytes_array []byte = big.NewInt(int_to_encode).Bytes()
fmt.Println("bytes array", bytes_array)
// Convert []byte to int
var decoded_int int64 = new(big.Int).SetBytes(bytes_array).Int64()
fmt.Println("decoded int", decoded_int)
}

This is the most straight forward (and shortest (and safest) (and maybe most performant)) way:
buf.Bytes() is of type bytes slice.
var val uint32 = 42
buf := new(bytes.Buffer)
err := binary.Write(buf, binary.LittleEndian, val)
if err != nil {
fmt.Println("binary.Write failed:", err)
}
fmt.Printf("% x\n", buf.Bytes())
see also https://stackoverflow.com/a/74819602/589493

What's wrong with converting it to a string?
[]byte(fmt.Sprintf("%d", myint))

Related

Golang number base conversion

I was wondering, how do you convert a base10 number from one base to another without usage of strconv in Golang ?
Could you please give me some advice ?
package main
import (
"fmt"
"math/big"
)
func main() {
fmt.Println(big.NewInt(1000000000000).Text(62))
}
Demo
Use the math package and a log identify:
log_77(x) = log(x) / log(77)
This is probably cheating but I guess you could look at the implementation of strconv.FormatInt, and build some of your own code using that as an example. That way you aren't using it directly, you have implemented it yourself.
You can use this function to convert any decimal number to any base with the character set of your choice.
func encode(nb uint64, buf *bytes.Buffer, base string) {
l := uint64(len(base))
if nb/l != 0 {
encode(nb/l, buf, base)
}
buf.WriteByte(base[nb%l])
}
func decode(enc, base string) uint64 {
var nb uint64
lbase := len(base)
le := len(enc)
for i := 0; i < le; i++ {
mult := 1
for j := 0; j < le-i-1; j++ {
mult *= lbase
}
nb += uint64(strings.IndexByte(base, enc[i]) * mult)
}
return nb
}
You can use it like that:
// encoding
var buf bytes.Buffer
encode(100, &buf, "0123456789abcdef")
fmt.Println(buf.String())
// 64
// decoding
val := decode("64", "0123456789abcdef")
fmt.Println(val)
// 100

Golang: Convert byte array to big.Int

I'm trying to create an RSA Public Key from a Modulus and Exponent stored in a byte array. After some experimentation I've got the following:
func bytes_to_int(b []byte) (acc uint64) {
length := len(b)
if length % 4 != 0 {
extra := (4 - length % 4)
b = append([]byte(strings.Repeat("\000", extra)), b...)
length += extra
}
var block uint32
for i := 0; i < length; i += 4 {
block = binary.BigEndian.Uint32(b[i:i+4])
acc = (acc << 32) + uint64(block)
}
return
}
func main() {
fmt.Println(bytes_to_int(data[:128]))
fmt.Println(bytes_to_int(data[128:]))
}
This appears to work (although I'm not convinced there isn't a better way). My next step was to convert it to use math/big in order to handle larger numbers. I can see an Lsh function to do the << but can't figure out how to recursively add the Uint32(block) to the big.Int.
For reference, the Public Key I'm attempting to import is a Mixmaster Key stored in a keyring (pubring.mix):
http://www.mixmin.net/draft-sassaman-mixmaster-XX.html#key-format
http://pinger.mixmin.net/pubring.mix
You want Int.SetBytes to make a big.int from a slice of []byte.
func (z *Int) SetBytes(buf []byte) *Int
SetBytes interprets buf as the bytes of a big-endian unsigned integer, sets z to that value, and returns z.
This should be quite straightforward to use in your application since your keys are in big-endian format according to the doc you linked.
import "math/big"
z := new(big.Int)
z.SetBytes(byteSliceHere)
Like Nick mentioned, you could use SetBytes, keep in mind the input is in base64 so you have to decode that first.
Example:
func Base64ToInt(s string) (*big.Int, error) {
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
return nil, err
}
i := new(big.Int)
i.SetBytes(data)
return i, nil
}

The binary representation of unsigned integer in Go

Is there a built-in function to convert a uint to a slice of binary integers {0,1} ?
>> convert_to_binary(2)
[1, 0]
I am not aware of such a function, however you can use strconv.FormatUint for that purpose.
Example (on play):
func Bits(i uint64) []byte {
bits := []byte{}
for _, b := range strconv.FormatUint(i, 2) {
bits = append(bits, byte(b - rune('0')))
}
return bits
}
FormatUint will return the string representation of the given uint to a base, in this case 2, so we're encoding it in binary. So the returned string for i=2 looks like this: "10". In bytes this is [49 48] as 1 is 49 and 0 is 48 in ASCII and Unicode. So we just need to iterate over the string, subtracting 48 from each rune (unicode character) and converting it to a byte.
Here is another method:
package main
import (
"bytes"
"fmt"
"math/bits"
)
func unsigned(x uint) []byte {
b := make([]byte, bits.UintSize)
for i := range b {
if bits.LeadingZeros(x) == 0 {
b[i] = 1
}
x = bits.RotateLeft(x, 1)
}
return b
}
func trimUnsigned(x uint) []byte {
return bytes.TrimLeft(unsigned(x), string(0))
}
func main() {
b := trimUnsigned(2)
fmt.Println(b) // [1 0]
}
https://golang.org/pkg/math/bits#LeadingZeros

Reading an integer from standard input

How do I use the fmt.Scanf function in Go to get an integer input from the standard input?
If this can't be done using fmt.Scanf, what's the best way to read a single integer?
http://golang.org/pkg/fmt/#Scanf
All the included libraries in Go are well documented.
That being said, I believe
func main() {
var i int
_, err := fmt.Scanf("%d", &i)
}
does the trick
An alternative that can be a bit more concise is to just use fmt.Scan:
package main
import "fmt"
func main() {
var i int
fmt.Scan(&i)
fmt.Println("read number", i, "from stdin")
}
This uses reflection on the type of the argument to discover how the input should be parsed.
http://golang.org/pkg/fmt/#Scan
Here is my "Fast IO" method for reading positive integers. It could be improved with bitshifts and laying out memory in advance.
package main
import (
"io/ioutil"
"bufio"
"os"
"strconv"
)
func main() {
out := bufio.NewWriter(os.Stdout)
ints := getInts()
var T int64
T, ints = ints[0], ints[1:]
..
out.WriteString(strconv.Itoa(my_num) + "\n")
out.Flush()
}
}
func getInts() []int64 {
//assumes POSITIVE INTEGERS. Check v for '-' if you have negative.
var buf []byte
buf, _ = ioutil.ReadAll(os.Stdin)
var ints []int64
num := int64(0)
found := false
for _, v := range buf {
if '0' <= v && v <= '9' {
num = 10*num + int64(v - '0') //could use bitshifting here.
found = true
} else if found {
ints = append(ints, num)
found = false
num = 0
}
}
if found {
ints = append(ints, num)
found = false
num = 0
}
return ints
}
Golang fmt.Scan is simpler than Golang fmt.Scanf (which is simpler than Clang scanf)
If fmt.Scan errors i.e. if not nil, log & return
1 Read single variable:
import (
"fmt"
"log"
)
var i int
if _, err := fmt.Scan(&i); err != nil {
log.Print(" Scan for i failed, due to ", err)
return
}
fmt.Println(i)
2 Read multiple variables:
import (
"fmt"
"log"
)
var i, j, k int
if _, err := fmt.Scan(&i, &j, &k); err != nil {
log.Print(" Scan for i, j & k failed, due to ", err)
return
}
fmt.Println(i, j, k)
Best of luck
Example from: http://www.sortedinf.com/?q=golang-in-1-hour
You can use fmt.Scanf with a format specifier. The format specifier for the integer is %d. So you can use standard input like below.
func main() {
var someVar int
fmt.Scanf("%d", &someVar)
}
or else you can use fmt.Scan or fmt.Scanln as below.
func main() {
var someVar int
fmt.Scanln(&someVar)
}
You could also use bufio.NewReader to read an integer from the standard input.
The below program:
Prompts for an integer input
Creates a bufio.Reader to read from standard input
Reads input till it encounters a newline character '\n' (Note that this will only read a single integer. Space separated values will not work)
Removes the newline character
Converts string to int
package main
import (
"fmt"
"bufio"
"os"
"strconv"
"strings"
)
func getInt() error {
fmt.Println("Enter an integer")
userInput := bufio.NewReader(os.Stdin)
userVal, err := userInput.ReadString('\n')
if err != nil {
return err
}
input := strings.TrimSpace(userVal)
intVal, err := strconv.Atoi(input)
if err != nil {
return err
}
fmt.Printf("You entered: %d\n", intVal)
return nil
}
func main() {
getInt()
}
Why can't we just use a scanf? just like we use in C? it's working though.
package main
import "fmt"
func main() {
var i int
fmt.Scanf("%d", &i)
fmt.Println(i)
}

Looking for Go equivalent of scanf

I'm looking for the Go equivalent of scanf().
I tried with following code:
1 package main
2
3 import (
4 "scanner"
5 "os"
6 "fmt"
7 )
8
9 func main() {
10 var s scanner.Scanner
11 s.Init(os.Stdin)
12 s.Mode = scanner.ScanInts
13 tok := s.Scan()
14 for tok != scanner.EOF {
15 fmt.Printf("%d ", tok)
16 tok = s.Scan()
17 }
18 fmt.Println()
19 }
I run it with input from a text with a line of integers.
But it always output -3 -3 ...
And how to scan a line composed of a string and some integers?
Changing the mode whenever encounter a new data type?
The Package documentation:
Package scanner
A general-purpose scanner for UTF-8
encoded text.
But it seems that the scanner is not for general use.
Updated code:
func main() {
n := scanf()
fmt.Println(n)
fmt.Println(len(n))
}
func scanf() []int {
nums := new(vector.IntVector)
reader := bufio.NewReader(os.Stdin)
str, err := reader.ReadString('\n')
for err != os.EOF {
fields := strings.Fields(str)
for _, f := range fields {
i, _ := strconv.Atoi(f)
nums.Push(i)
}
str, err = reader.ReadString('\n')
}
r := make([]int, nums.Len())
for i := 0; i < nums.Len(); i++ {
r[i] = nums.At(i)
}
return r
}
Improved version:
package main
import (
"bufio"
"os"
"io"
"fmt"
"strings"
"strconv"
"container/vector"
)
func main() {
n := fscanf(os.Stdin)
fmt.Println(len(n), n)
}
func fscanf(in io.Reader) []int {
var nums vector.IntVector
reader := bufio.NewReader(in)
str, err := reader.ReadString('\n')
for err != os.EOF {
fields := strings.Fields(str)
for _, f := range fields {
if i, err := strconv.Atoi(f); err == nil {
nums.Push(i)
}
}
str, err = reader.ReadString('\n')
}
return nums
}
Your updated code was much easier to compile without the line numbers, but it was missing the package and import statements.
Looking at your code, I noticed a few things. Here's my revised version of your code.
package main
import (
"bufio"
"fmt"
"io"
"os"
"strconv"
"strings"
"container/vector"
)
func main() {
n := scanf(os.Stdin)
fmt.Println()
fmt.Println(len(n), n)
}
func scanf(in io.Reader) []int {
var nums vector.IntVector
rd := bufio.NewReader(os.Stdin)
str, err := rd.ReadString('\n')
for err != os.EOF {
fields := strings.Fields(str)
for _, f := range fields {
if i, err := strconv.Atoi(f); err == nil {
nums.Push(i)
}
}
str, err = rd.ReadString('\n')
}
return nums
}
I might want to use any input file for scanf(), not just Stdin; scanf() takes an io.Reader as a parameter.
You wrote: nums := new(vector.IntVector), where type IntVector []int. This allocates an integer slice reference named nums and initializes it to zero, then the new() function allocates an integer slice reference and initializes it to zero, and then assigns it to nums. I wrote: var nums vector.IntVector, which avoids the redundancy by simply allocating an integer slice reference named nums and initializing it to zero.
You didn't check the err value for strconv.Atoi(), which meant invalid input was converted to a zero value; I skip it.
To copy from the vector to a new slice and return the slice, you wrote:
r := make([]int, nums.Len())
for i := 0; i < nums.Len(); i++ {
r[i] = nums.At(i)
}
return r
First, I simply replaced that with an equivalent, the IntVector.Data() method: return nums.Data(). Then, I took advantage of the fact that type IntVector []int and avoided the allocation and copy by replacing that by: return nums.
Although it can be used for other things, the scanner package is designed to scan Go program text. Ints (-123), Chars('c'), Strings("str"), etc. are Go language token types.
package main
import (
"fmt"
"os"
"scanner"
"strconv"
)
func main() {
var s scanner.Scanner
s.Init(os.Stdin)
s.Error = func(s *scanner.Scanner, msg string) { fmt.Println("scan error", msg) }
s.Mode = scanner.ScanInts | scanner.ScanStrings | scanner.ScanRawStrings
for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
txt := s.TokenText()
fmt.Print("token:", tok, "text:", txt)
switch tok {
case scanner.Int:
si, err := strconv.Atoi64(txt)
if err == nil {
fmt.Print(" integer: ", si)
}
case scanner.String, scanner.RawString:
fmt.Print(" string: ", txt)
default:
if tok >= 0 {
fmt.Print(" unicode: ", "rune = ", tok)
} else {
fmt.Print(" ERROR")
}
}
fmt.Println()
}
}
This example always reads in a line at a time and returns the entire line as a string. If you want to parse out specific values from it you could.
package main
import (
"fmt"
"bufio"
"os"
"strings"
)
func main() {
value := Input("Please enter a value: ")
trimmed := strings.TrimSpace(value)
fmt.Printf("Hello %s!\n", trimmed)
}
func Input(str string) string {
print(str)
reader := bufio.NewReader(os.Stdin)
input, _ := reader.ReadString('\n')
return input
}
In a comment to one of my answers, you said:
From the Language Specification: "When
memory is allocated to store a value,
either through a declaration or make()
or new() call, and no explicit
initialization is provided, the memory
is given a default initialization".
Then what's the point of new()?
If we run:
package main
import ("fmt")
func main() {
var i int
var j *int
fmt.Println("i (a value) = ", i, "; j (a pointer) = ", j)
j = new(int)
fmt.Println("i (a value) = ", i, "; j (a pointer) = ", j, "; *j (a value) = ", *j)
}
The declaration var i int allocates memory to store an integer value and initializes the value to zero. The declaration var j *int allocates memory to store a pointer to an integer value and initializes the pointer to zero (a nil pointer); no memory is allocated to store an integer value. We see program output similar to:
i (a value) = 0 ; j (a pointer) = <nil>
The built-in function new takes a type T and returns a value of type *T. The memory is initialized to zero values. The statement j = new(int) allocates memory to store an integer value and initializes the value to zero, then it stores a pointer to this integer value in j. We see program output similar to:
i (a value) = 0 ; j (a pointer) = 0x7fcf913a90f0 ; *j (a value) = 0
The latest release of Go (2010-05-27) has added two functions to the fmt package: Scan() and Scanln(). They don't take any pattern string. like in C, but checks the type of the arguments instead.
package main
import (
"fmt"
"os"
"container/vector"
)
func main() {
numbers := new(vector.IntVector)
var number int
n, err := fmt.Scan(os.Stdin, &number)
for n == 1 && err == nil {
numbers.Push(number)
n, err = fmt.Scan(os.Stdin, &number)
}
fmt.Printf("%v\n", numbers.Data())
}

Resources