Can't decode messages using goprotobuf - go

I have the following proto file:
package dogs;
enum Breed {
terrier = 0;
shepherd = 1;
hound = 2;
};
message Dog {
required int64 nbLegs = 1;
optional int64 nbTeeth = 2 [default=24];
optional Breed breed = 3;
optional string name = 4;
}
And the following Go program written using the goprotobuf package. The program
reads a Varint from stdin in order to get the length of the encoded message,
reads that number of bytes from stdin into a buffer, and
attempts to unmarshal the buffer into a Dog.
--START CODE--
package main
import "bufio"
import "encoding/binary"
import "os"
import "log"
import "fmt"
import "dogs"
import "code.google.com/p/goprotobuf/proto"
func render(dog *dogs.Dog) string {
return fmt.Sprintf("Dog: %v %v %v %v", dog.GetName(), dog.GetBreed(), dog.GetNbLegs(), dog.GetNbTeeth())
}
func main() {
var dog = new(dogs.Dog)
stdin := bufio.NewReader(os.Stdin)
sz, _ := binary.ReadVarint(stdin)
bytes := make([]byte, sz)
os.Stdin.Read(bytes)
buf := proto.NewBuffer(bytes)
err := buf.Unmarshal(dog)
if err != nil {
log.Fatal(err)
}
fmt.Fprintf(os.Stderr, "Receiving %s of length %d\n", render(dog), sz)
}
--END CODE--
And the this encoded message (preceded by a Varint) (in hexdump form)
0000000: 1408 0418 0222 0446 6964 6f .....".Fido
which i can successfully decode using c++ and libprotobuf as a Dog with
name: Fido
breed: hound
nbLegs: 4
nbTeeth: 24
Unmarshaling in the go program, however, always returns the error "illegal tag 0".
I suspect that I've misunderstood the decoding API but, so far, I can't see how.

Try just umarshalling the bytes instead of creating a buffer like this:
You may also want to check to see if os.Stdin.Read(bytes) is actually reading sz.
func main() {
var dog = new(dogs.Dog)
stdin := bufio.NewReader(os.Stdin)
sz, _ := binary.ReadVarint(stdin)
bytes := make([]byte, sz)
stdin.Read(bytes)
err := buf.Unmarshal(dog, bytes)
if err != nil {
log.Fatal(err)
}
fmt.Fprintf(os.Stderr, "Receiving %s of length %d\n", render(dog), sz)
}

Related

Find byte offset of a pattern in Golang

We can find the byte offset of a pattern from file by
"grep -ob pattern filename";
However, grep is not utf8 safe.
How do I find byte offset of a pattern in Go? The file is process log, which can be in TB.
This is what I want to get in Go:
$ cat fname
hello world
findme
hello 世界
findme again
...
$ grep -ob findme fname
12:findme
32:findme
FindAllStringIndex(s string, n int) returns byte start/finish indexes (i.e., slices) of all successive matches of the expression:
package main
import "fmt"
import "io/ioutil"
import "regexp"
func main() {
fname := "C:\\Users\\UserName\\go\\src\\so56798431\\fname"
b, err := ioutil.ReadFile(fname)
if err != nil {
panic(err)
}
re, err := regexp.Compile("findme")
if err != nil {
// handle error
}
fmt.Println(re.FindAllStringIndex(string(b), -1))
}
Output:
[[12 18] [32 38]]
Note: I did this on Microsoft Windows, but saved the file in UNIX format (linefeed); if input file saved in Windows format (carriage return & linefeed) the byte offsets would increment to 13 and 35, respectively.
UPDATE: for large files, use bufio.Scanner; for example:
package main
import (
"bufio"
"fmt"
"log"
"os"
"regexp"
)
func main() {
fname, err := os.Open("C:\\Users\\UserName\\go\\src\\so56798431\\fname")
if err != nil {
log.Fatal(err)
}
defer fname.Close()
re, err := regexp.Compile("findme")
if err != nil {
// handle error
}
scanner := bufio.NewScanner(fname)
bytesRead := 0
for scanner.Scan() {
b := scanner.Text()
//fmt.Println(b)
results := re.FindAllStringIndex(b, -1)
for _, result := range results {
fmt.Println(bytesRead + result[0])
}
// account for UNIX EOL marker
bytesRead += len(b) + 1
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
}
Output:
12
32

Go Lang Scan doesent scan for next line

This scanner dosent scan for the next line. I will explain it in more detail when you see results...
package main
import (
"fmt"
"io/ioutil"
"os"
"strings"
)
func main() {
var inputFileName string
var write string
fmt.Scanln(&inputFileName)
//func Join(a []string, sep string) string
s := []string{inputFileName, ".txt"}
inputFileName = strings.Join(s, "")
creator, err := os.Create(inputFileName)
check(err)
/*
*Writing
*/
fmt.Printf("The file name with %s what do you want to write?", inputFileName)
fmt.Scanln(&write)
if len(write) <= 0 {
panic("Cant be empty")
}
byteStringWrite := []byte(write)
//func (f *File) Write(b []byte) (n int, err error)
fmt.Println("BYTE : ", byteStringWrite)
fmt.Println("NONBYTE : ", write)
_, errWriter := creator.Write(byteStringWrite)
check(errWriter)
/**
*Reading File
*/
read, errRead := ioutil.ReadFile(inputFileName)
check(errRead)
readString := string(read)
fmt.Println("*******************FILE*********************")
fmt.Println(readString)
}
func check(e error) {
if e != nil {
panic(e)
}
}
Results:
Sample.txt //My User Input
The file name with Sample.txt what do you want to write?Hello World
BYTE : [72 101 108 108 111]
NONBYTE : Hello
*******************FILE*********************
Hello
So Here you can see it dosent look for the space. Meaning after the space it automatically quits. Can someone help me figure out this problem? Thankyou.
EDIT
Using bufio.ReadString();
package main
import (
"fmt"
"io/ioutil"
"os"
"strings"
"bufio"
)
func main() {
var inputFileName string
var write string
bio := bufio.NewReader(os.Stdin)
inputFileName, err := bio.ReadString('\n')
fmt.Println(inputFileName)
//func Join(a []string, sep string) string
s := []string{inputFileName, ".txt"}
inputFileName = strings.Join(s, "")
creator, err := os.Create(inputFileName)
check(err)
/*
*Writing
*/
fmt.Printf("The file name with %s what do you want to write?", inputFileName)
fmt.Scanln(&write)
if len(write) <= 0 {
panic("Cant be empty")
}
byteStringWrite := []byte(write)
//func (f *File) Write(b []byte) (n int, err error)
fmt.Println("BYTE : ", byteStringWrite)
fmt.Println("NONBYTE : ", write)
_, errWriter := creator.Write(byteStringWrite)
check(errWriter)
/**
*Reading File
*/
read, errRead := ioutil.ReadFile(inputFileName)
check(errRead)
readString := string(read)
fmt.Println("*******************FILE*********************")
fmt.Println(readString)
}
func check(e error) {
if e != nil {
panic(e)
}
}
Results:
amanuel2:~/workspace/pkg_os/07_Practice $ go run main.go
Sample
The file name with Sample
.txt what do you want to write?Something Else
BYTE : [83 111 109 101 116 104 105 110 103]
NONBYTE : Something
*******************FILE*********************
Something
Gives me correct .txt .. But same issue as above, it dosent take spaces
This is exactly what fmt.Scanln is supposed to do:
Scan scans text read from standard input, storing successive
space-separated values into successive arguments. Newlines count as
space. It returns the number of items successfully scanned. If that is
less than the number of arguments, err will report why.
If you want to read a line of text use bufio.Reader:
bio := bufio.NewReader(os.Stdin)
// in case you want a string which doesn't contain the newline
line, hasMoreInLine, err := bio.ReadLine()
s := string(line)
fmt.Println(s)
// in case you need a string which contains the newline
s, err := bio.ReadString('\n')
fmt.Println(s)

How can I generate a random int using the "crypto/rand" package?

Say I would like to generate a secure random int between 0 and 27 using:
func Int(rand io.Reader, max *big.Int) (n *big.Int, err error)
in the "crypto/rand" package.
How would I do that?
I do not really understand how this works, why does it not return one of the built in Go ints instead of pointer to some big.Int type?
EDIT:
Would this be considered secure enough for tokens?
func getToken(length int) string {
token := ""
codeAlphabet := "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
codeAlphabet += "abcdefghijklmnopqrstuvwxyz"
codeAlphabet += "0123456789"
for i := 0; i < length; i++ {
token += string(codeAlphabet[cryptoRandSecure(int64(len(codeAlphabet)))])
}
return token
}
func cryptoRandSecure(max int64) int64 {
nBig, err := rand.Int(rand.Reader, big.NewInt(max))
if err != nil {
log.Println(err)
}
return nBig.Int64()
}
func main() {
fmt.Println(getToken(32))
}
This would output something like this:
qZDbuPwNQGrgVmZCU9A7FUWbp8eIfn0Z
EwZVoQ5D5SEfdhiRsDfH6dU6tAovILCZ
cOqzODVP0GwbiNBwtmqLA78rFgV9d3VT
Here is some working code :
package main
import (
"fmt"
"crypto/rand"
"math/big"
)
func main() {
nBig, err := rand.Int(rand.Reader, big.NewInt(27))
if err != nil {
panic(err)
}
n := nBig.Int64()
fmt.Printf("Here is a random %T in [0,27) : %d\n", n, n)
}
But to generate a random token, I'd do something like this :
package main
import (
"crypto/rand"
"encoding/base32"
"fmt"
)
func main() {
token := getToken(10)
fmt.Println("Here is a random token : ", token)
}
func getToken(length int) string {
randomBytes := make([]byte, 32)
_, err := rand.Read(randomBytes)
if err != nil {
panic(err)
}
return base32.StdEncoding.EncodeToString(randomBytes)[:length]
}
If you're generating secure tokens for session IDs, OAuth Bearer tokens, CSRF or similar: you want to generate a token of (ideally) 256 bits (32 bytes) or no less than 192 bits (24 bytes).
A token with values between (0-27) can be brute-forced in less than a second and could not be considered secure.
e.g.
package main
import (
"crypto/rand"
"encoding/base64"
)
// GenerateRandomBytes returns securely generated random bytes.
// It will return an error if the system's secure random
// number generator fails to function correctly, in which
// case the caller should not continue.
func GenerateRandomBytes(n int) ([]byte, error) {
b := make([]byte, n)
_, err := rand.Read(b)
// Note that err == nil only if we read len(b) bytes.
if err != nil {
return nil, err
}
return b, nil
}
// GenerateRandomString returns a URL-safe, base64 encoded
// securely generated random string.
func GenerateRandomString(s int) (string, error) {
b, err := GenerateRandomBytes(s)
return base64.URLEncoding.EncodeToString(b), err
}
func main() {
// Example: this will give us a 44 byte, base64 encoded output
token, err := GenerateRandomString(32)
if err != nil {
// Serve an appropriately vague error to the
// user, but log the details internally.
}
}
The base64 output is safe for headers, HTTP forms, JSON bodies, etc.
If you need an integer it may help to explain your use-case, as it would be odd for a system to require tokens as ints.
If you only need a small number (i.e. [0, 255]), you could just read a byte out of the package's Reader:
b := []byte{0}
if _, err := rand.Reader.Read(b); err != nil {
panic(err)
}
n := b[0]
fmt.Println(n)
Playground: http://play.golang.org/p/4VO52LiEVh (the example won't work there, I don't know if it's working as intended or it's a playground bug).

How to read a file, abort with error if it's not valid UTF-8?

In Go, I want to read in a file line by line, into str's or []rune's.
The file should be encoded in UTF-8, but my program shouldn't trust it. If it contains invalid UTF-8, I want to properly handle the error.
There is bytes.Runes(s []byte) []rune, but that has no error return value. Will it panic on encountering invalid UTF-8?
For example,
package main
import (
"bufio"
"fmt"
"io/ioutil"
"os"
"strings"
"unicode/utf8"
)
func main() {
tFile := "text.txt"
t := []byte{'\xFF', '\n'}
ioutil.WriteFile(tFile, t, 0666)
f, err := os.Open(tFile)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer f.Close()
r := bufio.NewReader(f)
s, err := r.ReadString('\n')
if err != nil {
fmt.Println(err)
os.Exit(1)
}
s = strings.TrimRight(s, "\n")
fmt.Println(t, s, []byte(s))
if !utf8.ValidString(s) {
fmt.Println("!utf8.ValidString")
}
}
Output:
[255 10] � [255]
!utf8.ValidString
For example:
import (
"io/ioutil"
"log"
"unicode/utf8"
)
// ...
buf, err := ioutil.ReadAll(fname)
if error != nil {
log.Fatal(err)
}
size := 0
for start := 0; start < len(buf); start += size {
var r rune
if r, size = utf8.DecodeRune(buf[start:]); r == utf8.RuneError {
log.Fatalf("invalid utf8 encoding at ofs %d", start)
}
}
utf8.DecodeRune godocs:
DecodeRune unpacks the first UTF-8 encoding in p and returns the rune
and its width in bytes. If the encoding is invalid, it returns
(RuneError, 1), an impossible result for correct UTF-8.

Go io string to int Atoi invalid argument

I'm trying to parse a string from WebSockets connection in Go language. I'm implementing both sides of the connection, so the specification of data format is depending only on me.
As this is a simple app (generally for learning purposes), I've come up with ActionId Data, where ActionId is a uint8. BackendHandler is a handler for every request in WebSocket Connection.
Platform information
kuba:~$ echo {$GOARCH,$GOOS,`6g -V`}
amd64 linux 6g version release.r60.3 9516
code:
const ( // Specifies ActionId's
SabPause = iota
)
func BackendHandler(ws *websocket.Conn) {
buf := make([]byte, 512)
_, err := ws.Read(buf)
if err != nil { panic(err.String()) }
str := string(buf)
tmp, _ := strconv.Atoi(str[:0])
data := str[2:]
fmt.Println(tmp, data)
switch tmp {
case SabPause:
// Here I get `parsing "2": invalid argument`
// when passing "0 2" to websocket connection
minutes, ok := strconv.Atoui(data)
if ok != nil {
panic(ok.String())
}
PauseSab(uint8(minutes))
default:
panic("Unmatched input for BackendHandler")
}
}
All the output: (note the Println that I used for inspecting)
0 2
panic: parsing "2": invalid argument [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
I couldn't find the code from which this error is launch, only where the error code is defined (dependent on platform). I'd appreciate general ideas for improving my code, but mainly I just want to solve the conversion problem.
Is this related to my buffer -> string conversion and slice-manipulation(I didn't want to use SplitAfter methods)?
Edit
This code reproduces the problem:
package main
import (
"strconv"
"io/ioutil"
)
func main() {
buf , _ := ioutil.ReadFile("input")
str := string(buf)
_, ok := strconv.Atoui(str[2:])
if ok != nil {
panic(ok.String())
}
}
The file input has to contain 0 2\r\n (depending on the file ending, it may look different on other OSes). This code can be fixed by adding the ending index for reslice, this way:
_, ok := strconv.Atoui(str[2:3])
You didn't provide a small compilable and runnable program to illustrate your problem. Nor did you provide full and meaningful print diagnostic messages.
My best guess is that you have a C-style null-terminated string. For example, simplifying your code,
package main
import (
"fmt"
"strconv"
)
func main() {
buf := make([]byte, 512)
buf = []byte("0 2\x00") // test data
str := string(buf)
tmp, err := strconv.Atoi(str[:0])
if err != nil {
fmt.Println(err)
}
data := str[2:]
fmt.Println("tmp:", tmp)
fmt.Println("str:", len(str), ";", str, ";", []byte(str))
fmt.Println("data", len(data), ";", data, ";", []byte(data))
// Here I get `parsing "2": invalid argument`
// when passing "0 2" to websocket connection
minutes, ok := strconv.Atoui(data)
if ok != nil {
panic(ok.String())
}
_ = minutes
}
Output:
parsing "": invalid argument
tmp: 0
str: 4 ; 0 2 ; [48 32 50 0]
data 2 ; 2 ; [50 0]
panic: parsing "2": invalid argument
runtime.panic+0xac /home/peter/gor/src/pkg/runtime/proc.c:1254
runtime.panic(0x4492c0, 0xf840002460)
main.main+0x603 /home/peter/gopath/src/so/temp.go:24
main.main()
runtime.mainstart+0xf /home/peter/gor/src/pkg/runtime/amd64/asm.s:78
runtime.mainstart()
runtime.goexit /home/peter/gor/src/pkg/runtime/proc.c:246
runtime.goexit()
----- goroutine created by -----
_rt0_amd64+0xc9 /home/peter/gor/src/pkg/runtime/amd64/asm.s:65
If you add my print diagnostic statements to your code, what do you see?
Note that your tmp, _ := strconv.Atoi(str[:0]) statement is probably wrong, since str[:0] is equivalent to str[0:0], which is equivalent to the empty string "".
I suspect that your problem is that you are ignoring the n return value from ws.Read. For example (including diagnostic messages), I would expect,
buf := make([]byte, 512)
buf = buf[:cap(buf)]
n, err := ws.Read(buf)
if err != nil {
panic(err.String())
}
fmt.Println(len(buf), n)
buf = buf[:n]
fmt.Println(len(buf), n)
Also, try using this code to set tmp,
tmp, err := strconv.Atoi(str[:1])
if err != nil {
panic(err.String())
}

Resources