I'm trying to use go's net/rpc package to send data structures. The data structure includes a pointer to uint64. The pointer is never nil, but the value may be 0. I'm finding that when the value is 0, the receiver sees a nil pointer. When the value is non-0, the receives sees a non-nil pointer that points to a proper value. This is problematic, because it means that the RPC is breaking an invariant of my data structure: the pointer will never be nil.
I have a go playground that demonstrates this behavior here: https://play.golang.org/p/Un3bTe5F-P
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type P struct {
Zero, One int
Ptr *int
}
func main() {
// Initialize the encoder and decoder. Normally enc and dec would be
// bound to network connections and the encoder and decoder would
// run in different processes.
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
// Encode (send) the value.
var p P
p.Zero = 0
p.One = 1
p.Ptr = &p.Zero
fmt.Printf("p0: %s\n", p)
err := enc.Encode(p)
if err != nil {
log.Fatal("encode error:", err)
}
// Decode (receive) the value.
var q P
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("q0: %s\n", q)
p.Ptr = &p.One
fmt.Printf("p1: %s\n", p)
err = enc.Encode(p)
if err != nil {
log.Fatal("encode error:", err)
}
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("q1: %s\n", q)
}
The output from this code is:
p0: {%!s(int=0) %!s(int=1) %!s(*int=0x1050a780)}
q0: {%!s(int=0) %!s(int=1) %!s(*int=<nil>)}
p1: {%!s(int=0) %!s(int=1) %!s(*int=0x1050a784)}
q1: {%!s(int=0) %!s(int=1) %!s(*int=0x1050aba8)}
So when Ptr points to a 0, it becomes nil on the receiver side. When Ptr points to 1, it is passed through normally.
Is this a bug? Is there a way around this problem? I want to avoid having to unmarshall my detastructure on the receiver side to fix all the unexpected nil pointers...
This behaviour is a limitation of the gob protocol according the defect raised back in 2013 - see https://github.com/golang/go/issues/4609
Bear in mind that gob doesn't send pointers, the pointer is dereferenced and the value is passed. As such when the p.Ptr is set to &p.One, you'll find that q.Ptr != &q.One
Related
The first TCP connection running on localhost on osx always parses the binary sent to it correctly. Subsequent requests lose the binary data, only seeing the first byte [8]. How have I failed to set up my Reader?
package main
import (
"fmt"
"log"
"net"
"os"
"app/src/internal/handler"
"github.com/golang-collections/collections/stack"
)
func main() {
port := os.Getenv("SERVER_PORT")
s := stack.New()
ln, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatalf("net.Listen: %v", err)
}
fmt.Println("Serving on " + port)
for {
conn, err := ln.Accept()
// defer conn.Close()
if err != nil {
log.Fatal("ln.Accept")
}
go handler.Handle(conn, s)
}
}
package handler
import (
"fmt"
"io"
"log"
"net"
"github.com/golang-collections/collections/stack"
)
func Handle(c net.Conn, s *stack.Stack) {
fmt.Printf("Serving %s\n", c.RemoteAddr().String())
buf := make([]byte, 0, 256)
tmp := make([]byte, 128)
n, err := c.Read(tmp)
if err != nil {
if err != io.EOF {
log.Fatalf("connection Read() %v", err)
}
return
}
buf = append(buf, tmp[:n]...)
}
log:
Serving [::1]:51699
------------- value ---------------:QCXhoy5t
Buffer Length: 9. First Value: 8
Serving [::1]:51700
------------- value ---------------:
Buffer Length: 1. First Value: 8
Serving [::1]:51701
test sent over:
push random string:
QCXhoy5t
push random string:
GPh0EnbS
push random string:
4kJ0wN0R
The docs for Reader say:
Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n
<= len(p)) and any error encountered. Even if Read returns n < len(p), it may use
all of p as scratch space during the call. If some data is available but not
len(p) bytes, Read conventionally returns what is available instead of waiting
for more.
So the most likely cause of your issue is that Read is returning the data available (in this case a single character). You can fix this by using ioutil.ReadAll or performing the read in a loop (the fact the data is being added to a buffer makes it look like that was the original intention) with something like:
for {
n, err := c.Read(tmp)
if err != nil {
if err != io.EOF {
// Note that data might have also been received - you should process that
// if appropriate.
log.Fatalf("connection Read() %v", err)
return
}
break // All data received so process it
}
buf = append(buf, tmp[:n]...)
}
Note: There is no guarantee that any data is received; you should check the length before trying to access it (i.e. buf[0] may panic)
I'm building some server/client application in Go (the language is new to me). I searched a lot and read a whole bunch of different examples but there is still one thing I can't find. Lets say I have a single server client up and running. The client will send some kind of a message to the server and vice versa. Encoding and decoding is done by the package gob.
This example is not my application, it is only a quick example:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type Message struct {
Sender string
Receiver string
Command uint8
Value int64
}
func (message *Message) Set(sender string, receiver string, command uint8, value int64) *Message {
message.Sender = sender
message.Receiver = receiver
message.Command = command
message.Value = value
return message
}
func main() {
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
message := new(Message).Set("first", "second", 10, -1)
err := enc.Encode(*message) // send message
if err != nil {
log.Fatal("encode error:", err)
}
var m Message
err = dec.Decode(&m) // receice message
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q %q %d %d\n", m.Sender, m.Receiver, m.Command, m.Value)
}
This works fine, but I want the server to block until a new message is received so I can put the receiving process inside a infinite for loop inside a goroutine.
Something like that:
for {
// The server blocks HERE until a message from the client is received
fmt.Println("Received message:")
// Decode the new message
var m Message
err = dec.Decode(&m) // receice message
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q %q %d %d\n", m.Sender, m.Receiver, m.Command, m.Value)
}
The gob decoder blocks until it has read a full message or there's an error. The read loop in the question works as is.
working example on the playground
add a length header to the raw tcp stream.
that means, send a 4-bytes-length-header information to server before send the real load. and in server side read 4 bytes, allocate buffer, full read total message, and finally decode.
assume you have a tcp connection conn, in server side we could have:
func getInt(v []byte) int {
var r uint
r = 0
r |= uint(v[0]) << 24
r |= uint(v[1]) << 16
r |= uint(v[2]) << 8
r |= uint(v[3]) << 0
return int(r)
}
buf := make([]byte, 4)
_, err := io.ReadFull(conn, buf)
if err != nil {
return
}
length := getInt(buf)
buf = make([]byte, length)
_, err = io.ReadFull(conn, buf)
if err != nil {
return
}
//do gob decode from `buf` here
you may know client side refer the the server side source I think.
I am trying to read a buffered stream of signed 16 bit integers (wav format), but the bufio.Read method only accepts an array of bytes. My question is a 2-parter:
Can I preformat the byte stream into a buffered int16 array?
If I can't, whats the best way of post-processing the byte array into int16 array? My initial thought is to use tmp arrays and keep pushing/processing them, but I was curious if there was a more idiomatic way of doing this?
package main
import (
"bufio"
"io"
"log"
"os/exec"
)
func main() {
app := "someapp"
cmd := exec.Command(app)
stdout, err := cmd.StdoutPipe()
r := bufio.NewReader(stdout)
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
//"someapp" outputs signed 16bit integers (little endian))
buf := make([]byte, 0, 4*1024)
for {
n, err := r.Read(buf[:cap(buf)]) //r.Read only accepts type []byte
buf = buf[:n]
if n == 0 {
if err == nil {
continue
}
if err == io.EOF {
break
}
log.Fatal(err)
}
log.Printf("%x\n", buf)
//process buf here
if err != nil && err != io.EOF {
log.Fatal(err)
}
}
}
When working with IO, you always work with []bytes, there's no way to substitute that with []int16, or pre-format that as int16s, it's always a stream of bytes.
You can look at the encoding/binary package to decode this stream.
// to get the first uint16 as i
i := binary.LittleEndian.Uint16(buf[:2])
You can then iterate through the buf as needed.
You can also use binary.Read to read directly from the io.Reader.
var i uint16
for {
err := binary.Read(r, binary.LittleEndian, &i)
if err != nil {
log.Println(err)
break
}
fmt.Println(i)
}
It may worth noting the simplicity of what needs to be done. Each uint16 is created via:
func (littleEndian) Uint16(b []byte) uint16 {
return uint16(b[0]) | uint16(b[1])<<8
}
You can use encoding/binary.Read to fill an []int16 directly from your reader, although technically the answer to your first question is still no (check the source of binary.Read, it reads the data to a []byte first).
Since golang is a systems language, it should be capable of doing some user space hardware driver job then.
I want to mmap "/dev/mem" and do some read or writes to some memory mapped pci registers.
Since golang's syscall.Mmap return a byte array. How could I possibly do a 32-bit(for example) read or writes to the registers?
byte by byte access is not approperiate since some registers only support 32 bit access.
You can do this by using unsafe and getting a *uint32 pointer into the block at the right offset. Here's an example for a regular byte array, but an mmap'ed one should work the same.
package main
import (
"fmt"
"unsafe"
)
func main() {
a := make([]byte, 30)
p := (*uint32)(unsafe.Pointer(&a[8]))
*p = 0xabcd0123
fmt.Println(a)
}
Here is an example. You are welcome. ;)
func Readu32(baseAddress int64, offset int64) uint32 {
var value uint32 = 0xFFFFFFFF
const bufferSize int = 4096
file, err := os.Open("/dev/mem")
if err != nil {
log.Fatal(err)
}
defer file.Close()
mmap, err := syscall.Mmap(int(file.Fd()), baseAddress, bufferSize, syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil {
log.Fatal(err)
}
value = *(*uint32)(unsafe.Pointer(&mmap[offset]))
err = syscall.Munmap(mmap)
if err != nil {
log.Fatal(err)
}
return value
}
I'm trying to write functions that will allow me to marshal/unmarshal simple structs into byte arrays. I've succeeded in writing Marshal, with help from the kind folks at #go-nuts, but I'm running into trouble writing Unmarshal.
// Unmarshal unpacks the binary data and stores it in the packet using
// reflection.
func Unmarshal(b []byte, t reflect.Type) (pkt interface{}, err error) {
buf := bytes.NewBuffer(b)
p := reflect.New(t)
v := reflect.ValueOf(p)
for i := 0; i < t.NumField(); i++ {
f := v.Field(i)
switch f.Kind() {
case reflect.String:
// length of string
var l int16
var e error
e = binary.Read(buf, binary.BigEndian, &l)
if e != nil {
err = e
return
}
// read length-of-string bytes from the buffer
raw := make([]byte, l)
_, e = buf.Read(raw)
if e != nil {
err = e
return
}
// convert the bytes to a string
f.SetString(bytes.NewBuffer(raw).String())
default:
e := binary.Read(buf, binary.BigEndian, f.Addr())
if e != nil {
err = e
return
}
}
}
pkt = p
return
}
The problem with the code above is that the call to f.Addr() near the end is apparently trying to get the address of an unaddressable value.
If there is an alternative solution, I would appreciate that as well. Either way, any help would be much appreciated.
Thanks!
I think you should use
v := p.Elem() // Get the value that 'p' points to
instead of
v := reflect.ValueOf(p)
Working example with lots of assumptions and a trivial data format:
package main
import (
"fmt"
"reflect"
"strconv"
)
// example marshalled format. lets say that marshalled data will have
// four bytes of a formatted floating point number followed by two more
// printable bytes.
type m42 []byte
// example struct we'd like to unmarshal into.
type packet struct {
S string // exported fields required for reflection
F float64
}
// example usage
func main() {
var p packet
if err := Unmarshal(m42("3.14Pi"), &p); err == nil {
fmt.Println(p)
} else {
fmt.Println(err)
}
}
func Unmarshal(data m42, structPtr interface{}) error {
vp := reflect.ValueOf(structPtr)
ve := vp.Elem() // settable struct Value
vt := ve.Type() // type info for struct
nStructFields := ve.NumField()
for i := 0; i < nStructFields; i++ {
fv := ve.Field(i) // settable field Value
sf := vt.Field(i) // StructField type information
// struct field name indicates which m42 field to unmarshal.
switch sf.Name {
case "S":
fv.SetString(string(data[4:6]))
case "F":
s := string(data[0:4])
if n, err := strconv.ParseFloat(s, 64); err == nil {
fv.SetFloat(n)
} else {
return err
}
}
}
return nil
}
Appropriate alternative solutions would depend heavily on the real data you need to support.
I'm going to bet that the reason f.Addr() has the problem because it actually isn't addressable.
the reflect package Type object has a method that will tell you if the type is addressable called CanAddr(). Assuming the field is addressable if it's not a string is not always true. If the struct is not passed in as a pointer to a struct then it's fields won't be addressable. For more details about what is and isn't addressable see: http://weekly.golang.org/pkg/reflect/#Value.CanAddr which outlines the correct rules.
Essentially for your code to work I think you need to ensure you always call it with a pointer to a struct.