I'm trying to have some Go object implement io.Writer, but writes to a string instead of a file or file-like object. I thought bytes.Buffer would work since it implements Write(p []byte). However when I try this:
import "bufio"
import "bytes"
func main() {
var b bytes.Buffer
foo := bufio.NewWriter(b)
}
I get the following error:
cannot use b (type bytes.Buffer) as type io.Writer in function argument:
bytes.Buffer does not implement io.Writer (Write method has pointer receiver)
I am confused, since it clearly implements the interface. How do I resolve this error?
Pass a pointer to the buffer, instead of the buffer itself:
import "bufio"
import "bytes"
func main() {
var b bytes.Buffer
foo := bufio.NewWriter(&b)
}
package main
import "bytes"
import "io"
func main() {
var b bytes.Buffer
_ = io.Writer(&b)
}
You don't need use "bufio.NewWriter(&b)" to create an io.Writer. &b is an io.Writer itself.
Just use
foo := bufio.NewWriter(&b)
Because the way bytes.Buffer implements io.Writer is
func (b *Buffer) Write(p []byte) (n int, err error) {
...
}
// io.Writer definition
type Writer interface {
Write(p []byte) (n int, err error)
}
It's b *Buffer, not b Buffer. (I also think it is weird for we can call a method by a variable or its pointer, but we can't assign a pointer to a non-pointer type variable.)
Besides, the compiler prompt is not clear enough:
bytes.Buffer does not implement io.Writer (Write method has pointer receiver)
Some ideas, Go use Passed by value, if we pass b to buffio.NewWriter(), in NewWriter(), it is a new b (a new buffer), not the original buffer we defined, therefore we need pass the address &b.
bytes.Buffer is defined as:
type Buffer struct {
buf []byte // contents are the bytes buf[off : len(buf)]
off int // read at &buf[off], write at &buf[len(buf)]
bootstrap [64]byte // memory to hold first slice; helps small buffers avoid allocation.
lastRead readOp // last read operation, so that Unread* can work correctly.
}
using passed by value, the passed new buffer struct is different from the origin buffer variable.
Related
I have a question about a usage of pointer in Go. The link is here: https://golang.org/pkg/bytes/#example_Buffer.
In the type Buffer section, the first example:
type Buffer struct {
// contains filtered or unexported fields
}
func main() {
var b bytes.Buffer // A Buffer needs no initialization.
b.Write([]byte("Hello "))
fmt.Fprintf(&b, "world!")
b.WriteTo(os.Stdout)
}
and then in the
func (b *Buffer) Write(p []byte) (n int, err error)
I know that the receiver of func Write is (b *Buffer) then why in the main() function, after declaring/initializing b, we can simply use b.Write() but not (&b).Write()?
Thank you!
The receiver is a pointer, and in b.Write(), b is addressable. So Write is invoked on a pointer to b, not a copy of b. If b was not addressable, then you'd have received a compile error. For instance, this would fail:
bytes.Buffer{}.Write([]byte{1})
In general: you can call methods with pointer receivers only if you can take the address of the receiver object. The compiler passes the reference, not the copy for such methods.
The below golang(go1.10.2) code will give an unexpected output
package main
import (
"bytes"
"fmt"
)
func main() {
var b bytes.Buffer
//Commenting the below line will fix the problem
b.WriteString("aas-")
fmt.Printf("Before Calling - \"%s\"\n", b.String())
b = makeMeMad(b)
fmt.Printf("FinalValue - \"%s\"\n", b.String())
}
func makeMeMad(b bytes.Buffer) bytes.Buffer {
b.WriteString("xcxxcx asdasdas dasdsd asdasdasdasd")
fmt.Printf("Write More - \"%s\"\n", b.String())
/*
//This will fix the problem
var newBuffer bytes.Buffer
newBuffer.WriteString(b.String())
return newBuffer
*/
return b
}
Output
Before Calling - "aas-"
Write More - "aas-xcxxcx asdasdas dasdsd asdasdasdasd"
FinalValue - "aas- "
I was expecting "aas-xcxxcx asdasdas dasdsd asdasdasdasd" in the last line of output. Could anyone please explain.
Under the hood bytes.Buffer contain among other unexported fields bootstrap array and buf slice. While buffer content is small slice points to internal array to avoid allocation. When you pass bytes.Buffer argument as value, function receives a copy. Slice is reference type, so this copy's slice continue to point on the original buffer's array. When you write to this copy's slice you actually write to original's bootstrap array, copy's array stay unchanged("aas-" in our case). Then you return this copy and you can print it. But when you assign it to variable containing original, bootstrap array first assigned("aas-") and then buf slice pointed on it.
Bootstrap array is [64]byte, so you can use long string literals >64 in you code snippet and see things works as expected when buffer allocate buf slice.
Also here simplified example trying to show why all this looks so contrintuitive.
It is mentioned in Golang FAQ section as:
If an interface value contains a pointer *T, a method call can obtain
a value by dereferencing the pointer, but if an interface value
contains a value T, there is no useful way for a method call to obtain
a pointer.
Even in cases where the compiler could take the address of a value to
pass to the method, if the method modifies the value the changes will
be lost in the caller. As an example, if the Write method of
bytes.Buffer used a value receiver rather than a pointer, this code:
var buf bytes.Buffer
io.Copy(buf, os.Stdin)
would copy standard input into a copy of buf, not into buf itself
The error is because you're not passing the address of buffer inside makeMeMad function. That's why it has not override the original buffer inside main function.
Pass address to the created buffer to append string to the existing buffer value.
package main
import (
"bytes"
"fmt"
)
func main() {
var b bytes.Buffer
//Commenting the below line will fix the problem
b.WriteString("aas-")
fmt.Printf("Before Calling - \"%s\"\n", b.String())
makeMeMad(&b)
fmt.Printf("FinalValue - \"%s\"\n", b.String())
}
func makeMeMad(b *bytes.Buffer) {
b.WriteString("xcxxcx asdasdas dasdsd asdasdasdasd")
fmt.Printf("Write More - \"%s\"\n", b.String())
/*
//This will fix the problem
var newBuffer bytes.Buffer
newBuffer.WriteString(b.String())
return newBuffer
*/
}
Playground Example
Or you can assign the returned buffer value to a new variable and you will get the updated buffer value.
package main
import (
"bytes"
"fmt"
)
func main() {
var b bytes.Buffer
//Commenting the below line will fix the problem
b.WriteString("aas-")
fmt.Printf("Before Calling - \"%s\"\n", b.String())
ab := makeMeMad(b)
fmt.Printf("FinalValue - \"%s\"\n", ab.String())
}
func makeMeMad(b bytes.Buffer) bytes.Buffer {
b.WriteString("xcxxcx asdasdas dasdsd asdasdasdasd")
fmt.Printf("Write More - \"%s\"\n", b.String())
/*
//This will fix the problem
var newBuffer bytes.Buffer
newBuffer.WriteString(b.String())
return newBuffer
*/
return b
}
Working Code on Go Playground
Or you can create a global buffer to change the value inside the buffer whenever it is written by any function.
package main
import (
"bytes"
"fmt"
)
var b bytes.Buffer
func main() {
//Commenting the below line will fix the problem
b.WriteString("aas-")
fmt.Printf("Before Calling - \"%s\"\n", b.String())
b := makeMeMad(b)
fmt.Printf("FinalValue - \"%s\"\n", b.String())
}
func makeMeMad(b bytes.Buffer) bytes.Buffer {
b.WriteString("xcxxcx asdasdas dasdsd asdasdasdasd")
fmt.Printf("Write More - \"%s\"\n", b.String())
/*
//This will fix the problem
var newBuffer bytes.Buffer
newBuffer.WriteString(b.String())
return newBuffer
*/
return b
}
Playground Example
I'd like to create a situation where everything set to a particular log.Logger is also appended to a particular variable's array of strings.
The variable's type implements the io.Writer interface so it should be easy to add that via io.MultiWriter to log.New(), but I seem to have run into an intractable problem: the io.Writer interface is fixed and it's impossible for the variable to reference itself given golang's pass-by-value.
Maybe it will make more sense with an example:
package main
import "fmt"
import "io"
import "log"
import "os"
import "strings"
var Log *log.Logger
type Job_Result struct {
Job_ID int64
// other stuff
Log_Lines []string
}
// satisfies io.Writer interface
func (jr Job_Result) Write (p []byte) (n int, err error) {
s := strings.TrimRight(string(p),"\n ")
jr.Log_Lines= append(jr.Log_Lines,s)
return len(s), nil
}
func (jr Job_Result) Dump() {
fmt.Println("\nHere is a dump of the job result log lines:")
for n, s := range jr.Log_Lines{
fmt.Printf("\tline %d: %s\n",n,s)
}
}
func main() {
// make a Job_Result
var jr Job_Result
jr.Job_ID = 123
jr.Log_Lines = make([]string,0)
// create an io.MultiWriter that points to both stdout
// and that Job_Result var
var writers io.Writer
writers = io.MultiWriter(os.Stdout,jr)
Log = log.New(writers,
"",
log.Ldate|log.Ltime|log.Lshortfile)
// send some stuff to the log
Log.Println("program starting")
Log.Println("something happened")
Log.Printf("last thing that happened, should be %drd line\n",3)
jr.Dump()
}
This is the output, which is not surprising:
2016/07/28 07:20:07 testjob.go:43: program starting
2016/07/28 07:20:07 testjob.go:44: something happened
2016/07/28 07:20:07 testjob.go:45: last thing that happened, should be 3rd line
Here is a dump of the job result log lines:
I understand the problem - Write() is getting a copy of the Job_Result variable, so it's dutifully appending and then the copy vanishes as it's local. I should pass it a pointer to the Job_Result...but I'm not the one calling Write(), it's done by the Logger, and I can't change that.
I thought this was a simple solution to capturing log output into an array (and there is other subscribe/unsubscribe stuff I didn't show), but it all comes down to this problematic io.Write() interface.
Pilot error? Bad design? Something I'm not grokking? Thanks for any advice.
redefine the write function (is now pointer receiver)
// satisfies io.Writer interface
func (jr *Job_Result) Write (p []byte) (n int, err error) {
s := strings.TrimRight(string(p),"\n ")
jr.Log_Lines= append(jr.Log_Lines,s)
return len(s), nil
}
initialize
jr := new(Job_Result) // makes a pointer.
rest stays as is. This way, *Job_Result still implements io.Writer, but doesn't lose state.
The go tutorial already said, when a method modifies the receiver, you should probably use a pointer receiver, or the changes may be lost. Working with a pointer instead of the actual object has little downside, when you want to make sure, there is exactly one object. (And yes, it technically isn't an object).
I'm trying to get a better understanding of go. I created a little exercise for myself: pass a pointer slice to a function and modify it.
This is what I came up with:
package main
import (
"fmt"
"unsafe"
)
func main() {
var data *[]byte
fillData(data)
fmt.Println((*data)[0:5])
}
func fillData(data *[]byte) {
b := []byte("hello")
fmt.Println(b[0:5])
fmt.Println(string(b[0:5]))
data = (*[]byte)(unsafe.Pointer(&b[0]))
}
But it gives an invalid memory address or nil pointer dereference error. I know I wouldn't use something like this in real code but I was just curious how to pass a slice and modify it without returning it.
https://play.golang.org/p/_K5ltKKKNV
When you try to update data in fillData, you make two errors. First, you update the pointer rather than what it's pointed to. Second, data is a nil pointer, so writing through that pointer will cause a nil pointer error.
Here's one possible way to write the code. data starts as a zero'ed slice, and gets updated inside fillData. This will copy the slice information (len, cap, and pointer to array) from b to *data which means that data will share information with b (importantly, including sharing the underlying array).
package main
import "fmt"
func main() {
var data []byte
fillData(&data)
fmt.Println(data, data[0:5])
}
func fillData(data *[]byte) {
b := []byte("hello")
*data = b[0:1]
}
Another way would be to have data being a pointer, and updating it. Then you have to pass a double pointer into fillData. That would look like this:
package main
import "fmt"
func main() {
var data *[]byte
fillData(&data)
fmt.Println((*data)[0:5])
}
func fillData(data **[]byte) {
b := []byte("hello")
*data = &b
}
Finally, the best way to write this code isn't to use pointers at all, and just return the slice. Unlike C or C++, it's rarely needed to use "output" parameters to functions. That's because go allows multiple return values.
package main
import "fmt"
func main() {
data := getData()
fmt.Println(data, data[0:5])
}
func getData() []byte {
return []byte("hello")[:1]
}
I'm trying to assign a value to a field, but my program panics with runtime error: invalid memory address or nil pointer dereference.
package main
type Node struct {
Value int
}
func (n *Node) SetValue(value int) {
n.Value = value
}
func main() {
var n *Node
n.SetValue(1)
}
This is reasonable since variable is nil.
But I've fount some Go internal structs are allowed to do this, e.g. bytes.Buffer
package main
import "bytes"
import "io"
import "os"
func main() {
var b bytes.Buffer
b.Write([]byte("Hello world"))
io.Copy(os.Stdout, &b)
}
Here is the `bytes.Buffer source code
func (b *Buffer) Write(p []byte) (n int, err error) {
b.lastRead = opInvalid
m := b.grow(len(p))
return copy(b.buf[m:], p), nil
}
Is it the thing only builtin structs can do or it's possible to accomplish this in my code?
EDIT
Here is the working example. Thanks #twotwotwo for suggestion.
package main
import "fmt"
type Node struct {
Value int
}
func (n *Node) SetValue(value int) {
n.Value = value
}
func main() {
var n Node
n.SetValue(1)
fmt.Println(n.Value)
}
The crucial thing is var b bytes.Buffer doesn't get you a nil pointer, it gets you a bytes.Buffer object with all its fields initialized with their zero values (in machine terms, with zero bytes). The spec says the zero value is "false for booleans, 0 for integers, 0.0 for floats, "" for strings, and nil for pointers, functions, interfaces, slices, channels, and maps"; follow that link for more detail.
It is possible to make your own structs whose zero values work and the Go team encourages it. struct Position { x, y int } is an easy example and Effective Go gives a more realistic one. But note that that doesn't make the nil pointer work; you would still need new(Node) or var n Node to allocate the zero Node. Same for bytes.Buffer.
Another common use of zero values: wherever your users create structs of your type directly (as folks do with, say, http.Server), the zero value is the default for any fields they don't specify. It's the default in a lot of other places: what you get for a not-found map key, if you receive from a closed channel, and probably others.