io.Writer in Go - beginner trying to understand them - go

As a beginner in Go, I have problems understanding io.Writer.
My target: take a struct and write it into a json file.
Approach:
- use encoding/json.Marshal to convert my struct into bytes
- feed those bytes to an os.File Writer
This is how I got it working:
package main
import (
"os"
"encoding/json"
)
type Person struct {
Name string
Age uint
Occupation []string
}
func MakeBytes(p Person) []byte {
b, _ := json.Marshal(p)
return b
}
func main() {
gandalf := Person{
"Gandalf",
56,
[]string{"sourcerer", "foo fighter"},
}
myFile, err := os.Create("output1.json")
if err != nil {
panic(err)
}
myBytes := MakeBytes(gandalf)
myFile.Write(myBytes)
}
After reading this article, I changed my program to this:
package main
import (
"io"
"os"
"encoding/json"
)
type Person struct {
Name string
Age uint
Occupation []string
}
// Correct name for this function would be simply Write
// but I use WriteToFile for my understanding
func (p *Person) WriteToFile(w io.Writer) {
b, _ := json.Marshal(*p)
w.Write(b)
}
func main() {
gandalf := Person{
"Gandalf",
56,
[]string{"sourcerer", "foo fighter"},
}
myFile, err := os.Create("output2.json")
if err != nil {
panic(err)
}
gandalf.WriteToFile(myFile)
}
In my opinion, the first example is a more straightforward and easier to understand for a beginner... but I have the feeling that the 2nd example is the Go idiomatic way of achieving the target.
Questions:
1. is above assumption correct (that 2nd option is Go idiomatic) ?
2. Is there a difference in the above options ? Which option is better ?
3. other ways to achieve the same target ?
Thank you,
WM

The benefit of using the second method is that if you are passing a Writer interface, you can pass anything which implements Write -- that is not only a file but a http.ResponseWriter, for example, or stdout os.Stdout, without changing the struct methods.
You can see this handy blog post on the package io walkthrough. The author makes the case that passing as parameter readers and writers makes your code more flexible, in part because so many functions use the Reader and Writer interface.
As you come to use Go more, you'll notice how much the standard library leans on Reader and Writer interfaces, and probably come to appreciate it :)
So this function (renamed):
// writes json representation of Person to Writer
func (p *Person) WriteJson(w io.Writer) error {
b, err := json.Marshal(*p)
if err != nil {
return err
}
_, err = w.Write(b)
if err != nil {
return err
}
return err
}
Would write to a File, http Response, a user's Stdout, or even a simple byte Buffer; making testing a bit simpler.
I renamed it because of what is does; that is, this function takes a Person struct and:
Marshals the struct into a json representation
Writes the json to a Writer
Returns any errors arising from marshalling/writing
One more thing, you might be confused as to what a Writer is, because it is not a data type, but rather an interface -- that is a behavior of a data type, a predefined method that a type implements. Anything that implements the Write() method, then, is considered a writer.
This can be a bit difficult for beginners to grasp at first, but there are lots of resources online to help understand interfaces (and ReadWriters are some of the more common interfaces to encounter, along with Error() (ei. all errors)).

Related

convert from `bufio.Reader` to `io.ReadWriteCloser`

I have an io.ReadWriteCloser in which I want to peek into it without advancing the reader,
so I am using
bi := bufio.NewReader(i)
bi.Peek(1)
So far so good, but later when I want to reuse the original io.ReadWriteCloser (i) it has only EOF.
So my question is how to convert back from bufio.Reader back to io.ReadWriteCloser
Because the bufio.Reader buffers data from the underlying reader, the application must read from the bufio.Reader after the call to Peek.
To get an io.ReadWriteCloser that does this, wrap the bufio.Reader and the original io.ReadWriteCloser:
// BufferedReadWriteCloser has all of the methods
// from *bufio.Reader and io.ReadWriteCloser.
type BufferedReadWriteCloser struct {
*bufio.Reader
io.ReadWriteCloser
}
func (rw *BufferedReadWriteCloser) Read(p []byte) (int, error) {
return rw.Reader.Read(p)
}
Here's how to use it:
rw := &BufferedReadWriteCloser{bufio.NewReader(i), i}
p, err := rw.Peek(1)
The value of rw satisfies the io.ReadWriteCloser interface.
There is no requirement or assumption that the io.ReadWriteCloser has a Seek method.
As mentioned in my comment above, you need access to the original reader's Seek method. This means that passing the reader around as an io.ReadWriteCloser is insufficient. Having said that, the following helper function may be a workaround:
func peek(r io.Reader, n int) ([]byte, error) {
bi := bufio.NewReader(r)
peeked, err := bi.Peek(n)
if err != nil {
return nil, err
}
// Use type assertion to check if r implements the
// io.Seeker interface. If it does, then use it to
// reset the offset.
if seeker, ok := r.(io.Seeker); ok {
seeker.Seek(0, 0)
}
return peeked, nil
}
Now you can pass the io.ReadWriteCloser to this peek function. The peek function checks if the reader happens to implement the Seek method. If the Seek method is implemented, then peek will call it.

Writing non-string value into io.Writer Write method

I'm trying to write non-string value into io.Writer like integer, float, slices, or even map[string]interface{}. I'm expecting the result written returned as expected type written. If I wrote int into the Write, then I will get integer type value after decoding the []byte returned. How to do it in Go?
What you're probably looking for is encoding/gob since that encoding retains the Go type information. Out of the box it supports some of the builtin Go types and some of the basic gob types. If you want to encode/decode types not supported out of the box by gob you can use the gob.Register function to register those types.
To encode:
var v interface{} = uint8(123)
if err := gob.NewEncoder(w).Encode(&v); err != nil {
panic(err)
}
Note that the above passes a value of type *interface{} to Encode, this is necessary if, at the other end, the decoder doesn't know the type beforehand and has to also use type *interface{} as the argument to Decode. If you have a scenario where the decoder knows the concrete type the of the incoming data then you can also pass a value of that concrete type to Encode.
To decode:
var v interface{}
if err := gob.NewDecoder(r).Decode(&v); err != nil {
panic(err)
}
fmt.Println(v) // output: 123
fmt.Printf("%T", v) // output: uint8
https://play.golang.org/p/cCtQse8BoqZ
This seems to do it:
package main
import "encoding/json"
func main() {
a := []interface{}{
31, 3.1, []int{12,31}, map[string]interface{}{"month": 12, "day": 31},
}
b, err := json.Marshal(a)
if err != nil {
panic(err)
}
println(string(b)) // [31,3.1,[12,31],{"day":31,"month":12}]
}
https://pkg.go.dev/encoding/json#Marshal

Uses of io.ReadCloser

Could someone please explain (or/and share examples) when and why readers should to be closed explicitly, i.e. implement io.ReadCloser, not just io.Reader.
For example, when you are working with files, or any resource that should be closed to release the allocated resource (or memory for example for your resource, e.g. C code calling from Go).
You may use it when you have Read and Close methods, an example to show that you may use one common function to work with different types using io.ReadCloser:
package main
import (
"fmt"
"io"
"log"
"os"
)
func main() {
f, err := os.Open("./main.go")
if err != nil {
log.Fatal(err)
}
doIt(f)
doIt(os.Stdin)
}
func doIt(rc io.ReadCloser) {
defer rc.Close()
buf := make([]byte, 4)
n, err := rc.Read(buf)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", buf[:n])
}
Run and enter 12345 as an input, Output:
pack
12345
1234
See also:
Does Go automatically close resources if not explicitly closed?
It's for an explicit definition of Reader and Closer interface. So let's say you write some functionality that reads data, but you also want to close resource after doing it (again not to leak descriptors).
func ...(r io.ReaderCloser) {
defer r.Close()
... // some reading
}
anything you pass in it will need to have both interfaces defined, is it os.File or any custom struct, in this case, you are forcing client of your API to define Read and Close interfaces implementations, not just io.Reader.

How do I write a gota dataframe to a csv?

I have found many many code examples of writing to a CSV by passing in a [][]string. (like the following):
package main
import (
"os"
"log"
"encoding/csv"
)
var data = [][]string{
{"Row 1", "30"},
{"Row 2", "60"},
{"Row 3", "90"}}
func main() {
file, err := os.Create("tutorials_technology.csv")
if err != nil {
log.Fatal(err)
}
defer file.Close()
w := csv.NewWriter(file)
for _, value := range data {
if err := w.Write(value); err != nil {
log.Fatalln("Error writing record to csv: ", err)
}
}
w.Flush()
}
However, I haven't found any code examples that show how to use the gota dataframe.WriteCSV() function to write to a CSV. In the gota dataframe documentation, there isn't an example for writing to a csv, but there is an example for reading from a csv.
The dataframe function WriteCSV() requires an input of the io.Writer{} interface. I wasn't sure how to satsify that.
The following didn't work
writer := csv.NewWriter(f)
df.WriteCSV(writer) // TODO This writer needs to be a []byte writer
I've been working on this for quite a while. Does anyone have any clues?
I have looked into turning my gota dataframe into a [][]string type, but that's a little inconvenient because I put my data into a gota dataframe with the package's LoadStructs() function and I had read in some CSV in a semi-custom way before putting them into structs.
So I could write a function to turn my structs into a [][]string format, but I feel like that is pretty tedious and I'm sure there has got to be a better way. In fact, I'm sure there is because the dataframe type has the WriteCSV() method but I just haven't figured out how to use it.
Here are my structs
type CsvLine struct {
Index int
Date string
Symbol string
Open float64
High float64
Low float64
Close float64
// Volume float64
// Market_Cap float64
}
type File struct {
Rows []CsvLine
}
Disclaimer: I am a little bit of a golang newbie. I've only been using Go for a couple months, and this is the first time I've tried to write to a file. I haven't interacted much with the io.Writer interface, but I hear that it's very useful.
And yes, I frequently look at the Golang.org blog and I've read "Effective Go" and I keep referencing it.
So it turns out I misunderstood the io.Writer interface and I didn't understand what the os.Create() function returns.
It turns out the code is even simpler and easier than I thought it would be.
Here is the working code example:
df := dataframe.LoadStructs(file.Rows)
f, err := os.Create(outFileName)
if err != nil {
log.Fatal(err)
}
df.WriteCSV(f)

How to send a message to an object in Golang? (.send() equivalent in go)

I am a Go beginner, coming from Ruby land.
In Ruby, you could do something like this.
Time.send("now") is equivalent to Time.now, as you are sending the message now to the object Time
Is there something similar in golang?
There is no built in way of calling an arbitrary function from a string in Go.
You can create something similar by registering functions to a map[string].
A working example:
package main
import "fmt"
var m = map[string]func(){
"now": func() { fmt.Println("The time is now") },
"then": func() { fmt.Println("Once upon a time") },
}
func main() {
cmd := "then"
m[cmd]()
}
play.golang.org
There is also the possibility of using reflection in order to call a method by name. You can look at the reflect package for MethodByName and Call. You can also check this Stackoverflow question.
As other suggested, you can do it yourself by mapping strings to functions, but the strong-typing nature of Go makes it difficult to translate .send directly into Go.
You can still use reflection if you really need to access a field or method by name:
import "reflect"
import "fmt"
type A struct {
Number int
}
func (a *A) Method(i int) int {
return a.Number + i;
}
func main() {
a := &A{Number: 1}
// Direct access
fmt.Printf("Direct -> Nb: %d, Nb + 2: %d\n", a.Number, a.Method(2));
v := reflect.ValueOf(*a)
vp := reflect.ValueOf(a)
field := v.FieldByName("Number")
meth := vp.MethodByName("Method")
args := []reflect.Value{reflect.ValueOf(2)}
// Reflection access
fmt.Printf("Reflect -> Nb: %d, Nb + 2: %d\n",
field.Interface().(int),
meth.Call(args)[0].Interface().(int))
}
Outputs:
Direct -> Nb: 1, Nb + 2: 3
Reflect -> Nb: 1, Nb + 2: 3
play.golang.org
Note however:
How cumbersome that is. Usually, performing a map as suggested by #ANisus is a more idiomatic way of doing
You still have to perform your conversions in the end.
Using the reflect packages changes your typed variable into more flexible Value objects, but these are very cumbersome to use in practice. It is usually better if you can find a way to express your intent without relying on reflection.
Also note that here, we had to use two Values, one for a (a pointer to A) for the method, and one for *a (a A structure) for the field. Trying to get a method defined with a pointer receiver with a non-pointer Value (or conversely, trying to obtain a field via a pointer Value) will result in a panic. More generally, due to the dynamic nature of reflected Values and its difference with the usual typed Go, expect a lot of convenience features (such as automatic referencing/dereferencing) to be absent on Values.
Also, expect quite a bit of runtime panics while debugging, as it is the only way for dynamic Value calls to fail !
Reference: the reflect package
No. Work your way through http://tour.golang.org/ and http://golang.org/doc/effective_go.html and you will have a proper understanding of how method invocation works.
Here is a working example using reflect
package main
import (
"fmt"
"os"
"reflect"
)
// Send sends a message to(calls a method of) obj, with args.
// The return value of the method call is set to ret and any error to err.
func Send(obj interface{}, method string, args ...interface{}) (ret []reflect.Value, err error) {
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
}()
objValue := reflect.ValueOf(obj)
argsValue := make([]reflect.Value, 0, len(args))
for _, arg := range args {
argsValue = append(argsValue, reflect.ValueOf(arg))
}
mtd := objValue.MethodByName(method)
if !mtd.IsValid() {
return nil, fmt.Errorf("%v does not have a method %v", reflect.TypeOf(obj), method)
}
ret = mtd.Call(argsValue)
return
}
// Then do some tests.
type A struct {
value int
}
func (a A) Value() int {
return a.value
}
func (a *A) SetValue(v int) {
a.value = v
}
func main() {
var (
ret []reflect.Value
err error
)
// StdOut.WriteString("Hello, World!\n")
_, err = Send(os.Stdout, "WriteString", "Hello, World!\n")
handleError(err)
var a = &A{100}
// ret = a.Value()
ret, err = Send(a, "Value")
handleError(err)
fmt.Printf("Return value is: %v\n", ret[0].Int())
// a.SetValue(200)
_, err = Send(a, "SetValue", 200)
handleError(err)
// ret = a.Value()
ret, err = Send(a, "Value")
handleError(err)
fmt.Printf("Return value is: %v", ret[0].Int())
}
func handleError(err error) {
if err != nil {
panic(err)
}
}
I based my code on this description of send.
class Klass
def hello(*args)
"Hello " + args.join(' ')
end
end
k = Klass.new
k.send :hello, "gentle", "readers" #=> "Hello gentle readers"
http://play.golang.org/p/lXlzBf_fGZ
package main
import "strings"
type Klass struct{}
func (k Klass) Hello(args ...string) string {
return "Hello " + strings.Join(args, " ")
}
func (k Klass) Send(symbol func(Klass, ...string) string, args ...string) string {
return symbol(k, args...)
}
func main() {
k := new(Klass)
k.Send(Klass.Hello, "gentle", "readers") //=> "Hello gentle readers"
}
The big difference between the two is that Go's Send function is only implemented for Klass and only works on methods that take a variable number of strings as parameters and return a single string. This is because Go is a statically typed language where Ruby is dynamically typed. Go does support dynamic typing via the reflect library, but it is an unpleasant experience and not the way general Go code is meant to be written.

Resources