convert from `bufio.Reader` to `io.ReadWriteCloser` - go

I have an io.ReadWriteCloser in which I want to peek into it without advancing the reader,
so I am using
bi := bufio.NewReader(i)
bi.Peek(1)
So far so good, but later when I want to reuse the original io.ReadWriteCloser (i) it has only EOF.
So my question is how to convert back from bufio.Reader back to io.ReadWriteCloser

Because the bufio.Reader buffers data from the underlying reader, the application must read from the bufio.Reader after the call to Peek.
To get an io.ReadWriteCloser that does this, wrap the bufio.Reader and the original io.ReadWriteCloser:
// BufferedReadWriteCloser has all of the methods
// from *bufio.Reader and io.ReadWriteCloser.
type BufferedReadWriteCloser struct {
*bufio.Reader
io.ReadWriteCloser
}
func (rw *BufferedReadWriteCloser) Read(p []byte) (int, error) {
return rw.Reader.Read(p)
}
Here's how to use it:
rw := &BufferedReadWriteCloser{bufio.NewReader(i), i}
p, err := rw.Peek(1)
The value of rw satisfies the io.ReadWriteCloser interface.
There is no requirement or assumption that the io.ReadWriteCloser has a Seek method.

As mentioned in my comment above, you need access to the original reader's Seek method. This means that passing the reader around as an io.ReadWriteCloser is insufficient. Having said that, the following helper function may be a workaround:
func peek(r io.Reader, n int) ([]byte, error) {
bi := bufio.NewReader(r)
peeked, err := bi.Peek(n)
if err != nil {
return nil, err
}
// Use type assertion to check if r implements the
// io.Seeker interface. If it does, then use it to
// reset the offset.
if seeker, ok := r.(io.Seeker); ok {
seeker.Seek(0, 0)
}
return peeked, nil
}
Now you can pass the io.ReadWriteCloser to this peek function. The peek function checks if the reader happens to implement the Seek method. If the Seek method is implemented, then peek will call it.

Related

How to wrap net.Conn.Read() in Golang

I want to wrap Read function net.Conn.Read(). The purpose of this to read the SSL handshake messages. https://pkg.go.dev/net#TCPConn.Read
nc, err := net.Dial("tcp", "google.com:443")
if err != nil {
fmt.Println(err)
}
tls.Client(nc, &tls.Config{})
Are there any ways to do?
Thanks in advance
Use the following code to intercept Read on a net.Conn:
type wrap {
// Conn is the wrapped net.Conn.
// Because it's an embedded field, the
// net.Conn methods are automatically promoted
// to wrap.
net.Conn
}
// Read calls through to the wrapped read and
// prints the bytes that flow through. Replace
// the print statement with whatever is appropriate
// for your application.
func (w wrap) Read(p []byte) (int, error) {
n, err := w.Conn.Read()
fmt.Printf("%x\n", p[:n]) // example
return n, err
}
Wrap like this:
tnc, err :=tls.Client(wrap{nc}, &tls.Config{})
The previous answer gets the job done indeed.
However I would recommend Liz Rice's talk: GopherCon 2018: Liz Rice - The Go Programmer's Guide to Secure Connections
Going through her code in Github, you might find a more elegant way to achieve what you want.
Start with the client code on line 26.

How to correctly check io.Reader for nil?

If you create a variable of type bytes.Buffer (without initialization) and assign it to a field of type io.Reader, then after checking io.Reader for nil there will be an error: invalid memory address or nil pointer dereference. How to check this correctly to avoid such errors?
Playground
package main
import (
"bytes"
"io"
"io/ioutil"
)
type Request struct {
Body io.Reader
}
func main() {
var data *bytes.Buffer
request := &Request{
Body: data,
}
if request.Body != nil {
ioutil.ReadAll(request.Body) // panic: runtime error: invalid memory address or nil pointer dereference
}
}
To check if an io.Reader (or any other interface) value is nil, you simply compare it to nil.
Whether a non-nil io.Reader is a meaningful implementation, that's another question.
E.g. is this implementation meaningful?
type panicReader struct{}
func (panicReader) Read(p []byte) (int, error) {
panic("foo")
}
panicReader certainly implements io.Reader, but whenever you call its Read() method, it will always panic.
There is bytes.Buffer. A pointer to it implements io.Reader. But calling Buffer.Read() on a nil *bytes.Buffer pointer value will panic. But not because you can't call methods on nil pointer receivers, but because the implementation of bytes.Buffer.Read() tries to dereference the pointer receiver, and this dereference operation is what causes the panic:
// Excerpt from bytes.Buffer.Read implementation
func (b *Buffer) Read(p []byte) (n int, err error) {
b.lastRead = opInvalid
if b.empty() {
// ...
}
You can't make a general conclusion here (just yet). See this io.Reader implementation:
type myBuffer struct{}
var count int
func (*myBuffer) Read(p []byte) (int, error) {
if len(p) > 0 {
count++
if count >= 10 {
return 0, io.EOF
}
p[0] = 'a'
return 1, nil
}
return 0, nil
}
*myBuffer implements io.Reader, and its Read() method does not use the pointer receiver value. What does this mean? You can call Read() on a nil *myBuffer value:
var data *myBuffer
request := &Request{
Body: data,
}
if request.Body != nil {
data, err := ioutil.ReadAll(request.Body)
fmt.Println(string(data), err)
}
This will output (try it on the Go Playground):
aaaaaaaaa <nil>
So the conclusion is this: usually types that have methods with pointer receiver require a non-nil pointer because they use the pointed object (in case of bytes.Buffer they use the fields of the pointed struct). To use such types (to have a meaningful implementation of implemented interfaces), you often need a non-nil pointer value for the methods to "work". This–however–is not always a requirement as the above myBuffer implementation shows. It's your job to always read the documentation of the used types and methods to avoid such misuses (e.g. trying to use a nil *bytes.Buffer).
See related question:
Hiding nil values, understanding why Go fails here
Go reflection with interface embedded in struct - how to detect "real" functions?

Cast interface to concrete type - type switch

I am looking for a way to cast an interface into the concrete type to save a lot of source code.
The initial situation are two functions of a webserver handler. They differ only in that one function decodes an array of structs and the other decodes a single struct and stores it in the database. The functions which have to be called for saving are identical depending on the type.
To decide whether an array or a struct is passed, it is tried to cast the interface into the type and then to pass this appropriately as a parameter of the function. Similar as it is described in the documentation and in the stackoverflow post.
However, I do not get the concrete type as expected and the program always runs into the default section. What am I doing wrong or have I not taken into account?
These are the outputs of the default section:
# interface is a struct
... or a single repository struct: map[string]interface{}
# interface is an array of structs
... or a single repository struct: []interface{}
Below is the source code with the functions
func (rh *RouteHandler) AddOrUpdateRepository(rw http.ResponseWriter, req *http.Request) {
repository := new(types.Repository)
rh.addOrUpdateRepositories(rw, req, repository)
}
func (rh *RouteHandler) AddOrUpdateRepositories(rw http.ResponseWriter, req *http.Request) {
repositories := make([]*types.Repository, 0)
rh.addOrUpdateRepositories(rw, req, repositories)
}
func (rh *RouteHandler) addOrUpdateRepositories(rw http.ResponseWriter, req *http.Request, v interface{}) {
defer req.Body.Close()
switch req.Header.Get("Content-Type") {
case "application/xml":
xmlDecoder := xml.NewDecoder(req.Body)
err := xmlDecoder.Decode(&v)
if err != nil {
rw.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(rw, "Failed to decode repositories or repository")
rh.ulogger.Error("Failed to decode repositories or repository: %v", err)
return
}
case "application/json":
fallthrough
default:
jsonDecoder := json.NewDecoder(req.Body)
err := jsonDecoder.Decode(&v)
if err != nil {
rw.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(rw, "Failed to decode repositories or repository")
rh.ulogger.Error("Failed to decode repositories or repository: %v", err)
return
}
}
var err error
switch x := v.(type) {
case map[string]*types.Repository:
for _, repository := range x {
err = rh.manager.AddOrUpdateRepository(context.Background(), repository)
}
case *types.Repository:
err = rh.manager.AddOrUpdateRepository(context.Background(), x)
case map[string][]*types.Repository:
for i := range x {
for j := range x[i] {
err = rh.manager.AddOrUpdateRepository(context.Background(), x[i][j])
}
}
case []*types.Repository:
err = rh.manager.AddOrUpdateRepository(context.Background(), x...)
case nil:
rw.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(rw, "Failed to cast interface")
rh.ulogger.Error("Failed to cast interface. Interface is a type of nil")
return
default:
rw.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(rw, "Failed to cast interface")
rh.ulogger.Error("Failed to cast interface. Interface does not match onto an array of repositories or a single repository struct: %T", x)
return
}
if err != nil {
rw.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(rw, "Failed to add repositories or repository")
rh.ulogger.Error("Failed to add repositories or repository: %v", err)
return
}
rw.WriteHeader(http.StatusCreated)
}
(Simplified a bit.)
You have a function with the following signature:
func addOrUpdateRepositories(v interface{})
and then you call it like this:
repository := new(types.Repository)
addOrUpdateRepositories(repository)
and like this:
repositories := make([]*types.Repository, 0)
addOrUpdateRepositories(repositories)
In the first call, the concrete type of the value stored in v will be *types.Repository (as new returns a pointer to the allocated value) and in the second call the concrete type of the value stored in v will be []*types.Repository—because that's what make was told to create.
Now you do a type-switch on v, which reads:
switch x := v.(type) {
case map[string]*types.Repository:
case map[string][]*types.Repository:
case nil:
default:
}
Leaving aside the case that if you do not call addOrUpdateRepositories passing it a nil v which cannot happen in the snippet from your question, the switch will always pick the default branch because the type of the concrete value stored in v is never map[string]*types.Repository or map[string][]*types.Repository.
I am not sure why you fail to see this, so probably you should refine your question or may be try to clear up your confusion in a comment to my answer?
Another shot in the dark: type conversions (note that Go does not have type casts, as #Flimzy pointed out) and type switches in Go do not actually change the underlying representation of the value they operate on—except for the limited set of ("everyone expects this") cases such as type-converting a float64 to int64, which are precisely documented.
So you cannot take a []*types.Repository (a slice of pointers to values of type types.Repository) and somehow force it "to become" map[string][]*types.Repository: that'd be a nonsensical thing to do for a number of reasons, with the most glaring being this: if you were writing the Go compiler, how would you carry out such a "type cast"? Suppose you were to really allocate a map, but then which key in that map should have been assigned the original (source) slice? What about type-converting a []*types.Repository to struct {foo []*types.Repository; bar []*types.Repository}?

io.Writer in Go - beginner trying to understand them

As a beginner in Go, I have problems understanding io.Writer.
My target: take a struct and write it into a json file.
Approach:
- use encoding/json.Marshal to convert my struct into bytes
- feed those bytes to an os.File Writer
This is how I got it working:
package main
import (
"os"
"encoding/json"
)
type Person struct {
Name string
Age uint
Occupation []string
}
func MakeBytes(p Person) []byte {
b, _ := json.Marshal(p)
return b
}
func main() {
gandalf := Person{
"Gandalf",
56,
[]string{"sourcerer", "foo fighter"},
}
myFile, err := os.Create("output1.json")
if err != nil {
panic(err)
}
myBytes := MakeBytes(gandalf)
myFile.Write(myBytes)
}
After reading this article, I changed my program to this:
package main
import (
"io"
"os"
"encoding/json"
)
type Person struct {
Name string
Age uint
Occupation []string
}
// Correct name for this function would be simply Write
// but I use WriteToFile for my understanding
func (p *Person) WriteToFile(w io.Writer) {
b, _ := json.Marshal(*p)
w.Write(b)
}
func main() {
gandalf := Person{
"Gandalf",
56,
[]string{"sourcerer", "foo fighter"},
}
myFile, err := os.Create("output2.json")
if err != nil {
panic(err)
}
gandalf.WriteToFile(myFile)
}
In my opinion, the first example is a more straightforward and easier to understand for a beginner... but I have the feeling that the 2nd example is the Go idiomatic way of achieving the target.
Questions:
1. is above assumption correct (that 2nd option is Go idiomatic) ?
2. Is there a difference in the above options ? Which option is better ?
3. other ways to achieve the same target ?
Thank you,
WM
The benefit of using the second method is that if you are passing a Writer interface, you can pass anything which implements Write -- that is not only a file but a http.ResponseWriter, for example, or stdout os.Stdout, without changing the struct methods.
You can see this handy blog post on the package io walkthrough. The author makes the case that passing as parameter readers and writers makes your code more flexible, in part because so many functions use the Reader and Writer interface.
As you come to use Go more, you'll notice how much the standard library leans on Reader and Writer interfaces, and probably come to appreciate it :)
So this function (renamed):
// writes json representation of Person to Writer
func (p *Person) WriteJson(w io.Writer) error {
b, err := json.Marshal(*p)
if err != nil {
return err
}
_, err = w.Write(b)
if err != nil {
return err
}
return err
}
Would write to a File, http Response, a user's Stdout, or even a simple byte Buffer; making testing a bit simpler.
I renamed it because of what is does; that is, this function takes a Person struct and:
Marshals the struct into a json representation
Writes the json to a Writer
Returns any errors arising from marshalling/writing
One more thing, you might be confused as to what a Writer is, because it is not a data type, but rather an interface -- that is a behavior of a data type, a predefined method that a type implements. Anything that implements the Write() method, then, is considered a writer.
This can be a bit difficult for beginners to grasp at first, but there are lots of resources online to help understand interfaces (and ReadWriters are some of the more common interfaces to encounter, along with Error() (ei. all errors)).

Golang update pass by reference value

I am calling a function to do a http request, two pass by reference parameter is used for the function. I pass the []byte to v interface. I want the function to update the v interface reference value. The response body is a string, I want to pass the string value to v interface. However, tried many ways but not success.
Here is the code, you can see I declare byts as v.(*[]byte) in order to make v updated with the string value of response body. But it does not work. The v is always nil. Please suggest any way to make v can be updated with the string value.
func (s *BackendConfiguration) Do(req *http.Request, v interface{}) error {
res, err := s.HTTPClient.Do(req)
defer res.Body.Close()
resBody, err := ioutil.ReadAll(res.Body)
if v != nil {
byts, ok := v.(*[]byte)
if len(resBody) > 0 {
byts = append(byts, resBody...)
return nil
}
}
}
return nil
}
Well, the main reason this does not work is because you think of "call by reference", a concept completely unknown to Go. Absolutely everything is called by value in Go and once you spell out what is a byte slice, a pointer to a byte slice, a pointer to byte slice wrapped inside an interface, a copy of the pointer to a byte slice extracted from the interface, and so on you'll see how to update the value the pointer to byte slice points to:
package main
import "fmt"
func f(v interface{}) {
pbs := v.(*[]byte)
*pbs = append(*pbs, []byte{9,8,7}...)
}
func main() {
bs := []byte{1,2,3}
pbs := &bs
var v interface{} = pbs
f(v)
fmt.Printf("%v\n", *pbs)
}

Resources