Exercise: Readers
Implement a Reader type that emits an infinite stream of the ASCII character 'A'.
I don't understand the question, how to emit character 'A'? into which variable should I set that character?
Here's what I tried:
package main
import "golang.org/x/tour/reader"
type MyReader struct{}
// TODO: Add a Read([]byte) (int, error) method to MyReader.
func main() {
reader.Validate(MyReader{}) // what did this function expect?
}
func (m MyReader) Read(b []byte) (i int, e error) {
b = append(b, 'A') // this is wrong..
return 1, nil // this is also wrong..
}
Ah I understand XD
I think it would be better to say: "rewrite all values in []byte into 'A's"
package main
import "golang.org/x/tour/reader"
type MyReader struct{}
// TODO: Add a Read([]byte) (int, error) method to MyReader.
func (m MyReader) Read(b []byte) (i int, e error) {
for x := range b {
b[x] = 'A'
}
return len(b), nil
}
func main() {
reader.Validate(MyReader{})
}
An io.Reader.Read role is to write a given memory location with data read from its source.
To implement a stream of 'A', the function must write given memory location with 'A' values.
It is not required to fill in the entire slice provided in input, it can decide how many bytes of the input slice is written (Read reads up to len(p) bytes into p), it must return that number to indicate to the consumer the length of data to process.
By convention an io.Reader indicates its end by returning an io.EOF error. If the reader does not return an error, it behaves as an infinite source of data to its consumer which can never detect an exit condition.
Note that a call to Read that returns 0 bytes read can happen and does not indicate anything particular, Callers should treat a return of 0 and nil as indicating that nothing happened; Which makes this non-solution https://play.golang.org/p/aiUyc4UDYi2 fails with a timeout.
In regard to that, the solution provided here https://stackoverflow.com/a/68077578/4466350 return copy(b, "A"), nil is really just right. It writes the minimum required, with an elegant use of built-ins and syntax facilities, and it never returns an error.
The alleged answer is didn't work for me, even without the typos.
Try as I did, that string would not go into b.
func (r MyReader) Read(b []byte) (int, error) {
return copy(b, "A"), nil
}
My solution: just add one byte at a time, store the index i using closure.
package main
import (
"golang.org/x/tour/reader"
)
type MyReader struct{}
func (mr MyReader) Read(b []byte) (int, error) {
i := 0
p := func () int {
b[i] = 'A'
i += 1
return i
}
return p(), nil
}
func main() {
reader.Validate(MyReader{})
}
Simplest one:
func (s MyReader) Read(b []byte) (int, error) {
b[0] = byte('A')
return 1, nil
}
You can generalize the idea to create an eternal reader, alwaysReader, from which you always read the same byte value over and over (it never results in EOF):
package readers
type alwaysReader struct {
value byte
}
func (r alwaysReader) Read(p []byte) (n int, err error) {
for i := range p {
p[i] = r.value
}
return len(p), nil
}
func NewAlwaysReader(value byte) alwaysReader {
return alwaysReader { value }
}
NewAlwaysReader() is the constructor for alwaysReader (which isn't exported). The result of NewAlwaysReader('A') is a reader from whom you will always read 'A'.
A clarifying unit test for alwaysReader:
package readers_test
import (
"bytes"
"io"
"readers"
"testing"
)
func TestEmptyReader(t *testing.T) {
const numBytes = 128
const value = 'A'
buf := bytes.NewBuffer(make([]byte, 0, numBytes))
reader := io.LimitReader(readers.NewAlwaysReader(value), numBytes)
n, err := io.Copy(buf, reader)
if err != nil {
t.Fatal("copy failed: %w")
}
if n != numBytes {
t.Errorf("%d bytes read but %d expected", n, numBytes)
}
for i, elem := range buf.Bytes() {
if elem != value {
t.Errorf("byte at position %d has not the value %v but %v", i, value, elem)
}
}
}
Since we can read from the alwaysReader forever, we need to decorate it with a io.LimitReader so that we end up reading at most numBytes from it. Otherwise, the bytes.Buffer will eventually run out of memory for reallocating its internal buffer because of io.Copy().
Note that the following implementation of Read() for alwaysReader is also valid:
func (r alwaysReader) Read(p []byte) (n int, err error) {
if len(p) > 0 {
p[0] = r.value
return 1, nil
}
return 0, nil
}
The former Read() implementation fills the whole byte slice with the byte value, whereas the latter writes a single byte.
Related
Normally, the original value will change only if it's passed as a pointer to a method.
But I see this scenario that the original value is changed when it's passed as a value to a method. This happens a lot in implementations of Reader interface.
For example:
package main
import (
"fmt"
"io"
"strings"
)
func main() {
r := strings.NewReader("Hello, Reader!")
b := make([]byte, 8)
for {
n, err := r.Read(b)
fmt.Printf("n = %v err = %v b = %v\n", n, err, b)
fmt.Printf("b[:n] = %q\n", b[:n])
if err == io.EOF {
break
}
}
}
the variable b is passed to the Read method using value
n, err := r.Read(b)
but somehow the original value got changed and populated with data to print with.
If I look into the implementation of Read method
func (r *Reader) Read(b []byte) (n int, err error) {
if r.i >= int64(len(r.s)) {
return 0, io.EOF
}
r.prevRune = -1
n = copy(b, r.s[r.i:])
r.i += int64(n)
return
}
We can clearly see it's a value parameter?
This obeys how in go everything is passed and copied by value.
From my understand, b should at least be passed as a pointer like
func (r *Reader) Read(b *[]byte) (n int, err error) {
Why is this? Please help thanks.
A slice does not directly hold its contents. Instead a slice holds a pointer to its underlying array which holds the contents of the slice.
So, essentially yes, you are passing by value, but the value is a pointer the "backing array".
See: https://dave.cheney.net/2018/07/12/slices-from-the-ground-up
To help debug GO programs, I want to write two generic functions that will be called on entry and exit, which will print the values of input and output parameters respectively:
printInputParameters(input ...interface{})
printOutputParameters(output ...interface{})
Is there an equivalent of os.Args() for functions? I looked at runtime package and didn't find such functions.
For example lets say I have two functions with different input parameters and output parameters
func f1(int i, float f) (e error) {
... some code here
}
func f2(s string, b []byte) (u uint64, e error) {
.. some code here
}
I want to be able to do the following
func f1(int i, float f) (e error) {
printInputparameters( ? )
defer func() {
printOutputParameters( ? )
}()
... some code here
}
func f2(s string, b []byte) (u uint64, e error) {
printInputparameters( ? )
defer func() {
printOutputParameters( ? )
}()
... some code here
}
You cannot do this in Go since there is no way you can get the stack frame of the currently active function in the current goroutine. It is not impossible to do this as I'll show further below but the problem is that there is no public API to get this done reliably. That it can be done can be seen in the stack traces printed when a panic is raised: all values on the stack are dumped in that case.
Should you be interested in how the stack trace is actually generated then have a look at genstacktrace in the runtime package.
As for a solution to your problem, you can the source code parsing route as already suggested. If you feel adventurous, you can parse the stack trace provided by runtime.Stack. But beware, there are so many drawbacks that you will quickly realize that any solution is better than this one.
To parse the stack trace, just get the line of the previously called function (from the viewpoint of printInputParameters), get the name of that function and parse the parameter values according to the parameter types provided by reflection. Some examples of stack trace outputs of various function invocations:
main.Test1(0x2) // Test1(int64(2))
main.Test1(0xc820043ed5, 0x3, 0x3) // Test1([]byte{'A','B','C'})
main.Test1(0x513350, 0x4) // Test1("AAAA")
You can see that complex types (those which do not fit into a register) may use more than one 'parameter'. A string for example is a pointer to the data and the length. So you have to use the unsafe package to access these pointers and reflection to create values from this data.
If you want to try yourself, here's some example code:
import (
"fmt"
"math"
"reflect"
"runtime"
"strconv"
"strings"
"unsafe"
)
// Parses the second call's parameters in a stack trace of the form:
//
// goroutine 1 [running]:
// main.printInputs(0x4c4c60, 0x539038)
// /.../go/src/debug/main.go:16 +0xe0
// main.Test1(0x2)
// /.../go/src/debug/main.go:23
//
func parseParams(st string) (string, []uintptr) {
line := 1
start, stop := 0, 0
for i, c := range st {
if c == '\n' {
line++
}
if line == 4 && c == '\n' {
start = i + 1
}
if line == 5 && c == '\n' {
stop = i
}
}
call := st[start:stop]
fname := call[0:strings.IndexByte(call, '(')]
param := call[strings.IndexByte(call, '(')+1 : strings.IndexByte(call, ')')]
params := strings.Split(param, ", ")
parsedParams := make([]uintptr, len(params))
for i := range params {
iv, err := strconv.ParseInt(params[i], 0, 64)
if err != nil {
panic(err.Error())
}
parsedParams[i] = uintptr(iv)
}
return fname, parsedParams
}
func fromAddress(t reflect.Type, addr uintptr) reflect.Value {
return reflect.NewAt(t, unsafe.Pointer(&addr)).Elem()
}
func printInputs(fn interface{}) {
v := reflect.ValueOf(fn)
vt := v.Type()
b := make([]byte, 500)
if v.Kind() != reflect.Func {
return
}
runtime.Stack(b, false)
name, params := parseParams(string(b))
pidx := 0
fmt.Print(name + "(")
for i := 0; i < vt.NumIn(); i++ {
t := vt.In(i)
switch t.Kind() {
case reflect.Int64:
case reflect.Int:
// Just use the value from the stack
fmt.Print(params[pidx], ",")
pidx++
case reflect.Float64:
fmt.Print(math.Float64frombits(uint64(params[pidx])), ",")
pidx++
case reflect.Slice:
// create []T pointing to slice content
data := reflect.ArrayOf(int(params[pidx+2]), t.Elem())
svp := reflect.NewAt(data, unsafe.Pointer(params[pidx]))
fmt.Printf("%v,", svp.Elem())
pidx += 3
case reflect.String:
sv := fromAddress(t, params[pidx])
fmt.Printf("%v,", sv)
pidx += 2
case reflect.Map:
// points to hmap struct
mv := fromAddress(t,params[pidx])
fmt.Printf("%v,", mv)
pidx++
} /* switch */
}
fmt.Println(")")
}
Test:
func Test1(in int, b []byte, in2 int, m string) {
printInputs(Test1)
}
func main() {
b := []byte{'A', 'B', 'C'}
s := "AAAA"
Test1(2, b, 9, s)
}
Output:
main.Test1(2,[65 66 67],9,"AAAA",)
A slightly advanced version of this can be found on github:
go get github.com/githubnemo/pdump
To generically print your functions' arguments, you can do this:
func printInputParameters(input ...interface{}) {
fmt.Printf("Args: %v", input)
}
printInputParameters is a variadic function, and input is of type []interface{}.
I've got a Go text/template that renders a file, however I've found it difficult to structure the template cleanly while preserving the line breaks in the output.
I'd like to have additional, unnecessary newlines in the template to make it more readable, but strip them from the output. Any group of newlines more than a normal paragraph break should be condensed to a normal paragraph break, e.g.
lines with
too many breaks should become lines with
normal paragraph breaks.
The string is potentially too large to store safely in memory, so I want to keep it as an output stream.
My first attempt:
type condensingWriter struct {
writer io.Writer
lastLineIsEmpty bool
}
func (c condensingWriter) Write(b []byte) (n int, err error){
thisLineIsEmpty := strings.TrimSpace(string(b)) == ""
defer func(){
c.lastLineIsEmpty = thisLineIsEmpty
}()
if c.lastLineIsEmpty && thisLineIsEmpty{
return 0, nil
} else {
return c.writer.Write(b)
}
}
This doesn't work because I naively assumed that it would buffer on newline characters, but it doesn't.
Any suggestions on how to get this to work?
Inspired by zmb's approach, I've come up with the following package:
//Package striplines strips runs of consecutive empty lines from an output stream.
package striplines
import (
"io"
"strings"
)
// Striplines wraps an output stream, stripping runs of consecutive empty lines.
// You must call Flush before the output stream will be complete.
// Implements io.WriteCloser, Writer, Closer.
type Striplines struct {
Writer io.Writer
lastLine []byte
currentLine []byte
}
func (w *Striplines) Write(p []byte) (int, error) {
totalN := 0
s := string(p)
if !strings.Contains(s, "\n") {
w.currentLine = append(w.currentLine, p...)
return 0, nil
}
cur := string(append(w.currentLine, p...))
lastN := strings.LastIndex(cur, "\n")
s = cur[:lastN]
for _, line := range strings.Split(s, "\n") {
n, err := w.writeLn(line + "\n")
w.lastLine = []byte(line)
if err != nil {
return totalN, err
}
totalN += n
}
rem := cur[(lastN + 1):]
w.currentLine = []byte(rem)
return totalN, nil
}
// Close flushes the last of the output into the underlying writer.
func (w *Striplines) Close() error {
_, err := w.writeLn(string(w.currentLine))
return err
}
func (w *Striplines) writeLn(line string) (n int, err error) {
if strings.TrimSpace(string(w.lastLine)) == "" && strings.TrimSpace(line) == "" {
return 0, nil
} else {
return w.Writer.Write([]byte(line))
}
}
See it in action here: http://play.golang.org/p/t8BGPUMYhb
The general idea is you'll have to look for consecutive newlines anywhere in the input slice and if such cases exist, skip over all but the first newline character.
Additionally, you have to track whether the last byte written was a newline, so the next call to Write will know to eliminate a newline if necessary. You were on the right track by adding a bool to your writer type. However, you'll want to use a pointer receiver instead of a value receiver here, otherwise you'll be modifying a copy of the struct.
You would want to change
func (c condensingWriter) Write(b []byte)
to
func (c *condensingWriter) Write(b []byte)
You could try something like this. You'll have to test with larger inputs to make sure it handles all cases correctly.
package main
import (
"bytes"
"io"
"os"
)
var Newline byte = byte('\n')
type ReduceNewlinesWriter struct {
w io.Writer
lastByteNewline bool
}
func (r *ReduceNewlinesWriter) Write(b []byte) (int, error) {
// if the previous call to Write ended with a \n
// then we have to skip over any starting newlines here
i := 0
if r.lastByteNewline {
for i < len(b) && b[i] == Newline {
i++
}
b = b[i:]
}
r.lastByteNewline = b[len(b) - 1] == Newline
i = bytes.IndexByte(b, Newline)
if i == -1 {
// no newlines - just write the entire thing
return r.w.Write(b)
}
// write up to the newline
i++
n, err := r.w.Write(b[:i])
if err != nil {
return n, err
}
// skip over immediate newline and recurse
i++
for i < len(b) && b[i] == Newline {
i++
}
i--
m, err := r.Write(b[i:])
return n + m, nil
}
func main() {
r := ReduceNewlinesWriter{
w: os.Stdout,
}
io.WriteString(&r, "this\n\n\n\n\n\n\nhas\nmultiple\n\n\nnewline\n\n\n\ncharacters")
}
The case is :
I want read the log like "tail -f" *NIX
when I kill the program I can know how many bytes I have already read,and I can use the seek
when the program start again,will continue to read the log line by line depend by seek data in step 2
I want get the bytes when I use bufio.NewScanner as a line reader to read a line
eg:
import ...
func main() {
f, err := os.Open("111.txt")
if err != nil {
log.Fatal(err)
}
f.Seek(0,os.SEEK_SET)
scan := bufio.NewScanner(f)
for scan.Scan() {
log.Printf(scan.Text())
//what I want is how many bytes at this time when I read a line
}//This is a program for read line
}
thx!
==================================update==========================================
#twotwotwo this is close to what I want,but I want change the io.Reader to the io.ReaderAt, and it is what I want,I write a demo use the io.Reader:`
import (
"os"
"log"
"io"
)
type Reader struct {
reader io.Reader
count int
}
func (r *Reader) Read(b []byte) (int, error) {
n, err := r.reader.Read(b)
r.count += n
return n, err
}
func (r *Reader) Count() int {
return r.count
}
func NewReader(r io.Reader) *Reader {
return &Reader{reader: r}
}
func ReadLine(r *Reader) (ln int,line []byte,err error) {
line = make([]byte,0,4096)
for {
b := make([]byte,1)
n,er := r.Read(b)
if er == io.EOF {
err = er
break
}
if n > 0{
c := b[0]
if c == '\n' {
break
}
line = append(line, c)
}
if er != nil{
err = er
}
}
ln = r.Count()
return ln,line,err
}
func main() {
f, err := os.Open("111.txt")
if err != nil {
log.Fatal(err)
}
fi,_:=os.Stat("111.txt")
log.Printf("the file have %v bytes",fi.Size())
co := NewReader(f)
for {
count,line,er := ReadLine(co)
if er == io.EOF {
break
}
log.Printf("now read the line :%v",string(line))
log.Printf("in all we have read %v bytes",count)
}
}`
this Program can tell me how many bytes I have already read,but cannt read start from anywhere where I want,so I think that if we use io.ReaderAt must can do it.
thanks again!
You could consider another approach based on os.File.
See ActiveState/tail, which monitor the state of a file, and uses os.File#Seek() to resume tailing a file from within a certain point.
See tail.go.
Consider composition.
We know that bufio.NewScanner is interacting with its input through the io.Reader interface. So we may wrap an io.Reader with something else that counts how many bytes have been read so far.
package main
import (
"bufio"
"bytes"
"io"
"log"
)
type ReadCounter struct {
io.Reader
BytesRead int
}
func (r *ReadCounter) Read(p []byte) (int, error) {
n, err := r.Reader.Read(p)
r.BytesRead += n
return n, err
}
func main() {
b := &ReadCounter{Reader: bytes.NewBufferString("hello\nworld\testing\n")}
scan := bufio.NewScanner(b)
for scan.Scan() {
log.Println(scan.Text())
log.Println("Read", b.BytesRead, "bytes so far")
}
}
But we'll note that bufio.NewScanner is buffered, so we can see that it reads its input in chunks. So for your purposes, this might not be as useful as you want.
An alternative is to take the content of scan.Text() and count up the lengths. You can compensate for its removal of newline bytes in your internal count.
I am a Go beginner, coming from Ruby land.
In Ruby, you could do something like this.
Time.send("now") is equivalent to Time.now, as you are sending the message now to the object Time
Is there something similar in golang?
There is no built in way of calling an arbitrary function from a string in Go.
You can create something similar by registering functions to a map[string].
A working example:
package main
import "fmt"
var m = map[string]func(){
"now": func() { fmt.Println("The time is now") },
"then": func() { fmt.Println("Once upon a time") },
}
func main() {
cmd := "then"
m[cmd]()
}
play.golang.org
There is also the possibility of using reflection in order to call a method by name. You can look at the reflect package for MethodByName and Call. You can also check this Stackoverflow question.
As other suggested, you can do it yourself by mapping strings to functions, but the strong-typing nature of Go makes it difficult to translate .send directly into Go.
You can still use reflection if you really need to access a field or method by name:
import "reflect"
import "fmt"
type A struct {
Number int
}
func (a *A) Method(i int) int {
return a.Number + i;
}
func main() {
a := &A{Number: 1}
// Direct access
fmt.Printf("Direct -> Nb: %d, Nb + 2: %d\n", a.Number, a.Method(2));
v := reflect.ValueOf(*a)
vp := reflect.ValueOf(a)
field := v.FieldByName("Number")
meth := vp.MethodByName("Method")
args := []reflect.Value{reflect.ValueOf(2)}
// Reflection access
fmt.Printf("Reflect -> Nb: %d, Nb + 2: %d\n",
field.Interface().(int),
meth.Call(args)[0].Interface().(int))
}
Outputs:
Direct -> Nb: 1, Nb + 2: 3
Reflect -> Nb: 1, Nb + 2: 3
play.golang.org
Note however:
How cumbersome that is. Usually, performing a map as suggested by #ANisus is a more idiomatic way of doing
You still have to perform your conversions in the end.
Using the reflect packages changes your typed variable into more flexible Value objects, but these are very cumbersome to use in practice. It is usually better if you can find a way to express your intent without relying on reflection.
Also note that here, we had to use two Values, one for a (a pointer to A) for the method, and one for *a (a A structure) for the field. Trying to get a method defined with a pointer receiver with a non-pointer Value (or conversely, trying to obtain a field via a pointer Value) will result in a panic. More generally, due to the dynamic nature of reflected Values and its difference with the usual typed Go, expect a lot of convenience features (such as automatic referencing/dereferencing) to be absent on Values.
Also, expect quite a bit of runtime panics while debugging, as it is the only way for dynamic Value calls to fail !
Reference: the reflect package
No. Work your way through http://tour.golang.org/ and http://golang.org/doc/effective_go.html and you will have a proper understanding of how method invocation works.
Here is a working example using reflect
package main
import (
"fmt"
"os"
"reflect"
)
// Send sends a message to(calls a method of) obj, with args.
// The return value of the method call is set to ret and any error to err.
func Send(obj interface{}, method string, args ...interface{}) (ret []reflect.Value, err error) {
defer func() {
if e := recover(); e != nil {
err = fmt.Errorf("%v", e)
}
}()
objValue := reflect.ValueOf(obj)
argsValue := make([]reflect.Value, 0, len(args))
for _, arg := range args {
argsValue = append(argsValue, reflect.ValueOf(arg))
}
mtd := objValue.MethodByName(method)
if !mtd.IsValid() {
return nil, fmt.Errorf("%v does not have a method %v", reflect.TypeOf(obj), method)
}
ret = mtd.Call(argsValue)
return
}
// Then do some tests.
type A struct {
value int
}
func (a A) Value() int {
return a.value
}
func (a *A) SetValue(v int) {
a.value = v
}
func main() {
var (
ret []reflect.Value
err error
)
// StdOut.WriteString("Hello, World!\n")
_, err = Send(os.Stdout, "WriteString", "Hello, World!\n")
handleError(err)
var a = &A{100}
// ret = a.Value()
ret, err = Send(a, "Value")
handleError(err)
fmt.Printf("Return value is: %v\n", ret[0].Int())
// a.SetValue(200)
_, err = Send(a, "SetValue", 200)
handleError(err)
// ret = a.Value()
ret, err = Send(a, "Value")
handleError(err)
fmt.Printf("Return value is: %v", ret[0].Int())
}
func handleError(err error) {
if err != nil {
panic(err)
}
}
I based my code on this description of send.
class Klass
def hello(*args)
"Hello " + args.join(' ')
end
end
k = Klass.new
k.send :hello, "gentle", "readers" #=> "Hello gentle readers"
http://play.golang.org/p/lXlzBf_fGZ
package main
import "strings"
type Klass struct{}
func (k Klass) Hello(args ...string) string {
return "Hello " + strings.Join(args, " ")
}
func (k Klass) Send(symbol func(Klass, ...string) string, args ...string) string {
return symbol(k, args...)
}
func main() {
k := new(Klass)
k.Send(Klass.Hello, "gentle", "readers") //=> "Hello gentle readers"
}
The big difference between the two is that Go's Send function is only implemented for Klass and only works on methods that take a variable number of strings as parameters and return a single string. This is because Go is a statically typed language where Ruby is dynamically typed. Go does support dynamic typing via the reflect library, but it is an unpleasant experience and not the way general Go code is meant to be written.