Create an io.ReaderAt from io.Reader - go

Is there an implementation of io.ReaderAt that can be created from an implementation of io.Reader without first being read into a []byte or string?

Something like the below. Note bytes.Reader implements the ReadAt(...) method/function: https://golang.org/pkg/bytes/#Reader.ReadAt. So the line bytes.NewReader is esssentially what you are looking for.
Getting a bytes.Reader:
var ioReader io.Reader
...
buff := bytes.NewBuffer([]byte{})
size, err := io.Copy(buff, ioReader)
if err != nil {
return err
}
reader := bytes.NewReader(buff.Bytes())
// Do something with `reader`

Yes, this is possible. As mentioned in my comment above, the implementation is limited in that you cannot seek backward nor can you re-read a section that has already been read.
Here is a example implementation:
type unbufferedReaderAt struct {
R io.Reader
N int64
}
func NewUnbufferedReaderAt(r io.Reader) io.ReaderAt {
return &unbufferedReaderAt{R: r}
}
func (u *unbufferedReaderAt) ReadAt(p []byte, off int64) (n int, err error) {
if off < u.N {
return 0, errors.New("invalid offset")
}
diff := off - u.N
written, err := io.CopyN(ioutil.Discard, u.R, diff)
u.N += written
if err != nil {
return 0, err
}
n, err = u.R.Read(p)
u.N += int64(n)
return
}
Example usage:
s := strings.NewReader("hello world")
var b [5]byte
ura := NewUnbufferedReaderAt(s)
if _, err := ura.ReadAt(b[:], 0); err != nil {
panic(err)
}
fmt.Printf("%s\n", b[:]) // prints "hello"
/*
if _, err := ura.ReadAt(b[:], 0); err != nil {
panic(err) // panics
}
fmt.Printf("%s\n", b[:])
*/
if _, err := ura.ReadAt(b[:], 6); err != nil {
panic(err)
}
fmt.Printf("%s\n", b[:]) // prints "world"

Related

Golang unexpected EOF

Here's my code, I'm new to Go.
I tried googling the issue, but I can't quite put my finger on it.
I think it has something to do with the Read() method.
package main
import (
...
)
type compressor struct {
content []byte
}
func (r *compressor) compress() []byte {
...
}
func (r *compressor) decompress() []byte {
var buffer bytes.Buffer
dc := flate.NewReader(&buffer)
_, err := dc.Read(r.content)
if err != nil {
if err != io.EOF {
log.Fatal(err)
}
}
return buffer.Bytes()
}
func main() {
fileName := os.Args[1]
fmt.Println(os.Args)
contents, err := ioutil.ReadFile(fileName)
if err != nil {
log.Fatal(err)
}
fmt.Print("Uncompressed data: ")
fmt.Println(len(contents))
comp := compressor{contents}
buffer := comp.decompress()
fmt.Print("Uncompressed data: ")
fmt.Println(len(comp.decompress()))
err = ioutil.WriteFile(fileName+".decjc", buffer, 0644)
if err != nil {
log.Fatal(err)
}
}
Here's the output
dylan#skynet:~/Documents/EXP/jc$ ./jc data.txt.jc
[./jc data.txt.jc]
Uncompressed data: 2364480
2018/06/29 21:41:35 unexpected EOF
After doing a trace on the particular code in question I have come to the following answer.
/src/bytes/reader.go 70
func (r *Reader) ReadByte() (byte, error) {
...
if r.i >= int64(len(r.s)) {
return 0, io.EOF
}
....
}
There are four functions in bytes/reader that can return io.EOF, and zero functions that can return io.ErrUnexpectedEOF. The four functions that can return io.EOF are:
Read(b []byte)
ReadAt(b []byte, off int64)
ReadByte()
ReadRune()
/src/compress/flate/inflate.go 698
func (f *decompressor) moreBits() error {
c, err := f.r.ReadByte()
if err != nil {
return noEOF(err)
}
...
}
Of the four functions that can return io.EOF, only one function in flate/inflate.go calls any of them: moreBits() calls ReadByte()
/src/compress/flate/inflate.go 690
func noEOF(e error) error {
if e == io.EOF {
return io.ErrUnexpectedEOF
}
...
}
When moreBits() receives an error it calls noEOF(), which checks if it had received an io.EOF. If this was the case then io.ErrUnexpectedEOF is returned backed. Everything seems to be working as intended, and it appears that it is the user's responsibility to be on the look out for this particular case. A suggested edit to the code above to handle what appears to be defined behavior is:
func (r *compressor) decompress() []byte {
dc := flate.NewReader(bytes.NewReader(r.content))
defer dc.Close()
rb, err := ioutil.ReadAll(dc)
if err != nil {
if err != io.EOF && err != io.ErrUnexpectedEOF {
log.Fatalf("Err %v\n read %v", err, rb)
}
}
return rb
}
This was checked under go1.12.9
You got the in and outputs mixed up.
flate.NewReader takes the compressed input as an io.Reader and it returns a io.ReadCloser that can be used to get the uncompressed output:
func (r *compressor) decompress() []byte {
dc := flate.NewReader(bytes.NewReader(r.content))
defer dc.Close()
rb, err := ioutil.ReadAll(dc)
if err != nil {
if err != io.EOF {
log.Fatalf("Err %v\n read %v", err, rb)
}
}
return rb
}

Golang function generalization for different packages

Imagine these functions needs to be used how can I make this calls generic so that I don't repeat almost the same code.
with "encoding/csv"
func getDataFromCSVFiles(files []string) (error, Data) {
data := Data{}
for _, file := range files {
f, err := os.Open(file)
if err != nil {
panic(err)
return err, data
}
defer f.Close()
r := charmap.ISO8859_1.NewDecoder().Reader(f)
reader := csv.NewReader(r)
for i := 1;;i++ {
rec, err := reader.Read()
if i == 1 {
//Skipping header
continue
}
if err != nil {
if err == io.EOF {
break
}
//TODO log error line and csv filename
log.Fatal(err)
}
addWorkbook(rec, &data)
}
}
return nil, data
}
and with
import fw "github.com/hduplooy/gofixedwidth" which is almost the same except calling fw.NewReader
func getDataFromPRNFiles(files []string) (error, Data) {
data := Data{}
for _, file := range files {
f, err := os.Open(file)
if err != nil {
panic(err)
return err, data
}
defer f.Close()
r := charmap.ISO8859_1.NewDecoder().Reader(f)
reader := fw.NewReader(r)
for i := 1;;i++ {
rec, err := reader.Read()
if i == 1 {
//Skipping header
continue
}
if err != nil {
if err == io.EOF {
break
}
//TODO log error line and csv filename
log.Fatal(err)
}
addWorkbook(rec, &data)
}
}
return nil, data
}
The only apparent difference is:
reader := csv.NewReader(r)
versus:
reader := fw.NewReader(r)
I'm not sure what fw is but presumably both readers implement a common interface:
type StringSliceReader interface {
Read() ([]string, error)
}
So you could pass the openers (csv.NewReader and fw.NewReader) as function arguments:
func getDataFromFiles(files []string, func(r io.Reader) StringArrayReader) (error, Data) {
//...
}
but you'd need to wrap them in little functions to get around the return types:
func newCSVReader(r io.Reader) StringSliceReader {
return csv.NewReader(r)
}
func newFWReader(r io.Reader) StringSliceReader {
return fw.NewReader(r)
}
Also, defer queues up things to execute when the function exits, not on the next iteration of a loop. So if you do this:
for _, file := range files {
f, err := os.Open(file)
if err != nil {
panic(err)
return err, data
}
defer f.Close()
//...
}
and files has a hundred entries then you'll have a hundred open files before any of them are closed. You probably want to move that loop body to a separate function so that you only have one file open at a time.
Furthermore, error is usually the last return value from a function so you should return data, err to be more idiomatic.
The result could look something like this:
type StringSliceReader interface {
Read() ([]string, error)
}
type NewReader func(r io.Reader) StringSliceReader
func newCSVReader(r io.Reader) StringSliceReader {
return csv.NewReader(r)
}
func newFWReader(r io.Reader) StringSliceReader {
return fw.NewReader(r)
}
func getDataFrom(file string, data *Data, newReader NewReader) error {
f, err := os.Open(file)
if err != nil {
return err
}
defer f.Close()
r := charmap.ISO8859_1.NewDecoder().Reader(f)
reader := newReader(r)
for i := 1; ; i++ {
rec, err := reader.Read()
if i == 1 {
continue
}
if err != nil {
if err == io.EOF {
break
}
log.Fatal(err)
}
addWorkbook(rec, data)
}
return nil
}
func getDataFromFiles(files []string, newReader NewReader) (Data, error) {
data := Data{}
for _, file := range files {
err := getDataFrom(file, newReader, &data)
if err != nil {
panic(err)
return data, err
}
}
return data, nil
}
and you could say getDataFromFiles(files, newCSVReader) to read CSVs or getDataFromFiles(files, newFWReader) to read FW files. If you want to read from something else, you'd just need a NewReader function and something that implements the StringSliceReader interface.
You might want to bury/hide the charmap.ISO8859_1.NewDecoder().Reader(f) stuff inside the NewReader functions to make it easier to read non-Latin-1 encoded files. You could also replace newReader NewReader with a map[string]NewReader in getDataFromFiles and choose the NewReader to use based on the file's extension or other format identifier.

gob encoding to/decoding from *os.File not working

When I use a file pointer f *os.File I get a empty map back
func decode(f *os.File, b map[string]interface{}) error {
err := gob.NewDecoder(f).Decode(&b)
fmt.Printf("%+v\n", b)
return err
}
func encode(f *os.File, b map[string]interface{}) error {
bb := map[string]interface{}{
"X": 1,
"Greeting": "hello",
}
err := gob.NewEncoder(f).Encode(bb)
f.Sync()
//fmt.Println(buf.Bytes())
return err
}
prints map[]
If I replace it with a global buffer pointer buf *bytes.Buffer it works
func decode(f *os.File, b map[string]interface{}) error {
err := gob.NewDecoder(buf).Decode(&b)
fmt.Printf("%+v\n", b)
return err
}
func encode(f *os.File, b map[string]interface{}) error {
bb := map[string]interface{}{
"X": 1,
"Greeting": "hello",
}
err := gob.NewEncoder(buf).Encode(bb)
return err
}
prints map[Greeting:hello X:1]
f is opened in main like this
var (
buf *bytes.Buffer
f *os.File
memcache map[string]interface{}
)
func main() {
var err error
f, err = os.Create("_memcache.txt")
if err != nil {
log.Print(err.Error())
}
defer f.Close()
memcache = make(map[string]interface{})
buf = new(bytes.Buffer)
gob.Register(map[string]interface{}{})
err = write()
if err != nil {
log.Print(err.Error())
}
err = read()
if err != nil {
log.Print(err.Error())
}
}
func read() (err error) {
err = decode(f, memcache)
if err != nil && err != io.EOF {
return err
}
}
func write() (err error) {
err = encode(f, memcache)
if err != nil {
return err
}
}
The solution is already mentioned in the comments, so the update method main() is following:
func main() {
var err error
f, err = os.Create("_memcache.txt")
if err != nil {
log.Print(err.Error())
}
defer f.Close()
memcache = make(map[string]interface{})
buf = new(bytes.Buffer)
gob.Register(map[string]interface{}{})
err = write()
if err != nil {
log.Print(err.Error())
}
// after write the file pointer is at the end of the file,
// so it should be moved:
f.Seek(0, 0)
// alternatively, open the file in read()
err = read()
if err != nil {
log.Print(err.Error())
}
}

Reading CSV file in Go

Here is a code snippet that reads CSV file:
func parseLocation(file string) (map[string]Point, error) {
f, err := os.Open(file)
defer f.Close()
if err != nil {
return nil, err
}
lines, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, err
}
locations := make(map[string]Point)
for _, line := range lines {
name := line[0]
lat, laterr := strconv.ParseFloat(line[1], 64)
if laterr != nil {
return nil, laterr
}
lon, lonerr := strconv.ParseFloat(line[2], 64)
if lonerr != nil {
return nil, lonerr
}
locations[name] = Point{lat, lon}
}
return locations, nil
}
Is there a way to improve readability of this code? if and nil noise.
Go now has a csv package for this. Its is encoding/csv. You can find the docs here: https://golang.org/pkg/encoding/csv/
There are a couple of good examples in the docs. Here is a helper method I created to read a csv file and returns its records.
package main
import (
"encoding/csv"
"fmt"
"log"
"os"
)
func readCsvFile(filePath string) [][]string {
f, err := os.Open(filePath)
if err != nil {
log.Fatal("Unable to read input file " + filePath, err)
}
defer f.Close()
csvReader := csv.NewReader(f)
records, err := csvReader.ReadAll()
if err != nil {
log.Fatal("Unable to parse file as CSV for " + filePath, err)
}
return records
}
func main() {
records := readCsvFile("../tasks.csv")
fmt.Println(records)
}
Go is a very verbose language, however you could use something like this:
// predeclare err
func parseLocation(file string) (locations map[string]*Point, err error) {
f, err := os.Open(file)
if err != nil {
return nil, err
}
defer f.Close() // this needs to be after the err check
lines, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, err
}
//already defined in declaration, no need for :=
locations = make(map[string]*Point, len(lines))
var lat, lon float64 //predeclare lat, lon
for _, line := range lines {
// shorter, cleaner and since we already have lat and err declared, we can do this.
if lat, err = strconv.ParseFloat(line[1], 64); err != nil {
return nil, err
}
if lon, err = strconv.ParseFloat(line[2], 64); err != nil {
return nil, err
}
locations[line[0]] = &Point{lat, lon}
}
return locations, nil
}
//edit
A more efficient and proper version was posted by #Dustin in the comments, I'm adding it here for completeness sake:
func parseLocation(file string) (map[string]*Point, error) {
f, err := os.Open(file)
if err != nil {
return nil, err
}
defer f.Close()
csvr := csv.NewReader(f)
locations := map[string]*Point{}
for {
row, err := csvr.Read()
if err != nil {
if err == io.EOF {
err = nil
}
return locations, err
}
p := &Point{}
if p.lat, err = strconv.ParseFloat(row[1], 64); err != nil {
return nil, err
}
if p.lon, err = strconv.ParseFloat(row[2], 64); err != nil {
return nil, err
}
locations[row[0]] = p
}
}
playground
I basically copied my answer from here: https://www.dotnetperls.com/csv-go. For me, this was a better answer than what I found on stackoverflow.
import (
"bufio"
"encoding/csv"
"os"
"fmt"
"io"
)
func ReadCsvFile(filePath string) {
// Load a csv file.
f, _ := os.Open(filePath)
// Create a new reader.
r := csv.NewReader(f)
for {
record, err := r.Read()
// Stop at EOF.
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
// Display record.
// ... Display record length.
// ... Display all individual elements of the slice.
fmt.Println(record)
fmt.Println(len(record))
for value := range record {
fmt.Printf(" %v\n", record[value])
}
}
}
I also dislike the verbosity of the default Reader, so I made a new type that is
similar to bufio#Scanner:
package main
import "encoding/csv"
import "io"
type Scanner struct {
Reader *csv.Reader
Head map[string]int
Row []string
}
func NewScanner(o io.Reader) Scanner {
csv_o := csv.NewReader(o)
a, e := csv_o.Read()
if e != nil {
return Scanner{}
}
m := map[string]int{}
for n, s := range a {
m[s] = n
}
return Scanner{Reader: csv_o, Head: m}
}
func (o *Scanner) Scan() bool {
a, e := o.Reader.Read()
o.Row = a
return e == nil
}
func (o Scanner) Text(s string) string {
return o.Row[o.Head[s]]
}
Example:
package main
import "strings"
func main() {
s := `Month,Day
January,Sunday
February,Monday`
o := NewScanner(strings.NewReader(s))
for o.Scan() {
println(o.Text("Month"), o.Text("Day"))
}
}
https://golang.org/pkg/encoding/csv
You can also read contents of a directory to load all the CSV files. And then read all those CSV files 1 by 1 with goroutines
csv file:
101,300.00,11000901,1155686400
102,250.99,11000902,1432339200
main.go file:
const sourcePath string = "./source"
func main() {
dir, _ := os.Open(sourcePath)
files, _ := dir.Readdir(-1)
for _, file := range files {
fmt.Println("SINGLE FILE: ")
fmt.Println(file.Name())
filePath := sourcePath + "/" + file.Name()
f, _ := os.Open(filePath)
defer f.Close()
// os.Remove(filePath)
//func
go func(file io.Reader) {
records, _ := csv.NewReader(file).ReadAll()
for _, row := range records {
fmt.Println(row)
}
}(f)
time.Sleep(10 * time.Millisecond)// give some time to GO routines for execute
}
}
And the OUTPUT will be:
$ go run main.go
SINGLE FILE:
batch01.csv
[101 300.00 11000901 1155686400]
[102 250.99 11000902 1432339200]
----------------- -------------- ---------------------- -------
---------------- ------------------- ----------- --------------
Below example with the Invoice struct
func main() {
dir, _ := os.Open(sourcePath)
files, _ := dir.Readdir(-1)
for _, file := range files {
fmt.Println("SINGLE FILE: ")
fmt.Println(file.Name())
filePath := sourcePath + "/" + file.Name()
f, _ := os.Open(filePath)
defer f.Close()
go func(file io.Reader) {
records, _ := csv.NewReader(file).ReadAll()
for _, row := range records {
invoice := new(Invoice)
invoice.InvoiceNumber = row[0]
invoice.Amount, _ = strconv.ParseFloat(row[1], 64)
invoice.OrderID, _ = strconv.Atoi(row[2])
unixTime, _ := strconv.ParseInt(row[3], 10, 64)
invoice.Date = time.Unix(unixTime, 0)
fmt.Printf("Received invoice `%v` for $ %.2f \n", invoice.InvoiceNumber, invoice.Amount)
}
}(f)
time.Sleep(10 * time.Millisecond)
}
}
type Invoice struct {
InvoiceNumber string
Amount float64
OrderID int
Date time.Time
}

pass interface pointer and assignment value

I want to write a file cache in Go. I am using gob encoding, and saving to a file, but my get function has some problem:
package main
import (
"encoding/gob"
"fmt"
"os"
)
var (
file = "tmp.txt"
)
type Data struct {
Expire int64
D interface{}
}
type User struct {
Id int
Name string
}
func main() {
user := User{
Id: 1,
Name: "lei",
}
err := set(file, user, 10)
if err != nil {
fmt.Println(err)
return
}
user = User{}
err = get(file, &user)
if err != nil {
fmt.Println(err)
return
}
//user not change.
fmt.Println(user)
}
func set(file string, v interface{}, expire int64) error {
f, err := os.OpenFile(file, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600)
if err != nil {
return err
}
defer f.Close()
//wrapper data
//save v in data.D
data := Data{
Expire: expire,
D: v,
}
gob.Register(v)
enc := gob.NewEncoder(f)
err = enc.Encode(data)
if err != nil {
return err
}
return nil
}
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
//get v
v = data.D
fmt.Println(v)
return nil
}
The get function passes interface type and I want to change the value, but not change.
http://play.golang.org/p/wV7rBH028o
In order to insert an unknown value into v of type interface{}, you need to use reflection. This is somewhat involved, but if you want to support this in full, you can see how its done by walking through the decoding process in some of the encoding packages (json, gob).
To get you started, here's a basic version of your get function using reflection. This skips a number of checks, and will only decode something that was encoded as a pointer.
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
panic("need a non nil pointer")
}
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
dv := reflect.ValueOf(data.D)
if dv.Kind() != reflect.Ptr {
panic("didn't decode a pointer")
}
rv.Elem().Set(dv.Elem())
return nil
}
I would actually suggest an easier way to handle this in your own code, which is to have the Get function return an interface{}. Since you will know what the possible types are at that point, you can use a type switch to assert the correct value.
An alternative approach is to return directly the value from the file:
func get(file string) (interface{}, error) {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return nil,err
}
fmt.Println(data.D)
return data.D,nil
}
full working example: http://play.golang.org/p/178U_LVC5y

Resources