How to stop io.CopyN - go

I have some code that copies from a file to a tcp socket (like an ftp server) and want to be able to abort this copy if needed.
Im just using io.CopyN(socket, file, size) and cant see a way to signal an abort. Any ideas?

How about just closing the input file? io.CopyN will then return an error and abort.
Here is a demonstration (If not running on Linux change /dev/zero & /dev/null for your OS equivalent!)
package main
import (
"fmt"
"io"
"log"
"os"
"time"
)
func main() {
in, err := os.Open("/dev/zero")
if err != nil {
log.Fatal(err)
}
out, err := os.Create("/dev/null")
if err != nil {
log.Fatal(err)
}
go func() {
time.Sleep(time.Second)
in.Close()
}()
written, err := io.CopyN(out, in, 1E12)
fmt.Printf("%d bytes written with error %s\n", written, err)
}
When run it will print something like
9756147712 bytes written with error read /dev/zero: bad file descriptor

CopyN tries hard to copy N bytes. If you want to optionally copy less than N bytes then don't use CopyN in the first place. I would probably adapt the original code to something like (untested code):
func copyUpToN(dst Writer, src Reader, n int64, signal chan int) (written int64, err error) {
buf := make([]byte, 32*1024)
for written < n {
select {
default:
case <-signal:
return 0, fmt.Errorf("Aborted") // or whatever
}
l := len(buf)
if d := n - written; d < int64(l) {
l = int(d)
}
nr, er := src.Read(buf[0:l])
if nr > 0 {
nw, ew := dst.Write(buf[0:nr])
if nw > 0 {
written += int64(nw)
}
if ew != nil {
err = ew
break
}
if nr != nw {
err = io.ErrShortWrite
break
}
}
if er != nil {
err = er
break
}
}
return written, err
}

Related

Bufio scan function is waiting for sometime for the last buffer (tailed operation) in GO

I am new to golang so please review the code and suggest any changes required.
So the problem statement goes below,
We have a file whose contents are in binary and are encrypted. The only way to read that contents if by using a custom utility say (named decode_it).. The command just accepts filename like below
decode_it filename.d
Now what I have to do is live monitoring the output of the decode_it utility in GO. I have written the code which is working great but somehow it is not able to process the latest tailed output (it is waiting for some amount of time for reading the last latest chunk before more data comes in ). s.Scan() is the function which is not returning the latest changes in output of that utility. I have another terminal side by side so I know that a line is appended or not. The GO Scan() function only scans when another chunk is appended at the end.
Please help. Suggest any changes required and also if possible you can suggest any other alternative approach for this.
Output of utility is - These are huge and come in seconds
1589261318 493023 8=DECODE|9=59|10=053|34=1991|35=0|49=TEST|52=20200512-05:28:38|56=TEST|57=ADMIN|
1589261368 538427 8=DECODE|9=59|10=054|34=1992|35=0|49=TEST|52=20200512-05:29:28|56=TEST|57=ADMIN|
1589261418 579765 8=DECODE|9=59|10=046|34=1993|35=0|49=TEST|52=20200512-05:30:18|56=TEST|57=ADMIN|
1589261468 627052 8=DECODE|9=59|10=047|34=1994|35=0|49=TEST|52=20200512-05:31:08|56=TEST|57=ADMIN|
1589261518 680570 8=DECODE|9=59|10=053|34=1995|35=0|49=TEST|52=20200512-05:31:58|56=TEST|57=ADMIN|
1589261568 722516 8=DECODE|9=59|10=054|34=1996|35=0|49=TEST|52=20200512-05:32:48|56=TEST|57=ADMIN|
1589261618 766070 8=DECODE|9=59|10=055|34=1997|35=0|49=TEST|52=20200512-05:33:38|56=TEST|57=ADMIN|
1589261668 807964 8=DECODE|9=59|10=056|34=1998|35=0|49=TEST|52=20200512-05:34:28|56=TEST|57=ADMIN|
1589261718 853464 8=DECODE|9=59|10=057|34=1999|35=0|49=TEST|52=20200512-05:35:18|56=TEST|57=ADMIN|
1589261768 898758 8=DECODE|9=59|10=031|34=2000|35=0|49=TEST|52=20200512-05:36:08|56=TEST|57=ADMIN|
1589261818 948236 8=DECODE|9=59|10=037|34=2001|35=0|49=TEST|52=20200512-05:36:58|56=TEST|57=ADMIN|
1589261868 995181 8=DECODE|9=59|10=038|34=2002|35=0|49=TEST|52=20200512-05:37:48|56=TEST|57=ADMIN|
1589261918 36727 8=DECODE|9=59|10=039|34=2003|35=0|49=TEST|52=20200512-05:38:38|56=TEST|57=ADMIN|
1589261968 91253 8=DECODE|9=59|10=040|34=2004|35=0|49=TEST|52=20200512-05:39:28|56=TEST|57=ADMIN|
1589262018 129336 8=DECODE|9=59|10=032|34=2005|35=0|49=TEST|52=20200512-05:40:18|56=TEST|57=ADMIN|
1589262068 173247 8=DECODE|9=59|10=033|34=2006|35=0|49=TEST|52=20200512-05:41:08|56=TEST|57=ADMIN|
1589262118 214993 8=DECODE|9=59|10=039|34=2007|35=0|49=TEST|52=20200512-05:41:58|56=TEST|57=ADMIN|
1589262168 256754 8=DECODE|9=59|10=040|34=2008|35=0|49=TEST|52=20200512-05:42:48|56=TEST|57=ADMIN|
1589262218 299908 8=DECODE|9=59|10=041|34=2009|35=0|49=TEST|52=20200512-05:43:38|56=TEST|57=ADMIN|
1589262268 345560 8=DECODE|9=59|10=033|34=2010|35=0|49=TEST|52=20200512-05:44:28|56=TEST|57=ADMIN|
1589262318 392894 8=DECODE|9=59|10=034|34=2011|35=0|49=TEST|52=20200512-05:45:18|56=TEST|57=ADMIN|
1589262368 439936 8=DECODE|9=59|10=035|34=2012|35=0|49=TEST|52=20200512-05:46:08|56=TEST|57=ADMIN|
1589262418 484959 8=DECODE|9=59|10=041|34=2013|35=0|49=TEST|52=20200512-05:46:58|56=TEST|57=ADMIN|
1589262468 531136 8=DECODE|9=59|10=042|34=2014|35=0|49=TEST|52=20200512-05:47:48|56=TEST|57=ADMIN|
1589262518 577190 8=DECODE|9=59|10=043|34=2015|35=0|49=TEST|52=20200512-05:48:38|56=TEST|57=ADMIN|
1589262568 621673 8=DECODE|9=59|10=044|34=2016|35=0|49=TEST|52=20200512-05:49:28|56=TEST|57=ADMIN|
1589262618 661569 8=DECODE|9=59|10=036|34=2017|35=0|49=TEST|52=20200512-05:50:18|56=TEST|57=ADMIN|
1589262668 704912 8=DECODE|9=59|10=037|34=2018|35=0|49=TEST|52=20200512-05:51:08|56=TEST|57=ADMIN|
1589262718 751844 8=DECODE|9=59|10=043|34=2019|35=0|49=TEST|52=20200512-05:51:58|56=TEST|57=ADMIN|
1589262768 792980 8=DECODE|9=59|10=035|34=2020|35=0|49=TEST|52=20200512-05:52:48|56=TEST|57=ADMIN|
1589262818 840365 8=DECODE|9=59|10=036|34=2021|35=0|49=TEST|52=20200512-05:53:38|56=TEST|57=ADMIN|
1589262868 879185 8=DECODE|9=59|10=037|34=2022|35=0|49=TEST|52=20200512-05:54:28|56=TEST|57=ADMIN|
1589262918 925163 8=DECODE|9=59|10=038|34=2023|35=0|49=TEST|52=20200512-05:55:18|56=TEST|57=ADMIN|
1589262968 961584 8=DECODE|9=59|10=039|34=2024|35=0|49=TEST|52=20200512-05:56:08|56=TEST|57=ADMIN|
1589263018 10120 8=DECODE|9=59|10=045|34=2025|35=0|49=TEST|52=20200512-05:56:58|56=TEST|57=ADMIN|
1589263068 53127 8=DECODE|9=59|10=046|34=2026|35=0|49=TEST|52=20200512-05:57:48|56=TEST|57=ADMIN|
1589263118 92960 8=DECODE|9=59|10=047|34=2027|35=0|49=TEST|52=20200512-05:58:38|56=TEST|57=ADMIN|
1589263168 134768 8=DECODE|9=59|10=048|34=2028|35=0|49=TEST|52=20200512-05:59:28|56=TEST|57=ADMIN|
1589263218 180362 8=DECODE|9=59|10=035|34=2029|35=0|49=TEST|52=20200512-06:00:18|56=TEST|57=ADMIN|
1589263268 220070 8=DECODE|9=59|10=027|34=2030|35=0|49=TEST|52=20200512-06:01:08|56=TEST|57=ADMIN|
1589263318 269426 8=DECODE|9=59|10=033|34=2031|35=0|49=TEST|52=20200512-06:01:58|56=TEST|57=ADMIN|
1589263368 309432 8=DECODE|9=59|10=034|34=2032|35=0|49=TEST|52=20200512-06:02:48|56=TEST|57=ADMIN|
1589263418 356561 8=DECODE|9=59|10=035|34=2033|35=0|49=TEST|52=20200512-06:03:38|56=TEST|57=ADMIN|
Code -
package main
import (
"bytes"
"bufio"
"io"
"log"
"os/exec"
"fmt"
)
// dropCRLR drops a terminal \r from the data.
func dropCRLR(data []byte) []byte {
if len(data) > 0 && data[len(data)-1] == '\r' {
return data[0 : len(data)-1]
}
return data
}
func newLineSplitFunc(data []byte, atEOF bool) (advance int, token []byte, err error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
if i := bytes.IndexByte(data, '\n'); i >= 0 {
// We have a full newline-terminated line.
return i + 1, dropCRLR(data[0:i]), nil
}
// If we're at EOF, we have a final, non-terminated line. Return it.
if atEOF {
return len(data), dropCRLR(data), nil
}
// Request more data.
// fmt.Println("Returning 0,nil,nil")
return 0, nil, nil
}
func main() {
cmd := exec.Command("decode_it", "filename.d", "4", "1")
var out io.Reader
{
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
stderr, err := cmd.StderrPipe()
if err != nil {
log.Fatal(err)
}
out = io.MultiReader(stdout, stderr)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
// Make a new channel which will be used to ensure we get all output
done := make(chan struct{})
go func() {
// defer cmd.Process.Kill()
s := bufio.NewScanner(out)
s.Split(newLineSplitFunc)
for s.Scan() {
fmt.Println("---- " + s.Text())
}
if s.Err() != nil {
fmt.Printf("error: %s\n", s.Err())
}
}()
// Wait for all output to be processed
<-done
// Wait for the command to finish
if err := cmd.Wait(); err != nil{
fmt.Println("Error: " + string(err.Error()))
}
// if out closes, cmd closed.
log.Println("all done")
}
Also, Since scan() is taking a lot of time and goes into a loop from which I am not able to break as well. Please help for that too..
try something like this one, i fixed some issues and make it more simple:
package main
import (
"bufio"
"fmt"
"io"
"log"
"os/exec"
)
func main() {
var err error
// change to your command
cmd := exec.Command("sh", "test.sh")
var out io.Reader
{
var stdout, stderr io.ReadCloser
stdout, err = cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
stderr, err = cmd.StderrPipe()
if err != nil {
log.Fatal(err)
}
out = io.MultiReader(stdout, stderr)
}
if err = cmd.Start(); err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(out)
for scanner.Scan() {
fmt.Println("---- " + scanner.Text())
}
if err = scanner.Err(); err != nil {
fmt.Printf("error: %v\n", err)
}
log.Println("all done")
}
test.sh that i used in test:
#!/bin/bash
while [[ 1 = 1 ]]; do
echo 1
sleep 1
done
:)
I tried resolving the above issue using stdbuf -
cmd := exec.Command("stdbuf", "-o0", "-e0", "decode_it", FILEPATH, "4", "1")
Reference link - STDIO Buffering
When programs write to stdout they write with line bufferring. If they are writing to something else, then they use fully buffered mode. golang exec.Command seems to end up using fully buffered mode so using stdbuf forces no buffering.

How do I read in a large flat file

I have a flat file that has 339276 line of text in it for a size of 62.1 MB. I am attempting to read in all the lines, parse them based on some conditions I have and then insert them into a database.
I originally attempted to use a bufio.Scan() loop and bufio.Text() to get the line but I was running out of buffer space. I switched to using bufio.ReadLine/ReadString/ReadByte (I tried each) and had the same problem with each. I didn't have enough buffer space.
I tried using read and setting the buffer size but as the document says it actually a const that can be made smaller but never bigger that 64*1024 bytes. I then tried to use File.ReadAt where I set the starting postilion and moved it along as I brought in each section to no avail. I have looked at the following examples and explanations (not an exhaustive list):
Read text file into string array (and write)
How to Read last lines from a big file with Go every 10 secs
reading file line by line in go
How do I read in an entire file (either line by line or the whole thing at once) into a slice so I can then go do things to the lines?
Here is some code that I have tried:
file, err := os.Open(feedFolder + value)
handleError(err)
defer file.Close()
// fileInfo, _ := file.Stat()
var linesInFile []string
r := bufio.NewReader(file)
for {
path, err := r.ReadLine("\n") // 0x0A separator = newline
linesInFile = append(linesInFile, path)
if err == io.EOF {
fmt.Printf("End Of File: %s", err)
break
} else if err != nil {
handleError(err) // if you return error
}
}
fmt.Println("Last Line: ", linesInFile[len(linesInFile)-1])
Here is something else I tried:
var fileSize int64 = fileInfo.Size()
fmt.Printf("File Size: %d\t", fileSize)
var bufferSize int64 = 1024 * 60
bytes := make([]byte, bufferSize)
var fullFile []byte
var start int64 = 0
var interationCounter int64 = 1
var currentErr error = nil
for currentErr != io.EOF {
_, currentErr = file.ReadAt(bytes, st)
fullFile = append(fullFile, bytes...)
start = (bufferSize * interationCounter) + 1
interationCounter++
}
fmt.Printf("Err: %s\n", currentErr)
fmt.Printf("fullFile Size: %s\n", len(fullFile))
fmt.Printf("Start: %d", start)
var currentLine []string
for _, value := range fullFile {
if string(value) != "\n" {
currentLine = append(currentLine, string(value))
} else {
singleLine := strings.Join(currentLine, "")
linesInFile = append(linesInFile, singleLine)
currentLine = nil
}
}
I am at a loss. Either I don't understand exactly how the buffer works or I don't understand something else. Thanks for reading.
bufio.Scan() and bufio.Text() in a loop perfectly works for me on a files with much larger size, so I suppose you have lines exceeded buffer capacity. Then
check your line ending
and which Go version you use path, err :=r.ReadLine("\n") // 0x0A separator = newline? Looks like func (b *bufio.Reader) ReadLine() (line []byte, isPrefix bool, err error) has return value isPrefix specifically for your use case
http://golang.org/pkg/bufio/#Reader.ReadLine
It's not clear that it's necessary to read in all the lines before parsing them and inserting them into a database. Try to avoid that.
You have a small file: "a flat file that has 339276 line of text in it for a size of 62.1 MB." For example,
package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
)
func readLines(filename string) ([]string, error) {
var lines []string
file, err := ioutil.ReadFile(filename)
if err != nil {
return lines, err
}
buf := bytes.NewBuffer(file)
for {
line, err := buf.ReadString('\n')
if len(line) == 0 {
if err != nil {
if err == io.EOF {
break
}
return lines, err
}
}
lines = append(lines, line)
if err != nil && err != io.EOF {
return lines, err
}
}
return lines, nil
}
func main() {
// a flat file that has 339276 lines of text in it for a size of 62.1 MB
filename := "flat.file"
lines, err := readLines(filename)
fmt.Println(len(lines))
if err != nil {
fmt.Println(err)
return
}
}
It seems to me this variant of readLines is shorter and faster than suggested peterSO
func readLines(filename string) (map[int]string, error) {
lines := make(map[int]string)
data, err := ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
for n, line := range strings.Split(string(data), "\n") {
lines[n] = line
}
return lines, nil
}
package main
import (
"fmt"
"os"
"log"
"bufio"
)
func main() {
FileName := "assets/file.txt"
file, err := os.Open(FileName)
if err != nil {
log.Fatal(err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}

How to read a file line by line and return how many bytes have been read?

The case is :
I want read the log like "tail -f" *NIX
when I kill the program I can know how many bytes I have already read,and I can use the seek
when the program start again,will continue to read the log line by line depend by seek data in step 2
I want get the bytes when I use bufio.NewScanner as a line reader to read a line
eg:
import ...
func main() {
f, err := os.Open("111.txt")
if err != nil {
log.Fatal(err)
}
f.Seek(0,os.SEEK_SET)
scan := bufio.NewScanner(f)
for scan.Scan() {
log.Printf(scan.Text())
//what I want is how many bytes at this time when I read a line
}//This is a program for read line
}
thx!
==================================update==========================================
#twotwotwo this is close to what I want,but I want change the io.Reader to the io.ReaderAt, and it is what I want,I write a demo use the io.Reader:`
import (
"os"
"log"
"io"
)
type Reader struct {
reader io.Reader
count int
}
func (r *Reader) Read(b []byte) (int, error) {
n, err := r.reader.Read(b)
r.count += n
return n, err
}
func (r *Reader) Count() int {
return r.count
}
func NewReader(r io.Reader) *Reader {
return &Reader{reader: r}
}
func ReadLine(r *Reader) (ln int,line []byte,err error) {
line = make([]byte,0,4096)
for {
b := make([]byte,1)
n,er := r.Read(b)
if er == io.EOF {
err = er
break
}
if n > 0{
c := b[0]
if c == '\n' {
break
}
line = append(line, c)
}
if er != nil{
err = er
}
}
ln = r.Count()
return ln,line,err
}
func main() {
f, err := os.Open("111.txt")
if err != nil {
log.Fatal(err)
}
fi,_:=os.Stat("111.txt")
log.Printf("the file have %v bytes",fi.Size())
co := NewReader(f)
for {
count,line,er := ReadLine(co)
if er == io.EOF {
break
}
log.Printf("now read the line :%v",string(line))
log.Printf("in all we have read %v bytes",count)
}
}`
this Program can tell me how many bytes I have already read,but cannt read start from anywhere where I want,so I think that if we use io.ReaderAt must can do it.
thanks again!
You could consider another approach based on os.File.
See ActiveState/tail, which monitor the state of a file, and uses os.File#Seek() to resume tailing a file from within a certain point.
See tail.go.
Consider composition.
We know that bufio.NewScanner is interacting with its input through the io.Reader interface. So we may wrap an io.Reader with something else that counts how many bytes have been read so far.
package main
import (
"bufio"
"bytes"
"io"
"log"
)
type ReadCounter struct {
io.Reader
BytesRead int
}
func (r *ReadCounter) Read(p []byte) (int, error) {
n, err := r.Reader.Read(p)
r.BytesRead += n
return n, err
}
func main() {
b := &ReadCounter{Reader: bytes.NewBufferString("hello\nworld\testing\n")}
scan := bufio.NewScanner(b)
for scan.Scan() {
log.Println(scan.Text())
log.Println("Read", b.BytesRead, "bytes so far")
}
}
But we'll note that bufio.NewScanner is buffered, so we can see that it reads its input in chunks. So for your purposes, this might not be as useful as you want.
An alternative is to take the content of scan.Text() and count up the lengths. You can compensate for its removal of newline bytes in your internal count.

Reading from serial port with while-loop

I’ve written a short program in Go to communicate with a sensor through a serial port:
package main
import (
"fmt"
"github.com/tarm/goserial"
"time"
)
func main() {
c := &serial.Config{Name: "/dev/ttyUSB0", Baud: 9600}
s, err := serial.OpenPort(c)
if err != nil {
fmt.Println(err)
}
_, err = s.Write([]byte("\x16\x02N0C0 G A\x03\x0d\x0a"))
if err != nil {
fmt.Println(err)
}
time.Sleep(time.Second/2)
buf := make([]byte, 40)
n, err := s.Read(buf)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(buf[:n]))
s.Close()
}
It works fine, but after writing to the port I have to wait about half a second before I can start reading from it. I would like to use a while-loop instead of time.Sleep to read all incoming data. My attempt doesn’t work:
buf := make([]byte, 40)
n := 0
for {
n, _ := s.Read(buf)
if n > 0 {
break
}
}
fmt.Println(string(buf[:n]))
I guess buf gets overwritten after every loop pass. Any suggestions?
Your problem is that Read() will return whenever it has some data - it won't wait for all the data. See the io.Reader specification for more info
What you want to do is read until you reach some delimiter. I don't know exactly what format you are trying to use, but it looks like maybe \x0a is the end delimiter.
In which case you would use a bufio.Reader like this
reader := bufio.NewReader(s)
reply, err := reader.ReadBytes('\x0a')
if err != nil {
panic(err)
}
fmt.Println(reply)
Which will read data until the first \x0a.
I guess buf gets overwritten after every loop pass. Any suggestions?
Yes, buf will get overwritten with every call to Read().
A timeout on the file handle would be the approach I would take.
s, _ := os.OpenFile("/dev/ttyS0", syscall.O_RDWR|syscall.O_NOCTTY|syscall.O_NONBLOCK, 0666)
t := syscall.Termios{
Iflag: syscall.IGNPAR,
Cflag: syscall.CS8 | syscall.CREAD | syscall.CLOCAL | syscall.B115200,
Cc: [32]uint8{syscall.VMIN: 0, syscall.VTIME: uint8(20)}, //2.0s timeout
Ispeed: syscall.B115200,
Ospeed: syscall.B115200,
}
// syscall
syscall.Syscall6(syscall.SYS_IOCTL, uintptr(s.Fd()),
uintptr(syscall.TCSETS), uintptr(unsafe.Pointer(&t)),
0, 0, 0)
// Send message
n, _ := s.Write([]byte("Test message"))
// Receive reply
for {
buf := make([]byte, 128)
n, err = s.Read(buf)
if err != nil { // err will equal io.EOF
break
}
fmt.Printf("%v\n", string(buf))
}
Also note, if there is no more data read and there is no error, os.File.Read() will return an error of io.EOF,
as you can see here.

How to calculate checksum of a file efficiently

I want to efficiently calculate the checksum of a very large file (multiple GB). This Go program has two approaches one chunks the file and calculates the checksum quicksha but it's not correct. Another classical approach slowsha works well.
Can you help me fix quicksha?
package main
import (
"bufio"
"crypto/sha256"
"encoding/hex"
"io"
"log"
"net/http"
"net/http/pprof"
"os"
)
func slowsha(fname string) {
f, err := os.Open(fname)
if err != nil {
log.Fatal(err)
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
log.Fatal(err)
}
log.Printf("%s %s", hex.EncodeToString(h.Sum(nil)), os.Args[1])
}
func quicksha(fname string) {
f, err := os.Open(fname)
if err != nil {
log.Fatal(err)
}
defer f.Close()
buf := make([]byte, 16*1024)
pr, pw := io.Pipe()
go func() {
w := bufio.NewWriter(pw)
for {
n, err := f.Read(buf)
if n > 0 {
buf = buf[:n]
w.Write(buf)
}
if err == io.EOF {
pw.Close()
break
}
}
}()
h := sha256.New()
io.Copy(h, pr)
log.Printf("%s %s", hex.EncodeToString(h.Sum(nil)), os.Args[1])
}
func main() {
fname := os.Args[2]
choice := os.Args[1]
for i := 0; i < 100; i++ {
if choice == "-s" {
slowsha(fname)
} else if choice == "-f" {
quicksha(fname)
} else {
log.Fatal("Bad choice")
}
}
}
Output
shasum -a 256 lessthan20MBTest.doc >> reference answer
d91b998a372035c2378fc40a6d0eee17b9f16d60207343f9fc3558eb77f90b71 lessthan20MBTest.doc
./quicksha -f lessthan20MBTest.doc >> wrong answer
b97d5167bbe945ca90223b7503653df89ba9e7d420268da27851fca6db3fcdcf lessthan20MBTest.doc
./quicksha -s lessthan20MBTest.doc . >>> right answer
d91b998a372035c2378fc40a6d0eee17b9f16d60207343f9fc3558eb77f90b71 lessthan20MBTest.doc
There are several problems in your program:
First: you are already using a buffer for reading/writing, so there is no need to use a bufio.Writer. You are double-buffering with that. Which also happens to be the reason why you don't get the result you want: you have to w.Flush() before closing the pipe, because you haven't written what's in the bufio.Writer's buffers to the pipe:
if err == io.EOF {
w.Flush()
pw.Close()
break
}
Second: you are making your buffer shorter. In general, read does not have to read to fill the buffer. If the underlying stream is a network stream, read may read less than the buffer size, and that doesn't mean the end of stream reached. For files, this does not make any difference in practice but in general, you should do:
if n > 0 {
w.Write(buf[:n])
}
Third: Did you measure? It is unlikely that the 'faster' implementation is actually faster. Including the buffering in io.Copy, you're triple-buffering with this implementation.

Resources