When the buffered io writer is used, and some error occur, how can I perform the retry?
For example, I've written 4096B using Write() and an error occur when the bufwriter automatically flushes the data. Then I want to retry writing the 4096B, how I can do it?
It seems I must keep a 4096B buffer myself to perform the retrying. Othersize I'm not able to get the data failed to be flushed.
Any suggestions?
You'll have to use a custom io.Writer that keeps a copy of all data, so that it can be re-used in case of a retry.
This functionality is not part of the standard library, but shouldn't be hard to implement yourself.
When bufio.Writer fails on a Write(..) it will return the amount of bytes written (n) to the buffer the reason why (err).
What you could do is the following. (Note I haven't yet tried this so it may be a little wrong and could use some cleaning up)
func writeSomething(data []byte, w *bufio.Writer) (err error) {
var pos, written int = 0
for pos != len(data) {
written, err = w.Write(data[pos:])
if err != nil {
if err == io.ErrShortWrite {
pos += written // Write was shot. Update pos and keep going
continue
} else netErr, ok := err.(net.Error); ok && netErr.Temporary() {
continue // Temporary error, don't update pos so it will try writing again
} else {
break // Unrecoverable error, bail
}
} else {
pos += written
}
}
return nil
}
Related
-edit-
This was not caused by typos and it is reproducible, contrary to the designation given by stackoverflow. The code provided was written specifically to be self-contained examples of this issue (the issue is right there in the code, all you have to do is run it to reproduce the issues...)
Furthermore, this is the exact question I had and the exact solution I needed that I did not find on StackOverflow prior to asking the question. The answer was derived from a conversation on IRC and I have forwarded the solution here for others like myself who may experience this same problem. This is very much a relevant Q/A and certainly can help future readers.
Thank you
-edit-
I am trying to copy data from an io.Reader implementation to an io.Writer implementation with a delay (time.Sleep) before the next iteration. Ideally I want to control the process (i.e. io.Copy is not ideal because I may want to perform some action between the Read and Write).
In any case, there are 4 approaches attempted in the code below, it should run in go.dev/play. I was unable to get any of the approaches to Write anything but an empty string, though all of the approaches to Write did report the correct number of written bytes (the same as reported by Read). I am probably missing something fundamental, but any help/explanation is greatly appreciated as I am quite confused.
The following code contains comments and functions for 4 different approaches to copying and logging data from custom implementations of io.Reader to an io.Writer with a specified delay (time.Sleep) before the next iteration. You may comment/uncomment the desired function below to observe the results.
package main
import (
"bufio"
"fmt"
"io"
"log"
"math/rand"
"time"
)
// READ
type MyReader struct{}
func (r *MyReader) Read(p []byte) (read int, err error) {
someString := fmt.Sprintf("testing-%d", rand.Int())
p = []byte(someString)
read = len(p)
log.Printf("io read %d bytes: %s\n", read, string(p))
return read, io.EOF
}
// WRITE
type MyWriter struct{}
func (w *MyWriter) Write(p []byte) (wrote int, err error) {
wrote = len(p)
log.Printf("io wrote %d bytes: %s\n", wrote, string(p))
return
}
// MAIN
func main() {
// The following contains comments and functions for 4 different approaches to copying and logging data from
// custom implementations of io.Reader to an io.Writer with a specified delay (time.Sleep) before the next
// iteration. You may comment/uncomment the desired function below to observe the results.
// AutoIoCopy - Approach 1) io.Copy
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called and the MyWriter.Write method of the io.Writer implementation
// is executed, but the output logged by MyWriter.Write is empty instead of the expected string reported by the
// MyReader.Read log.
//
AutoIoCopy()
// ReadBytesApproach - Approach 2) Using ReadBytes('\n')
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called and reports written bytes, but the Write method of MyWriter
// io.Writer implementation is never executed, it is skipped and another Read occurs
//
//ReadBytesApproach()
// ReadLineApproach - Approach 3) Using ReadLine()
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called and reports written bytes, but the Write method of MyWriter
// io.Writer implementation is never executed, it is skipped and another Read occurs
//
//ReadLineApproach()
// WriteToApproach - Approach 4) Using WriteTo()
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called and reports written bytes, but the Write method of MyWriter
// io.Writer implementation is never executed, it is skipped and another Read occurs
//
//WriteToApproach()
}
// Approaches:
// AutoIoCopy - Approach 1) io.Copy
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called and the MyWriter.Write method of the io.Writer implementation
// is executed, but the output logged by MyWriter.Write is empty instead of the expected string reported by the
// MyReader.Read log.
//
func AutoIoCopy() {
reader := MyReader{}
writer := MyWriter{}
for {
_, _ = io.Copy(&writer, &reader)
time.Sleep(1000 * time.Millisecond)
}
}
// ReadBytesApproach - Approach 2) Using ReadBytes('\n')
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called but the Write method of MyWriter io.Writer implementation
// is never executed, it is skipped and another Read occurs
//
func ReadBytesApproach() {
reader := MyReader{}
writer := MyWriter{}
bufRead := bufio.NewReader(&reader)
bufWrite := bufio.NewWriter(&writer)
for {
// Using ReadBytes('\n')
readBytes, err := bufRead.ReadBytes('\n')
if err != nil {
switch err {
case io.EOF:
log.Printf("io.EOF detected\n")
wrote, err := bufWrite.Write(readBytes)
if err != nil {
log.Printf("error writing: %s\n", err)
}
convertedValue := string(readBytes)
log.Printf("bufio wrote %d bytes: %s\n", wrote, convertedValue)
break
default:
log.Printf("bufio error reading: %s\n", err.Error())
break
}
} else {
log.Printf("no error, continue to read\n")
}
time.Sleep(1000 * time.Millisecond)
}
}
// ReadLineApproach - Approach 3) Using ReadLine()
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that the bufio.Write method is called but the Write method of MyWriter io.Writer implementation
// is never executed, it is skipped and another Read occurs
//
func ReadLineApproach() {
reader := MyReader{}
writer := MyWriter{}
bufRead := bufio.NewReader(&reader)
bufWrite := bufio.NewWriter(&writer)
for {
// Using ReadLine()
readBytes, _, err := bufRead.ReadLine()
if err != nil {
switch err {
case io.EOF:
log.Printf("io.EOF detected\n")
wrote, err := bufWrite.Write(readBytes)
if err != nil {
log.Printf("error writing: %s\n", err)
}
convertedValue := string(readBytes)
log.Printf("bufio wrote %d bytes: %s\n", wrote, convertedValue)
break
default:
log.Printf("bufio error reading: %s\n", err.Error())
break
}
} else {
log.Printf("no error, continue to read\n")
}
time.Sleep(1000 * time.Millisecond)
}
}
// WriteToApproach - Approach 4) Using WriteTo()
//
// Expected is to read and log the correct value (generated in the method)
// then write the value (output another log)
//
// Actual is that neither the bufio.Write or the Write method of MyWriter io.Writer implementation is executed,
// it is skipped and another Read occurs
//
func WriteToApproach() {
reader := MyReader{}
writer := MyWriter{}
bufRead := bufio.NewReader(&reader)
bufWrite := bufio.NewWriter(&writer)
for {
wrote, _ := bufRead.WriteTo(bufWrite)
log.Printf("bufio wrote %d bytes\n", wrote)
time.Sleep(1000 * time.Millisecond)
}
}
The issue was in the MyReader.Read method
The line
p = []byte(someString)
Should be
read = copy(p, someString)
According to the go documentation, the net.Conn.Write() function will return an error, if it could not send all bytes in the given slice.
How do I know which type of error is returned in this case? Should I just check if there is an error, and if n > 0 and n < len(p) ? Or is it enough to just check if n < len(p) alone? Are there unrecoverable errors for which n < len(p) ?
Say I want to send X gigabytes of data for example. What would a "correct" Write() loop look like?
If the output buffers are full, will Write() simply just block until it has sent everything from p? making a check for n < len(p) superfluous?
Well, net.Conn is an interface. This means that it is completely up to the implementation to determine which errors to send back. A TCP connection and UNIX socket connection can have very different reasons why a write can't be fully completed.
The net.Conn.Write signature is exactly the same as the io.Writer signature. Which means that every implementation of net.Conn also implements io.Writer. So you can use any existing method like io.Copy or io.CopyN to write data to a connection.
How do I know which type of error is returned in this case? Should I just check if there is an error, and if n > 0 and n < len(p) ?
Use n < len(p) to detect the case where the write stopped early. The error is not nil in this case.
Are there unrecoverable errors for which n < len(p) ?
Yes. A network connection can fail after some data is written.
There are also recoverable errors. If write fails because the write deadline is exceeded, a later write with a new deadline can succeed.
Say I want to send X gigabytes of data for example. What would a "correct" Write() loop look like?
If you are asking how to write a []byte to a connection, then a loop is not needed in most cases. The Write call blocks until the data is written or an error occurs. Use this code:
_, err := c.Write(p)
If you are asking how to copy an io.Reader to a network connection, the use _, err := io.Copy(conn, r).
Write is different from Read. Read can return before filling the buffer.
If the output buffers are full, will Write() simply just block until it has sent everything from p?
Write blocks until all data is sent or the write fails with an error (deadline exceeded, network failure, network connection closed in other goroutine, ...).
The net.Conn.Write() is to implement the io.Writer interface, which has the following contract regarding errors:
Write must return a non-nil error if it returns n < len(p).
There is no single correct write loop. For certain cases, it might be important to know how much data was written. However, for network connections, based on this contract, the following should work:
var err error
for data, done := getNextSegment(); !done&&err==nil; data, done = getNextSegment() {
_,err=conn.Write(data)
}
To keep the total number of bytes written:
var err error
written:=0
for data, done := getNextSegment(); !done&&err==nil; data, done = getNextSegment() {
n,err=conn.Write(data)
written+=n
}
So, I went digging into the Go source code itself. I tracked the Write() call to a file named src/internal/poll/fd_unix.go.
// Write implements io.Writer.
func (fd *FD) Write(p []byte) (int, error) {
if err := fd.writeLock(); err != nil {
return 0, err
}
defer fd.writeUnlock()
if err := fd.pd.prepareWrite(fd.isFile); err != nil {
return 0, err
}
var nn int
for {
max := len(p)
if fd.IsStream && max-nn > maxRW {
max = nn + maxRW
}
n, err := ignoringEINTRIO(syscall.Write, fd.Sysfd, p[nn:max])
if n > 0 {
nn += n
}
if nn == len(p) {
return nn, err
}
if err == syscall.EAGAIN && fd.pd.pollable() {
if err = fd.pd.waitWrite(fd.isFile); err == nil {
continue
}
}
if err != nil {
return nn, err
}
if n == 0 {
return nn, io.ErrUnexpectedEOF
}
}
}
This seems to handle the retransmits already. So, Write() does actually guarantee that either everything is sent, or a fatal error occurs, which is unrecoverable.
It seems to me, that there is no need at all, to care about the value of n, other than for logging purposes. If an error ever occurs, it is severe enough that there is no reason to try and retransmit the remaining len(p)-n bytes.
I have a goroutine that is constantly blocked reading the stdin, like this:
func routine() {
for {
data := make([]byte, 8)
os.Stdin.Read(data);
otherChannel <-data
}
}
The routine waits to read 8 bytes via stdin and feeds another channel.
I want to gracefully stop this goroutine from the main thread. However, since the goroutine will almost always be blocked reading from stdin, I can't find a good solution to force it to stop. I thought about something like:
func routine(stopChannel chan struct{}) {
for {
select {
case <-stopChannel:
return
default:
data := make([]byte, 8)
os.Stdin.Read(data);
otherChannel <-data
}
}
}
However, the problem is that if there is no more input in the stdin when the stopChannel is closed, the goroutine will stay blocked and not return.
Is there a good approach to make it return immediately when the main thread wants?
Thanks.
To detect that os.Stdin has been closed : check the error value returned by os.Stdin.Read().
One extra point : although you state that in your case you will always receive 8 bytes chunks, you should still check that you indeed received 8 bytes of data.
func routine() {
for {
data := make([]byte, 8)
n, err := os.Stdin.Read(data)
// error handling : the basic thing to do is "on error, return"
if err != nil {
// if os.Stdin got closed, .Read() will return 'io.EOF'
if err == io.EOF {
log.Printf("stdin closed, exiting")
} else {
log.Printf("stdin: %s", err)
}
return
}
// check that 'n' is big enough :
if n != 8 {
log.Printf("short read: only %d byte. exiting", n)
return // instead of returning, you may want to keep '.Read()'ing
// or you may use 'io.ReadFull(os.Stdin, data)' instead of '.Read()'
}
// a habit to have : truncate your read buffers to 'n' after a .Read()
otherChannel <-data[:n]
}
}
I have a golang app which needs to listen for input on stdin - not as a command line utility but to keep running and listening. The following code, slightly edited down, works but has very high CPU load when 'idle' - and I am not sure why - nor clear how this could be done better. So I need the same functionality without the CPU load! (this is part of an authentication handler for ejabberd)
bioIn := bufio.NewReader(os.Stdin)
bioOut := bufio.NewWriter(os.Stdout)
var err error
var success bool
var length uint16
var result uint16
for {
binary.Read(bioIn, binary.BigEndian, &length)
buf := make([]byte, length)
r, _ := bioIn.Read(buf)
if r == 0 {
continue
}
if err == nil {
data := strings.Split(string(buf), ":")
// I have code to handle the incoming data here...
} else {
success = false
}
length = 2
binary.Write(bioOut, binary.BigEndian, &length)
if success != true {
result = 0
} else {
result = 1
}
binary.Write(bioOut, binary.BigEndian, &result)
bioOut.Flush()
}
ANSWER:
I added a short sleep as suggested and this has worked a charm; didn't need to be long and no noticeable impact on the authentication the service is providing. With the reassurance incoming data is buffered this is perfect fix. Thanks all.
r, _ := bioIn.Read(buf)
if r == 0 {
time.Sleep(25 * time.Millisecond)
continue
}
In this part of your code:
r, _ := bioIn.Read(buf)
if r == 0 {
continue
}
You check for a return value of 0, which means EOF. i.e. the input stream has ended/terminated. Once stdin is terminated, it's not going to come back. So you have an endless loop here. it'll just spin around on this test having bioIn.Read() return 0 every time.
You can't really do much more than quit/exit at that point, stdin doesn't have any more data for you, ever.
Note that your code also discard the error returned from bioIn.Read() - don't do that, deal with errors - and you would have discovered the cause for yourself in this case.
You have a tight infinite loop, which will always take as much CPU as the OS will allow. You are always checking to see if there's any input and (in your sample code) you never wait for new input - you just continuously poll stdin to see if there is any. You should probably use time.Sleep() to wait some time between each check.
I want to write test for function which includes a call to fmt.Scanf(), but I am having problem in passing the required parameter to function.
Is there a better way to do this or I need to mock fmt.Scanf()
Function to be tested is given here:
https://github.com/apsdehal/Konsoole/blob/master/parser.go#L28
// Initializes the network interface by finding all the available devices
// displays them to user and finally selects one of them as per the user
func Init() *pcap.Pcap {
devices, err := pcap.Findalldevs()
if err != nil {
fmt.Fprintf(errWriter, "[-] Error, pcap failed to iniaitilize")
}
if len(devices) == 0 {
fmt.Fprintf(errWriter, "[-] No devices found, quitting!")
os.Exit(1)
}
fmt.Println("Select one of the devices:")
var i int = 1
for _, x := range devices {
fmt.Println(i, x.Name)
i++
}
var index int
fmt.Scanf("%d", &index)
handle, err := pcap.Openlive(devices[index-1].Name, 65535, true, 0)
if err != nil {
fmt.Fprintf(errWriter, "Konsoole: %s\n", err)
errWriter.Flush()
}
return handle
}
It's theoretically possible to change the behavior of Scanf by hotswapping the value of os.Stdin with some other os.File. I wouldn't particularly recommend it just for testing purposes, though.
A better option would just be to make your Init take in an io.Reader that you pass to Fscanf.
Overall, however, it would likely be better to separate your device initialization code from your input as much as possible. This probably means having a device list returning function and a device opening function. You only need to prompt for selection in live/main code.