Another question How to read/write from/to file using Go? got into safe closing of file descriptors in a comment.
Note that these examples aren't checking the error return from
fo.Close(). From the Linux man pages close(2): Not checking the return
value of close() is a common but nevertheless serious programming
error. It is quite possible that errors on a previous write(2)
operation are first reported at the final close(). Not checking the
return value when closing the file may lead to silent loss of data.
This can especially be observed with NFS and with disk quota. – Nick
Craig-Wood Jan 25 '13 at 7:12
The solution that updated the post used a panic:
// close fo on exit and check for its returned error
defer func() {
if err := fo.Close(); err != nil {
panic(err)
}
}()
I want to hand this error as a value instead of panicking.
If we are afraid of writes not being completed close isn't enough, so updating the error is still not correct.
The correct solution if you want to not hit this is to fsync the file(s):
defer(fd.Close())
// Do stuff
return fd.Sync()
It's easier to read then returning a non-nil modified error either through defer or maintaining throughout the function.
This will be a performance hit, but will catch both close errors for writing to buffers and the physical write to disk.
Related
I am currently building a CLI using Go and am trying to disable any backtrace that is produced as a result of a panic. I believe my code has great error handling, but would now like to suppress any panic messages (fairly new to Go).
I currently put in the following in my main.go function (to purposely cause a panic):
var myarr [2]string
myarr[0] = "Foo"
myarr[1] = "Bar"
for i := range myarr {
fmt.Println(myarr[i+1])
}
And I get this as a result:
goroutine 1 [running]:
Bar
panic: runtime error: index out of range [2] with length 2
main.main()
{directory where main.go is located}/main.go:23 +0x207
How can I suppress this error such that anyone with the executable binary file will not be able to see this error?
I've tried utilizing the GOBACKTRACE environment variable when building my binary and setting its value to GOBACKTRACE=none, but this has no effect on other operating systems I've tested on.
I've tried utilizing the GOBACKTRACE environment variable when building my binary and setting its value to GOBACKTRACE=none, but this has no effect on other operating systems I've tested on.
The environment variable is called GOTRACEBACK, not GOBACKTRACE.
Also, you can use debug.SetTraceback("none") to achieve the same effect, although this can still be overridden via the GOTRACEBACK environment variable.
If you use the correct naming, it should work. If it does not work, congratulations: you found a bug in golang and you should likely report it.
As #Burak mentioned you want to use the built-in Go function recover. There's a good Go blog post on all the subtleties of panic and recovery.
If you want to blanket cover your entire application stack, then simply register recover via a defer function at the main level:
func main() {
defer func() {
if r := recover(); r != nil {
fmt.Println("unexpected problem encountered - aborting")
// optionally log `r` to an exception log - so users may email to developer
os.Exit(1)
}
}()
run(os.Args...)
}
https://play.golang.org/p/b8qYnlNZsr5
Is it a good idea to generate a secure random hex string until the process succeeds?
All examples I've come across show that if rand.Read returns error, we should panic, os.Exit(1) or return empty string and the error.
I need my program to continue to function in case of such errors and wait until a random string is generated. Is it a good idea to loop until the string is generated, any pitfalls with that?
import "crypto/rand"
func RandomHex() string {
var buf [16]byte
for {
_, err := rand.Read(buf[:])
if err == nil {
break
}
}
return hex.EncodeToString(buf[:])
}
No. It may always return an error in certain contexts.
Example: playground: don't use /dev/urandom in crypto/rand
Imagine that a machine does not have the source that crypto/rand gets data from or the program runs in a context that doesn't have access to that source. In that case you might consider having the program return that error in a meaningful way rather than spin.
More explicitly, if you are serious in your use of crypto/rand then consider writing RandomHex such that it is exceptionally clear to the caller that it is meant for security contexts (possibly rename it) and return the error from RandomHex. The calling function needs to handle that error and let the user know that something is very wrong. For example in a rest api, I'd expect that error to surface to the request handler, fail & return a 500 at that point, and log a high severity error.
Is it a good idea to loop until the string is generated,
That depends. Probably yes.
any pitfalls with that?
You discard the random bytes read on error. And this in a tight loop.
This may drain you entropy source (depending on the OS) faster than
it can be filled.
Instead of an unbound infinite loop: Break after n rounds and give up.
Graceful degradation or stopping is best: If your program is stuck in
an endless loop it is also not "continue"ing.
Think of a large project which deals with tons of concurrent requests handled by its own goroutine. It happens that there is a bug in the code and one of these requests will cause panic due to a nil reference.
In Java, C# and many other languages, this would end up in a exception which would stop the request without any harm to other healthy requests. In go, that would crash the entire program.
AFAIK, I'd have to have recover() for every single new go routine creation. Is that the only way to prevent entire program from crashing?
UPDATE: adding recover() call for every gorouting creation seems OK. What about third-party libraries? If third party creates goroutines without recover() safe net, it seems there is NOTHING to be done.
If you go the defer-recover-all-the-things, I suggest investing some time to make sure that a clear error message is collected with enough information to promptly act on it.
Writing the panic message to stderr/stdout is not great as it will be very hard to find where the problem is. In my experience the best approach is to invest a bit of time to get your Go programs to handle errors in a reasonable way. errors.Wrap from "github.com/pkg/errors" for instance allows you to wrap all errors and get a stack-trace.
Recovering panic is often a necessary evil. Like you say, it's not ideal to crash the entire program just because one requested caused a panic. In most cases recovering panics will not back-fire, but it is possible for a program to end up in a undefined not-recoverable state that only a manual restart can fix. That being said, my suggestion in this case is to make sure your Go program exposes a way to create a core dump.
Here's how to write a core dump to stderr when SIGQUIT is sent to the Go program (eg. kill pid -QUIT)
go func() {
// Based on answers to this stackoverflow question:
// https://stackoverflow.com/questions/19094099/how-to-dump-goroutine-stacktraces
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGQUIT)
for {
<-sigs
fmt.Fprintln(os.Stderr, "=== received SIGQUIT ===")
fmt.Fprintln(os.Stderr, "*** goroutine dump...")
var buf []byte
var bufsize int
var stacklen int
// Create a stack buffer of 1MB and grow it to at most 100MB if
// necessary
for bufsize = 1e6; bufsize < 100e6; bufsize *= 2 {
buf = make([]byte, bufsize)
stacklen = runtime.Stack(buf, true)
if stacklen < bufsize {
break
}
}
fmt.Fprintln(os.Stderr, string(buf[:stacklen]))
fmt.Fprintln(os.Stderr, "*** end of dump")
}
}()
there is no way you can handle panic without recover function, a good practice would be using a middleware like function for your safe function, checkout this snippet
https://play.golang.org/p/d_fQWzXnlAm
Do I understand correctly that crypto/rand.Reader can return Read error only on platforms not listed below, i.e. when it is not actually implemented?
// Reader is a global, shared instance of a cryptographically
// strong pseudo-random generator.
//
// On Linux, Reader uses getrandom(2) if available, /dev/urandom otherwise.
// On OpenBSD, Reader uses getentropy(2).
// On other Unix-like systems, Reader reads from /dev/urandom.
// On Windows systems, Reader uses the CryptGenRandom API.
var Reader io.Reader
TL;DR; crypto/rand's Read() (and Reader.Read()) methods may fail due to a variety of reasons, even on the platforms listed as supported. Do not assume that calls to this functions will always succeed. Always check the error return value.
Do I understand correctly that crypto/rand.Reader can return Read error only on platforms not listed below, i.e. when it is not actually implemented?
No. For example, have a look at the Linux implementation of rand.Reader. If available, this implementation will use the getrandom Linux system call, which may fail with a number of errors (most importantly, EAGAIN):
EAGAIN - The requested entropy was not available, and getrandom() would
have blocked if the GRND_NONBLOCK flag was not set.
The EAGAIN error quite literally tells you to "try again later"; the official meaning according to man 3 errno is "Resource temporarily unavailable". So when receiving an EAGAIN error you could simply keep trying for a certain time.
If getrandom is not available, the crypto/rand module will try to open and read from /dev/urandom (see source code), which might also fail for any number of reasons. These errors might not necessarily be of temporary nature (for example, issues with file system permissions); if your application depends on the availability of random data, you should treat an error like any other kind of non-recoverable error in your application.
For these reasons, you should not assume that rand.Read() will always succeed on Linux/UNIX and always check rand.Read()'s error return value.
type io.Reader
Reader is the interface that wraps the basic Read method.
Read reads up to len(p) bytes into p. It returns the number of bytes
read (0 <= n <= len(p)) and any error encountered. Even if Read
returns n < len(p), it may use all of p as scratch space during the
call. If some data is available but not len(p) bytes, Read
conventionally returns what is available instead of waiting for more.
When Read encounters an error or end-of-file condition after
successfully reading n > 0 bytes, it returns the number of bytes read.
It may return the (non-nil) error from the same call or return the
error (and n == 0) from a subsequent call. An instance of this general
case is that a Reader returning a non-zero number of bytes at the end
of the input stream may return either err == EOF or err == nil. The
next Read should return 0, EOF.
Callers should always process the n > 0 bytes returned before
considering the error err. Doing so correctly handles I/O errors that
happen after reading some bytes and also both of the allowed EOF
behaviors.
Implementations of Read are discouraged from returning a zero byte
count with a nil error, except when len(p) == 0. Callers should treat
a return of 0 and nil as indicating that nothing happened; in
particular it does not indicate EOF.
Implementations must not retain p.
type Reader interface {
Read(p []byte) (n int, err error)
}
No. io.Readers return errors.
I am running a program in Go which sends data continuously after reading a file /proc/stat.
Using ioutil.ReadFile("/proc/stat")
After running for about 14 hrs i got err: too many files open /proc/stat
Click here for snippet of code.
I doubt that defer f.Close is ignored by Go sometimes or it is skipping it.
The snippet of code (in case play.golang.org dies sooner than stackoverflow.com):
package main
import ("fmt";"io/ioutil")
func main() {
for {
fmt.Println("Hello, playground")
fData,err := ioutil.ReadFile("/proc/stat")
if err != nil {
fmt.Println("Err is ",err)
}
fmt.Println("FileData",string(fData))
}
}
The reason probably is that somewhere in your program:
you are forgetting to close files, or
you are leaning on the garbage collector to automatically close files on object finalization, but Go's conservative garbage collector fails to do so. In this case you should check your program's memory consumption (whether it is steadily increasing while the program is running).
In either case, try to check the contents of /proc/PID/fd to see whether the number of open files is increasing while the program is running.
If you are sure you Do the f.Close(),it still has the proble,Maybe it is because your other connection,for example the connection to MYSQL,also will be cause the problem,especially,in a loop,and you forget to close the connection.
Always do :
db.connection....
**defer db.Close()**
If it is in loop
loop
db.connection....
**defer db.Close()**
end
Do not put the db.connection before the loop