In the code bellow, why does the Write() operation not work?
package main
import (
"fmt"
"bufio"
"os"
)
func main() {
fmt.Println("Hello, playground")
writer := bufio.NewWriter(os.Stdout)
//var msg []byte
msg := []byte{104, 101, 108, 108, 111, 10}
_, err := writer.Write(msg)
if err != nil {
fmt.Println("some error")
}
}
The output is:
Hello, playground
But it should be:
Hello, playground
hello
Also, I don’t want to use fmt.Println(). To be more specific, I get the data as []byte type
As Cerise Limón noted in a comment, a writer.Write() call merely queues up more data to be written (depending on the buffer size and the amount of data). The actual write may happen later, or never. In your case, since you never tell the writer to finish any delayed write, no write ever happens.
You'll need to invoke writer.Flush() (not writer.WriterFlush(), that's a typo of sorts). This can return an error, if the write fails, so ideally you should check for that, not just defer the call and hope. However, there's not much you can do about the failure, if there is one.
You can do an explicit, in-line check for error, as I did here for instance, or you can just defer the flush call and throw away any error.
Related
Being fairly new to Go, I wrote a simple Go program that continually reads a file as it is written - for example to watch a log file for specific output, which I want to capture as soon as it has been written.
My (naive) code:
package main
import (
"bufio"
"bytes"
"fmt"
"io"
"os"
)
func main() {
fileHandle, err := os.Open("test.log")
if err != nil {
fmt.Println(err)
}
fileReader := bufio.NewReader(fileHandle)
line := bytes.NewBufferString("")
for true {
content, err := fileReader.ReadBytes('\n')
line.Write(content)
if err == io.EOF {
continue
} else {
if line.String() == "done\r\n" {
break
}
fmt.Println(line.String())
line.Reset()
}
}
err = fileHandle.Close()
if err != nil {
fmt.Println(err)
}
}
This works just fine, it prints every line of the log file as it is being written, never writing anything until it has a complete line (which is the desired behaviour).
My question: what is the idiomatic way in Go to change this code so that I can run this as a routine that blocks until there is more to read?
I know how to take the code and turn it into a go routine, but it would still be constantly trying to read the file, dealing with the io.EOF and continuing. Is there a more idiomatic way of writing this as a go routine that would avoid needlessly checking?
Is it a matter of constantly checking .Buffered() instead? Even that seems wasteful and intensive. I can slow it down with time.Sleep() whenever it encounters io.EOF, but I want to minimise the amount of delay. In Go, what is the way to wait for something to be written to the file, with a minimum of delay, and the minimum computational and I/O overhead?
(Note: I'm not asking for a rewrite of the code above, but really just an answer to the question itself, perhaps pointing at an existing example, or the appropriate functions in the documentation)
Could someone please explain (or/and share examples) when and why readers should to be closed explicitly, i.e. implement io.ReadCloser, not just io.Reader.
For example, when you are working with files, or any resource that should be closed to release the allocated resource (or memory for example for your resource, e.g. C code calling from Go).
You may use it when you have Read and Close methods, an example to show that you may use one common function to work with different types using io.ReadCloser:
package main
import (
"fmt"
"io"
"log"
"os"
)
func main() {
f, err := os.Open("./main.go")
if err != nil {
log.Fatal(err)
}
doIt(f)
doIt(os.Stdin)
}
func doIt(rc io.ReadCloser) {
defer rc.Close()
buf := make([]byte, 4)
n, err := rc.Read(buf)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", buf[:n])
}
Run and enter 12345 as an input, Output:
pack
12345
1234
See also:
Does Go automatically close resources if not explicitly closed?
It's for an explicit definition of Reader and Closer interface. So let's say you write some functionality that reads data, but you also want to close resource after doing it (again not to leak descriptors).
func ...(r io.ReaderCloser) {
defer r.Close()
... // some reading
}
anything you pass in it will need to have both interfaces defined, is it os.File or any custom struct, in this case, you are forcing client of your API to define Read and Close interfaces implementations, not just io.Reader.
Given the following function:
func main() {
l := createListener(8080)
r := ksws.CreateRouter()
if err := http.Serve(l, r); err != nil {
fmt.Println("An error occured.")
}
}
I'm wondering why I should catch the 'error' returned from the 'http.Serve' method?
It seems that an error is never returned here.
However, according to the documentation https://golang.org/pkg/net/http/#Serve the Serve method always returns a non-nill error.
Can someone provide me some guidance on this?
Simple case: when port 8080 already used you'll have error:
listen tcp :8080: bind: address already in use
Another case: http.Serve calls srv.trackListener which also may fail in case go didn't manage to add listener.
Also: http.Serve calls l.Accept() which also may fail...
So there are many possible cases...
And also it's idiomatic for go to check all errors returned by any function.
PS: It's way better to have redundant error check than to have silent not working program (imho)...
Take a look at the source code and it might shine some light into your question.
https://golang.org/src/net/http/server.go?s=75585:75634#L2838
// Serve always returns a non-nil error and closes l.
// After Shutdown or Close, the returned error is ErrServerClosed.
So the error will alway be return either with a real error if something went wrong or the ErrServerClosed in case of a shutdown or close, which happen for several reasons.
in your code when you ctrl-c, note that the prints does not occur because the program has ended. you should listen to signals to prevent that behavior.
package main
import (
"fmt"
"net/http"
"os"
"os/signal"
"time"
)
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
r := http.NewServeMux()
srv := &http.Server{
Addr: ":0",
Handler: r,
}
go func() {
if err := srv.ListenAndServe(); err != nil {
fmt.Println("An error occured.")
}
}()
// Block until a signal is received.
<-c
srv.Close()
fmt.Println("rr")
<-time.After(time.Second)
}
How to obtain the stack trace of the last (ideally of all) go routine (the app has multiple go routines) which panicked and recovered and logged only a not much descriptive error message? I don't know which routine recovered. Also, please keep in mind that I will not alter the code of any imported package. This panic happened in some of the imported packages which creates multiple go routines so I need a way to grab the stack trace of the last recovered routine in order to find where it panic.
The short answer is: Not possible but there are exceptions.
Golang has a few stack control methods and types.
You can control the stack levels with runtime/debug/SetTraceback
func SetTraceback(level string)
SetTraceback sets the amount of detail printed by the runtime
inthe traceback it prints before exiting due to an
unrecovered panic or an internal runtime error.
The level argument takes the same values as the GOTRACEBACK
environment variable. For example, SetTraceback("all") ensure
that the program prints all goroutines when it crashes.
See the package runtime documentation for details. If
SetTraceback is called with a level lower than that of the
environment variable, the call is ignored.
You can also print the stack strace with runtime/debug/Stack
func Stack() []byte
Stack returns a formatted stack trace of the goroutine that calls it. It calls runtime.Stack with a large enough buffer to capture the entire trace.
Also you need to understand how the Built-in funct recover works.
The recover built-in function allows a program to manage behavior of a
panicking goroutine. Executing a call to recover inside a deferred
function (but not any function called by it) stops the panicking sequence
by restoring normal execution and retrieves the error value passed to the
call of panic. If recover is called outside the deferred function it will
not stop a panicking sequence. In this case, or when the goroutine is not
panicking, or if the argument supplied to panic was nil, recover returns
nil. Thus the return value from recover reports whether the goroutine is
panicking.
func recover() interface{}
Working Example
This example assumes that the package does not call recover (detailed in another section).
Golang Playground Link
package main
import (
"log"
"errors"
"runtime/debug"
"time"
)
func f2() {
panic(errors.New("oops")) // line 11
}
func f1() {
f2() // line 15
}
func main() {
defer func() {
if e := recover(); e != nil {
log.Printf("%s: %s", e, debug.Stack()) // line 20
}
}()
go f1() // line 25
time.Sleep(time.Second * 1)
}
If package calls recover
If the code is recovering from the panic you need to use a debugger or remove the recover to understand what is going on as seen on the example below which demonstrate that recovered panics can not be "recovered" again.
Golang Playground Link
package main
import (
"log"
"errors"
"runtime/debug"
"time"
)
func f2() {
panic(errors.New("oops")) // line 11
}
func f1() {
defer func() {
if e := recover(); e != nil {
log.Printf("internal %s: %s", e, debug.Stack()) // line 20
}
}()
f2() // line 15
}
func main() {
defer func() {
if e := recover(); e != nil {
log.Printf("external %s: %s", e, debug.Stack()) // line 20
} else {
log.Println("Nothing to print")
}
}()
go f1() // line 25
time.Sleep(time.Second * 1)
}
Lesser then two evils
Debug with Delve Or temporaly edit the package so it logs the full message (once understood you can revert the changes).
Also if you find the problem let the package author know so it can be fixed.
for an assignment we are using go and one of the things we are going to do is to parse a uniprotdatabasefile line-by-line to collect uniprot-records.
I prefer not to share too much code, but I have a working code snippet that does parse such a file (2.5 GB) correctly in 48 s (measured using the time go-package). It parses the file iteratively and add lines to a record until a record end signal is reached (a full record), and metadata on the record is created. Then the record string is nulled, and a new record is collected line-by-line. Then I thought that I would try to use go-routines.
I have got some tips before from stackoverflow, and then to the original code I simple added a function to handle everything concerning the metadata-creation.
So, the code is doing
create an empty record,
iterate the file and add lines to the record,
if a record stop signal is found (now we have a full record) - give it to a go routine to create the metadata
null the record string and continue from 2).
I also added a sync.WaitGroup() to make sure that I waited (in the end) for each routine to finish. I thought that this would actually lower the time spent on parsing the databasefile as it continued to parse while the goroutines would act on each record. However, the code seems to run for more than 20 minutes indicating that something is wrong or the overhead went crazy. Any suggestions?
package main
import (
"bufio"
"crypto/sha1"
"fmt"
"io"
"log"
"os"
"strings"
"sync"
"time"
)
type producer struct {
parser uniprot
}
type unit struct {
tag string
}
type uniprot struct {
filenames []string
recordUnits chan unit
recordStrings map[string]string
}
func main() {
p := producer{parser: uniprot{}}
p.parser.recordUnits = make(chan unit, 1000000)
p.parser.recordStrings = make(map[string]string)
p.parser.collectRecords(os.Args[1])
}
func (u *uniprot) collectRecords(name string) {
fmt.Println("file to open ", name)
t0 := time.Now()
wg := new(sync.WaitGroup)
record := []string{}
file, err := os.Open(name)
errorCheck(err)
scanner := bufio.NewScanner(file)
for scanner.Scan() { //Scan the file
retText := scanner.Text()
if strings.HasPrefix(retText, "//") {
wg.Add(1)
go u.handleRecord(record, wg)
record = []string{}
} else {
record = append(record, retText)
}
}
file.Close()
wg.Wait()
t1 := time.Now()
fmt.Println(t1.Sub(t0))
}
func (u *uniprot) handleRecord(record []string, wg *sync.WaitGroup) {
defer wg.Done()
recString := strings.Join(record, "\n")
t := hashfunc(recString)
u.recordUnits <- unit{tag: t}
u.recordStrings[t] = recString
}
func hashfunc(record string) (hashtag string) {
hash := sha1.New()
io.WriteString(hash, record)
hashtag = string(hash.Sum(nil))
return
}
func errorCheck(err error) {
if err != nil {
log.Fatal(err)
}
}
First of all: your code is not thread-safe. Mainly because you're accessing a hashmap
concurrently. These are not safe for concurrency in go and need to be locked. Faulty line in your code:
u.recordStrings[t] = recString
As this will blow up when you're running go with GOMAXPROCS > 1, I'm assuming that you're not doing that. Make sure you're running your application with GOMAXPROCS=2 or higher to achieve parallelism.
The default value is 1, therefore your code runs on one single OS thread which, of course, can't be scheduled on two CPU or CPU cores simultaneously. Example:
$ GOMAXPROCS=2 go run udb.go uniprot_sprot_viruses.dat
At last: pull the values from the channel or otherwise your program will not terminate.
You're creating a deadlock if the number of goroutines exceeds your limit. I tested with a
76MiB file of data, you said your file was about 2.5GB. I have 16347 entries. Assuming linear growth,
your file will exceed 1e6 and therefore there are not enough slots in the channel and your program
will deadlock, giving no result while accumulating goroutines which don't run to fail at the end
(miserably).
So the solution should be to add a go routine which pulls the values from the channel and does
something with them.
As a side note: If you're worried about performance, do not use strings as they're always copied. Use []byte instead.