The sample of code:
package main
import (
"io"
"os"
"os/signal"
"sync"
"syscall"
)
func main() {
sigintCh := make(chan os.Signal, 1)
signal.Notify(sigintCh, syscall.SIGINT, syscall.SIGTERM)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
io.Copy(os.Stdout, os.Stdin)
}()
<-sigintCh
os.Stdin.Close()
wg.Wait()
}
If run this sample and try to interrupt by ^C it waits for any input and stops only after sending something to stdin (e.g. just press enter).
I expect that closing Stdin will be like sending EOF, but it doesn't work.
Closing os.Stdin will cause io.Copy to return with error file already closed next time it reads from it (after CTRL-C, try pressing Enter).
As explained in the File.Close docs:
Close closes the File, rendering it unusable for I/O.
You cannot force an EOF return from os.Stdin by closing it (or any other way). Instead, you would need to either wrap os.Stdin and implement your own Read method that conditionally returns EOF, or read a limited number of bytes in a loop.
You can see some more discussion and possible workarounds on this golang-nuts thread.
You can interrupt an io.Copy without closing the source side - by passing an io.Reader that has been wrapped with logic that takes a cancelable context.Context outlined here.
Modify your above goroutine like so:
ctx, cancel := context.WithCancel(context.Background())
go func() {
defer wg.Done()
r := NewReader(ctx, os.Stdin) // wrap io.Reader to make it context-aware
_, err := io.Copy(os.Stdout, r)
if err != nil {
// context.Canceled error if interrupted
}
}()
<-sigintCh
cancel() // canceling context will interrupt io.Copy operation
You can import NewReader from an external package like github.com/jbenet/go-context/io or inline a snippet from the blog link above:
type readerCtx struct {
ctx context.Context
r io.Reader
}
func (r *readerCtx) Read(p []byte) (n int, err error) {
if err := r.ctx.Err(); err != nil {
return 0, err
}
return r.r.Read(p)
}
// NewReader gets a context-aware io.Reader.
func NewReader(ctx context.Context, r io.Reader) io.Reader {
return &readerCtx{ctx: ctx, r: r}
}
Related
I wish to put a timeout on a function called foo. Consider the following
func fooWithTimeout(d time.Duration) error {
ch := make(chan error, 1)
go func() {
ch <- foo()
}()
select {
case err := <-ch:
return err
case <-time.After(d):
return errors.New("foo has timed out")
}
}
If foo has timed out, then will foo ever be able to write to channel ch or is there a risk the goroutine blocks or panics?
What happens to channel ch once fooWithTimeout has exited?
Is this code potentially problematic?
Should I add defer close(ch) within go func(){...}() just before calling foo?
Does it matter that I use a buffered (with size 1) or an unbuffered channel in this example?
After the timer tick, fooWithTimeout will return. The goroutine will continue running until foo returns.
If foo times out, it will write to channel ch because it is buffered.
The channel ch will be garbage collected eventually if foo returns.
You don't need to close the channel. Once it is out of scope, it will be garbage collected.
A large burst of calls fooWithTimeout will create large amount of resources. Each call creates two goroutines. The proper way of timing this out is to change foo to use a context.
Building on https://stackoverflow.com/a/73611534/1079543, here is foo with a context:
package main
import (
"context"
"fmt"
"log"
"time"
)
func foo(ctx context.Context) (string, error) {
ch := make(chan string, 1)
go func() {
fmt.Println("Sleeping...")
time.Sleep(time.Second * 1)
fmt.Println("Wake up...")
ch <- "foo"
}()
select {
case <-ctx.Done():
return "", fmt.Errorf("context cancelled: %w", ctx.Err())
case result := <-ch:
return result, nil
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*3)
defer cancel()
res, err := foo(ctx)
if err != nil {
log.Fatalf("foo failed: %v", err)
}
log.Printf("res: %s", res)
}
I want to write a mime/multipart message in Python to standard output and read that message in Golang using the mime/multipart package. This is just a learning exercise.
I tried simulating this example.
output.py
#!/usr/bin/env python2.7
import sys
s = "--foo\r\nFoo: one\r\n\r\nA section\r\n" +"--foo\r\nFoo: two\r\n\r\nAnd another\r\n" +"--foo--\r\n"
print s
main.go
package main
import (
"io"
"os/exec"
"mime/multipart"
"log"
"io/ioutil"
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
pr,pw := io.Pipe()
defer pw.Close()
cmd := exec.Command("python","output.py")
cmd.Stdout = pw
mr := multipart.NewReader(pr,"foo")
wg.Add(1)
go func() {
defer wg.Done()
for {
p, err := mr.NextPart()
if err == io.EOF {
fmt.Println("EOF")
return
}
if err != nil {
log.Fatal(err)
}
slurp, err := ioutil.ReadAll(p)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Part : %q\n", slurp)
return
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
cmd.Wait()
wg.Wait()
}
Output of go run main.go:
fatal error: all goroutines are asleep - deadlock!
Other answers regarding this topic on StackOverflow are related to channels not being closed, but I am not even using a channel. I understand that somewhere, there is infinite loop or something similar, but I don't see it.
Try something like this (explanation below):
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"mime/multipart"
"os"
"os/exec"
"sync"
"github.com/pkg/errors"
)
func readCommand(cmdStdout io.ReadCloser, wg *sync.WaitGroup, resc chan<- []byte, errc chan<- error) {
defer wg.Done()
defer close(errc)
defer close(resc)
mr := multipart.NewReader(cmdStdout, "foo")
for {
part, err := mr.NextPart()
if err != nil {
if err == io.EOF {
fmt.Println("EOF")
} else {
errc <- errors.Wrap(err, "failed to get next part")
}
return
}
slurp, err := ioutil.ReadAll(part)
if err != nil {
errc <- errors.Wrap(err, "failed to read part")
return
}
resc <- slurp
}
}
func main() {
cmd := exec.Command("python", "output.py")
cmd.Stderr = os.Stderr
pr, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(1)
resc := make(chan []byte)
errc := make(chan error)
go readCommand(pr, &wg, resc, errc)
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
for {
select {
case err, ok := <-errc:
if !ok {
errc = nil
break
}
if err != nil {
log.Fatal(errors.Wrap(err, "error from goroutine"))
}
case res, ok := <-resc:
if !ok {
resc = nil
break
}
fmt.Printf("Part from goroutine: %q\n", res)
}
if errc == nil && resc == nil {
break
}
}
cmd.Wait()
wg.Wait()
}
In no particular order:
Rather than using an io.Pipe() as the command's Stdout, just ask the command for it's StdoutPipe(). cmd.Wait() will ensure it's closed for you.
Set cmd.Stderr to os.Stderr so that you can see errors generated by your Python program.
I noticed this program was hanging anytime the Python program wrote to standard error. Now it doesn't :)
Don't make the WaitGroup a global variable; pass a reference to it to the goroutine.
Rather than log.Fatal()ing inside the goroutine, create an error channel to communicate errors back to main().
Rather than printing results inside the goroutine, create a result channel to communicate results back to main().
Ensure channels are closed to prevent blocking/goroutine leaks.
Separate out the goroutine into a proper function to make the code easier to read and follow.
In this example, we can create the multipart.Reader() inside our goroutine, since this is the only part of our code that uses it.
Note that I am using Wrap() from the errors package to add context to the error messages. This is, of course, not relevant to your question, but is a good habit.
The for { select { ... } } part may be confusing. This is one article I found introducing the concept. Basically, select is letting us read from whichever of these two channels (resc and errc) are currently readable, and then setting each to nil when the channel is closed. When both channels are nil, the loop exits. This lets us handle "either a result or an error" as they come in.
Edit: As johandalabacka said on the Golang Forum, it looks like the main issue here was that Python on Windows was adding an extra \r to the output, and that the problem is your Python program should omit the \r in the output string or sys.stdout.write() instead of print() ing. The output could also be cleaned up on the Golang side, but, aside from not being able to parse properly without modifying the Python side, this answer will still improve the concurrency mechanics of your program.
I have the following code:
package main
import "net"
import "fmt"
import "bufio"
func main() {
conn, _ := net.Dial("tcp", "irc.freenode.net:6667")
reader := bufio.NewReader(conn)
go func() {
str, err := reader.ReadString('\n')
if err != nil {
// handle it
fmt.Println(err)
}
fmt.Println(str)
}()
}
If I don't have the code that reads from the buffer in a goroutine, it outputs a message like this, which is what I expect to happen:
:zelazny.freenode.net NOTICE * :*** Looking up your hostname...
However, having it inside a goroutine prints nothing.
Can someone explain why that is?
Your program will exit when the main() function finishes. This is likely to happen before your goroutine has time to run and print its output.
One option would be to have the main goroutine block reading from a channel, and have the goroutine write to the channel when it has completed its work.
Another common way to "wait for a goroutines end", is using WaitGroup:
http://golang.org/pkg/sync/#WaitGroup . You can use waitGroup.Add(1) for each started goroutine, then use waitGroup.Done() in each goroutine after it finishes. In the main function you can use waitGroup.Wait() and this will wait until waitGroup.Done() has been called for each added goroutine.
Write data to a channel ch at the end of goroutine and read data from ch out of goroutine can make the main function waiting for goroutine print message.
Here is an example:
package main
import "net"
import "fmt"
import "bufio"
func main() {
conn, _ := net.Dial("tcp", "irc.freenode.net:6667")
reader := bufio.NewReader(conn)
ch := make(chan byte, 1)
go func() {
str, err := reader.ReadString('\n')
if err != nil {
// handle it
fmt.Println(err)
}
fmt.Println(str)
ch <- 1
}()
<-ch
}
You need to add a time delay like time.Sleep(3 * time.Second) (this waits 3 seconds).
The goroutine closes when the program terminates. I had the exact same problem.
I wrote a short script to write a file concurrently.
One goroutine is supposed to write strings to a file while the others are supposed to send the messages through a channel to it.
However, for some really strange reason the file is created but no message is added to it through the channel.
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
func concurrent(n uint64) {
output <- fmt.Sprint(n)
defer wg.Done()
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666);
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg+"\n")
}
}
func main() {
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
printOutput()
}
The printOutput() goroutine is executed completely, if I tried to write something after the for loop it would actually get into the file. So this leads me to think that range output might be null
You need to have something taking from the output channel as it is blocking until something removes what you put on it.
Not the only/best way to do it but: I moved printOutput() to above the other funcs and run it as a go routine and it prevents the deadlock.
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
func concurrent(n uint64) {
output <- fmt.Sprint(n)
defer wg.Done()
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg + "\n")
}
}
func main() {
go printOutput()
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
}
One of the reason why you get a null output is because channels are blocking for both send/receive.
According to your flow, the code snippet below will never reach wg.Done(), as sending channel is expecting a receiving end to pull the data out. This is a typical deadlock example.
func concurrent(n uint64) {
output <- fmt.Sprint(n) // go routine is blocked until data in channel is fetched.
defer wg.Done()
}
Let's examine the main func:
func main() {
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait() // the main thread will be waiting indefinitely here.
close(output)
printOutput()
}
My take on the problem:
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
var donePrinting = make(chan struct{})
func concurrent(n uint) {
defer wg.Done() // It only makes sense to defer
// wg.Done() before you do something.
// (like sending a string to the output channel)
output <- fmt.Sprint(n)
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg + "\n")
}
donePrinting <- struct{}{}
}
func main() {
wg.Add(2)
go printOutput()
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
<-donePrinting
}
Each concurrent function will deduct from the wait-group.
After the two concurrent goroutines finish, the wg.Wait() will unblock, and the next instruction (close(output)) will be executed. You have to wait for the two goroutines to finish before closing the channel. If, instead, you try the following:
go printOutput()
go concurrent(1)
go concurrent(2)
close(output)
wg.Wait()
you could end up with the close(output) instruction executing before any one of the concurrent goroutines concludes. If the channel closes before the concurrent goroutines run, they will crash at runtime, (while trying to write to a closed channel).
If, then, you don’t wait up for the printOutput() goroutine to finish, you could actually quit main() before printOutput() has gotten the chance to finish writing to its file.
Because I want to wait for the printOutput() goroutine to finish before I quit the program, I also created a separate channel just to signal that printOutput() is done.
The <-donePrinting instruction blocks until main receives something over the donePrinting channel.
Once main receives anything (even the empty structure that printOutput() sends), it will unblock and run to conclusion.
https://play.golang.org/p/nXJoYLI758m
I have the following code:
package main
import "net"
import "fmt"
import "bufio"
func main() {
conn, _ := net.Dial("tcp", "irc.freenode.net:6667")
reader := bufio.NewReader(conn)
go func() {
str, err := reader.ReadString('\n')
if err != nil {
// handle it
fmt.Println(err)
}
fmt.Println(str)
}()
}
If I don't have the code that reads from the buffer in a goroutine, it outputs a message like this, which is what I expect to happen:
:zelazny.freenode.net NOTICE * :*** Looking up your hostname...
However, having it inside a goroutine prints nothing.
Can someone explain why that is?
Your program will exit when the main() function finishes. This is likely to happen before your goroutine has time to run and print its output.
One option would be to have the main goroutine block reading from a channel, and have the goroutine write to the channel when it has completed its work.
Another common way to "wait for a goroutines end", is using WaitGroup:
http://golang.org/pkg/sync/#WaitGroup . You can use waitGroup.Add(1) for each started goroutine, then use waitGroup.Done() in each goroutine after it finishes. In the main function you can use waitGroup.Wait() and this will wait until waitGroup.Done() has been called for each added goroutine.
Write data to a channel ch at the end of goroutine and read data from ch out of goroutine can make the main function waiting for goroutine print message.
Here is an example:
package main
import "net"
import "fmt"
import "bufio"
func main() {
conn, _ := net.Dial("tcp", "irc.freenode.net:6667")
reader := bufio.NewReader(conn)
ch := make(chan byte, 1)
go func() {
str, err := reader.ReadString('\n')
if err != nil {
// handle it
fmt.Println(err)
}
fmt.Println(str)
ch <- 1
}()
<-ch
}
You need to add a time delay like time.Sleep(3 * time.Second) (this waits 3 seconds).
The goroutine closes when the program terminates. I had the exact same problem.