I'm trying out the net.Pipe(). I thought writing the "haha" string and then reading it back might be a good experiment.
Here is my first version. It blocks on the Write
func TestNetPipe(t *testing.T) {
out1 := make([]byte, 10)
c1, c2 := net.Pipe()
c1.Write([]byte("haha"))
c2.Read(out1)
}
I tried to use a goroutine
func TestNetPipe(t *testing.T) {
out1 := make([]byte, 10)
c1, c2 := net.Pipe()
go func() {
c1.Write([]byte("haha"))
}()
fmt.Printf("%v\n", out1)
c2.Read(out1)
fmt.Printf("%v\n", out1)
}
It works. But I felt there is no guarantee that the Read will read the whole "haha" string. It might only read the "hah" part.
I'm wondering if there is a better way to demo the usage of net.Pipe()
Use ReadAll function from package io/ioutil.
As ReadAll function blocks until EOF the following code needs no synchronization of goroutines. The call of close method causes the EOF on the stream.
package main
import (
"fmt"
"io/ioutil"
"log"
"net"
)
func main() {
r, w := net.Pipe()
go func() {
w.Write([]byte("haha"))
w.Close()
}()
b, err := ioutil.ReadAll(r)
if err != nil {
log.Fatalf(err.Error())
}
fmt.Println(string(b))
}
Playground
Related
I am writing code to control external LEDs over UDP, and need to keep a frame rate (e.g. 60Hz) that is as constant as possible. Too much jitter will look bad. I've written a simple test using time.NewTicker and the results are not ideal. I'm wondering if there is a different way to execute code on a more accurate interval. This test was run on macOS, but needs to run on Windows and Linux too. For what it's worth, it needs to sync to audio, so maybe there is an audio ticker API on each OS it could potentially sync with?
package main
import (
"encoding/csv"
"fmt"
"log"
"os"
"strconv"
"time"
)
var f *os.File
var testTimeSeconds = 30
func appendToCsv(t time.Time) {
w := csv.NewWriter(f)
defer w.Flush()
records := []string{strconv.FormatInt(t.UnixMicro(), 10)}
w.Write(records)
}
func init() {
var err error
f, err = os.Create("newTicker.csv")
if err != nil {
log.Fatal(err)
}
w := csv.NewWriter(f)
defer w.Flush()
records := []string{"timestamps"}
w.Write(records)
}
func main() {
usPerFrame := 16666
ticker := time.NewTicker(time.Duration(usPerFrame) * time.Microsecond)
defer ticker.Stop()
done := make(chan bool)
go func() {
time.Sleep(time.Duration(testTimeSeconds) * time.Second)
done <- true
}()
for {
select {
case <-done:
fmt.Println("Done!")
return
case t := <-ticker.C:
appendToCsv(t)
}
}
}
UPDATE:
I ran another test comparing the first method with the method in #jochen's answer, still not very accurate.
One idea would be to just use time.Sleep() instead of a ticker. This takes the channel send/receive out of the loop and may lead to more accurate timing. To do this, you could run a function like the following in a separate goroutine:
func ticker(step time.Duration, done <-chan struct{}) {
next := time.Now().Add(step)
for {
time.Sleep(time.Until(next))
appendToCsv(time.Now())
select { // check whether `done` was closed
case <-done:
return
default:
// pass
}
next = next.Add(step)
}
}
I want to write a mime/multipart message in Python to standard output and read that message in Golang using the mime/multipart package. This is just a learning exercise.
I tried simulating this example.
output.py
#!/usr/bin/env python2.7
import sys
s = "--foo\r\nFoo: one\r\n\r\nA section\r\n" +"--foo\r\nFoo: two\r\n\r\nAnd another\r\n" +"--foo--\r\n"
print s
main.go
package main
import (
"io"
"os/exec"
"mime/multipart"
"log"
"io/ioutil"
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
pr,pw := io.Pipe()
defer pw.Close()
cmd := exec.Command("python","output.py")
cmd.Stdout = pw
mr := multipart.NewReader(pr,"foo")
wg.Add(1)
go func() {
defer wg.Done()
for {
p, err := mr.NextPart()
if err == io.EOF {
fmt.Println("EOF")
return
}
if err != nil {
log.Fatal(err)
}
slurp, err := ioutil.ReadAll(p)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Part : %q\n", slurp)
return
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
cmd.Wait()
wg.Wait()
}
Output of go run main.go:
fatal error: all goroutines are asleep - deadlock!
Other answers regarding this topic on StackOverflow are related to channels not being closed, but I am not even using a channel. I understand that somewhere, there is infinite loop or something similar, but I don't see it.
Try something like this (explanation below):
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"mime/multipart"
"os"
"os/exec"
"sync"
"github.com/pkg/errors"
)
func readCommand(cmdStdout io.ReadCloser, wg *sync.WaitGroup, resc chan<- []byte, errc chan<- error) {
defer wg.Done()
defer close(errc)
defer close(resc)
mr := multipart.NewReader(cmdStdout, "foo")
for {
part, err := mr.NextPart()
if err != nil {
if err == io.EOF {
fmt.Println("EOF")
} else {
errc <- errors.Wrap(err, "failed to get next part")
}
return
}
slurp, err := ioutil.ReadAll(part)
if err != nil {
errc <- errors.Wrap(err, "failed to read part")
return
}
resc <- slurp
}
}
func main() {
cmd := exec.Command("python", "output.py")
cmd.Stderr = os.Stderr
pr, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(1)
resc := make(chan []byte)
errc := make(chan error)
go readCommand(pr, &wg, resc, errc)
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
for {
select {
case err, ok := <-errc:
if !ok {
errc = nil
break
}
if err != nil {
log.Fatal(errors.Wrap(err, "error from goroutine"))
}
case res, ok := <-resc:
if !ok {
resc = nil
break
}
fmt.Printf("Part from goroutine: %q\n", res)
}
if errc == nil && resc == nil {
break
}
}
cmd.Wait()
wg.Wait()
}
In no particular order:
Rather than using an io.Pipe() as the command's Stdout, just ask the command for it's StdoutPipe(). cmd.Wait() will ensure it's closed for you.
Set cmd.Stderr to os.Stderr so that you can see errors generated by your Python program.
I noticed this program was hanging anytime the Python program wrote to standard error. Now it doesn't :)
Don't make the WaitGroup a global variable; pass a reference to it to the goroutine.
Rather than log.Fatal()ing inside the goroutine, create an error channel to communicate errors back to main().
Rather than printing results inside the goroutine, create a result channel to communicate results back to main().
Ensure channels are closed to prevent blocking/goroutine leaks.
Separate out the goroutine into a proper function to make the code easier to read and follow.
In this example, we can create the multipart.Reader() inside our goroutine, since this is the only part of our code that uses it.
Note that I am using Wrap() from the errors package to add context to the error messages. This is, of course, not relevant to your question, but is a good habit.
The for { select { ... } } part may be confusing. This is one article I found introducing the concept. Basically, select is letting us read from whichever of these two channels (resc and errc) are currently readable, and then setting each to nil when the channel is closed. When both channels are nil, the loop exits. This lets us handle "either a result or an error" as they come in.
Edit: As johandalabacka said on the Golang Forum, it looks like the main issue here was that Python on Windows was adding an extra \r to the output, and that the problem is your Python program should omit the \r in the output string or sys.stdout.write() instead of print() ing. The output could also be cleaned up on the Golang side, but, aside from not being able to parse properly without modifying the Python side, this answer will still improve the concurrency mechanics of your program.
Say I want to accept an animal. The user can either set the type of animal at the What type of animal? prompt on a terminal, or she can go to http://localhost:1234/animal?type=kitten
Whichever she does, the terminal will read What type of animal? kitten (assuming she chose a kitten) and the program will then prompt the user on the terminal (and only the terminal) What is the kitten's name?
My thought was to use channels to go routines, but since both go routines will be stuck in a function call (Scan() for the terminal, ListenAndServe() for http) then I'm not clear how to stop the go routine that is still in a function call once the input is received. The normal method of selecting on channels won't work.
In C I'd just do a select(2) on the two relevant file descriptors (fd 0 and the fd from listen()) but I don't think that's how to do it in Go.
Note: I answered this down below. If you're down voting the question this I'd be curious to know why as I would have found the answer I came up with useful. If you're down voting the answer, I'd really be interested in knowing why. If it's non-idiomatic Go or if it has other issues I'm missing, I'd really like to fix them.
OK, I figured out a solution for what I was trying to do.
This might not be the most Go idiomatic way though. If the channels took a struct with input source and the string I could probably toss the select and the webinput channel.
package main
import (
"bufio"
"context"
"fmt"
"io"
"net/http"
"os"
"time"
)
func getHTTPAnimal(input chan string) *http.Server {
srv := &http.Server{Addr: ":1234"}
http.HandleFunc("/animal", func(w http.ResponseWriter, r *http.Request) {
animal, ok := r.URL.Query()["type"]
if !ok || len(animal[0]) <= 0 {
io.WriteString(w, "Animal not understood.\n")
return
}
io.WriteString(w, "Animal understood.\n")
input <- string(animal[0])
})
go func() {
if err := srv.ListenAndServe(); err != nil {
if err.Error() != "http: Server closed" {
fmt.Printf("getHTTPAnimal error with ListenAndServe: %s", err)
}
}
}()
return srv
}
func getTerminalInput(input chan string) {
scanner := bufio.NewScanner(os.Stdin)
for {
scanner.Scan()
input <- scanner.Text()
}
}
func main() {
input := make(chan string)
webinput := make(chan string)
srv := getHTTPAnimal(webinput)
fmt.Print("Enter animal type: ")
go getTerminalInput(input)
var animal string
select {
case ta := <-input:
animal = ta
case webta := <-webinput:
animal = webta
fmt.Printf("%s\n", animal)
}
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
srv.Shutdown(ctx)
close(webinput)
fmt.Printf("Enter animal name: ")
name := <-input
fmt.Printf("Congrats on getting %s the half-%s\n", name, animal)
}
You can stop goroutine with close statement.
Example:
package main
import "sync"
func main() {
var wg sync.WaitGroup
wg.Add(1)
ch := make(chan int)
go func() {
for {
foo, ok := <- ch
if !ok {
println("done")
wg.Done()
return
}
println(foo)
}
}()
ch <- 1
ch <- 2
ch <- 3
close(ch)
wg.Wait()
}
There is couple of ways to stop goroutine:
1) by closing the read channel:
msgCh := make(chan string)
go func(){
for msg, ok := range msgCh {
if !ok {
return
}
}
}()
msgCh <- "String"
close(msgCh)
2) passing some sort of kill switch
msgCh := make(chan string)
killSwitch := make(chan struct{})
go func(){
for {
select{
case msg <- msgCh:
log.Println(msg)
case <-killSwitch:
return
}
}
}
msgCh <- "String"
close(killSwitch)
With approach NR2 you are avoiding writing to closed chan that would end up as panic. Also close signal is populated as fanout pattern (all places where that chan is used will receive that signal)
I'm trying to read from Stdin in Golang as I'm trying to implement a driver for Erlang. I have the following code:
package main
import (
"fmt"
"os"
"bufio"
"time"
)
func main() {
go func() {
stdout := bufio.NewWriter(os.Stdin)
p := []byte{121,100,125,'\n'}
stdout.Write(p)
}()
stdin := bufio.NewReader(os.Stdin)
values := make([]byte,4,4)
for{
fmt.Println("b")
if read_exact(stdin) > 0 {
stdin.Read(values)
fmt.Println("a")
give_func_write(values)
}else{
continue
}
}
}
func read_exact(r *bufio.Reader) int {
bits := make([]byte,3,3)
a,_ := r.Read(bits)
if a > 0 {
r.Reset(r)
return 1
}
return -1
}
func give_func_write(a []byte) bool {
fmt.Println("Yahu")
return true
}
However it seems that the give_func_write is never reached. I tried to start a goroutine to write to standard input after 2 seconds to test this.
What am I missing here?
Also the line r.Reset(r). Is this valid in go? What I tried to achieve is simply restart the reading from the beginning of the file. Is there a better way?
EDIT
After having played around I was able to find that the code is stuck at a,_ := r.Read(bits) in the read_exact function
I guess that I will need to have a protocol in which I send a \n to
make the input work and at the same time discard it when reading it
No, you don't. Stdin is line-buffered only if it's bound to terminal. You can run your program prog < /dev/zero or cat file | prog.
bufio.NewWriter(os.Stdin).Write(p)
You probably don't want to write to stdin. See "Writing to stdin and reading from stdout" for details.
Well, it's not particular clear for me what you're trying to achieve. I'm assuming, that you just want to read data from stdin by fixed-size chunks. Use io.ReadFull for this. Or if you want to use buffers, you can use Reader.Peek or Scanner to ensure, that specific number of bytes is available. I've changed your program to demonstrate the usage of io.ReadFull:
package main
import (
"fmt"
"io"
"time"
)
func main() {
input, output := io.Pipe()
go func() {
defer output.Close()
for _, m := range []byte("123456") {
output.Write([]byte{m})
time.Sleep(time.Second)
}
}()
message := make([]byte, 3)
_, err := io.ReadFull(input, message)
for err == nil {
fmt.Println(string(message))
_, err = io.ReadFull(input, message)
}
if err != io.EOF {
panic(err)
}
}
You can easily split it in two programs and test it that way. Just change input to os.Stdin.
Code below works fine with hard coded JSON data however doesn't work when I read JSON data from a file. I'm getting fatal error: all goroutines are asleep - deadlock error when using sync.WaitGroup.
WORKING EXAMPLE WITH HARD-CODED JSON DATA:
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
)
func connect(host string) {
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string) {
for {
host := <-c
go connect(host)
}
}
func main() {
hosts := [2]string{"user1#111.79.154.111", "user2#111.79.190.222"}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
c <- hosts[i]
}
var input string
fmt.Scanln(&input)
}
OUTPUT:
user#user-VirtualBox:~/go$ go run channel.go
user1#111.79.154.111: " 09:46:40 up 86 days, 18:16, 0 users, load average: 5"
user2#111.79.190.222: " 09:46:40 up 86 days, 17:27, 1 user, load average: 9"
user1#111.79.154.111: DONE
user2#111.79.190.222: DONE
NOT WORKING - EXAMPLE WITH READING JSON DATA FILE:
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
"encoding/json"
"os"
"sync"
)
func connect(host string) {
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string) {
for {
host := <-c
go connect(host)
}
}
type Content struct {
Username string `json:"username"`
Ip string `json:"ip"`
}
func main() {
var wg sync.WaitGroup
var source []Content
var hosts []string
data := json.NewDecoder(os.Stdin)
data.Decode(&source)
for _, value := range source {
hosts = append(hosts, value.Username + "#" + value.Ip)
}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
wg.Add(1)
c <- hosts[i]
defer wg.Done()
}
var input string
fmt.Scanln(&input)
wg.Wait()
}
OUTPUT
user#user-VirtualBox:~/go$ go run deploy.go < hosts.txt
user1#111.79.154.111: " 09:46:40 up 86 days, 18:16, 0 users, load average: 5"
user2#111.79.190.222: " 09:46:40 up 86 days, 17:27, 1 user, load average: 9"
user1#111.79.154.111 : DONE
user2#111.79.190.222: DONE
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc210000068)
/usr/lib/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc210047020)
/usr/lib/go/src/pkg/sync/waitgroup.go:127 +0x14b
main.main()
/home/user/go/deploy.go:64 +0x45a
goroutine 3 [chan receive]:
main.listener(0xc210038060)
/home/user/go/deploy.go:28 +0x30
created by main.main
/home/user/go/deploy.go:53 +0x30b
exit status 2
user#user-VirtualBox:~/go$
HOSTS.TXT
[
{
"username":"user1",
"ip":"111.79.154.111"
},
{
"username":"user2",
"ip":"111.79.190.222"
}
]
Go program ends when the main function ends.
From the language specification
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
Therefore, you need to wait for your goroutines to finish. The common solution for this is to use sync.WaitGroup object.
The simplest possible code to synchronize goroutine:
package main
import "fmt"
import "sync"
var wg sync.WaitGroup // 1
func routine() {
defer wg.Done() // 3
fmt.Println("routine finished")
}
func main() {
wg.Add(1) // 2
go routine() // *
wg.Wait() // 4
fmt.Println("main finished")
}
And for synchronizing multiple goroutines
package main
import "fmt"
import "sync"
var wg sync.WaitGroup // 1
func routine(i int) {
defer wg.Done() // 3
fmt.Printf("routine %v finished\n", i)
}
func main() {
for i := 0; i < 10; i++ {
wg.Add(1) // 2
go routine(i) // *
}
wg.Wait() // 4
fmt.Println("main finished")
}
WaitGroup usage in order of execution.
Declaration of global variable. Making it global is the easiest way to make it visible to all functions and methods.
Increasing the counter. This must be done in main goroutine because there is no guarantee that newly started goroutine will execute before 4 due to memory model guarantees.
Decreasing the counter. This must be done at the exit of goroutine. Using deferred call, we make sure that it will be called whenever function ends no matter but no matter how it ends.
Waiting for the counter to reach 0. This must be done in main goroutine to prevent program exit.
* The actual parameters are evaluated before starting new gouroutine. Thus it is needed to evaluate them explicitly before wg.Add(1) so the possibly panicking code would not leave increased counter.
Use
param := f(x)
wg.Add(1)
go g(param)
instead of
wg.Add(1)
go g(f(x))
Thanks for the very nice and detailed explanation Grzegorz Żur.
One thing that I want to point it out that typically the func that needs to be threaded wont be in main(), so we would have something like this:
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"math/rand"
"os"
"reflect"
"regexp"
"strings"
"sync"
"time"
)
var wg sync.WaitGroup // VERY IMP to declare this globally, other wise one //would hit "fatal error: all goroutines are asleep - deadlock!"
func doSomething(arg1 arg1Type) {
// cured cancer
}
func main() {
r := rand.New(rand.NewSource(time.Now().UnixNano()))
randTime := r.Intn(10)
wg.Add(1)
go doSomething(randTime)
wg.Wait()
fmt.Println("Waiting for all threads to finish")
}
The thing that I want to point it out is that global declaration of wg is very crucial for all threads to finish before main()
try this code snippest
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
"sync"
)
func connect(host string, wg *sync.WaitGroup) {
defer wg.Done()
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string，wg *sync.WaitGroup) {
for {
host，ok := <-c
// check channel is closed or not
if !ok{
break
}
go connect(host)
}
}
func main() {
var wg sync.WaitGroup
hosts := [2]string{"user1#111.79.154.111", "user2#111.79.190.222"}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
wg.Add(1)
c <- hosts[i]
}
close(c)
var input string
fmt.Scanln(&input)
wg.Wait()
}