How do I select on os.Stdin and http in Go? - go

Say I want to accept an animal. The user can either set the type of animal at the What type of animal? prompt on a terminal, or she can go to http://localhost:1234/animal?type=kitten
Whichever she does, the terminal will read What type of animal? kitten (assuming she chose a kitten) and the program will then prompt the user on the terminal (and only the terminal) What is the kitten's name?
My thought was to use channels to go routines, but since both go routines will be stuck in a function call (Scan() for the terminal, ListenAndServe() for http) then I'm not clear how to stop the go routine that is still in a function call once the input is received. The normal method of selecting on channels won't work.
In C I'd just do a select(2) on the two relevant file descriptors (fd 0 and the fd from listen()) but I don't think that's how to do it in Go.
Note: I answered this down below. If you're down voting the question this I'd be curious to know why as I would have found the answer I came up with useful. If you're down voting the answer, I'd really be interested in knowing why. If it's non-idiomatic Go or if it has other issues I'm missing, I'd really like to fix them.

OK, I figured out a solution for what I was trying to do.
This might not be the most Go idiomatic way though. If the channels took a struct with input source and the string I could probably toss the select and the webinput channel.
package main
import (
"bufio"
"context"
"fmt"
"io"
"net/http"
"os"
"time"
)
func getHTTPAnimal(input chan string) *http.Server {
srv := &http.Server{Addr: ":1234"}
http.HandleFunc("/animal", func(w http.ResponseWriter, r *http.Request) {
animal, ok := r.URL.Query()["type"]
if !ok || len(animal[0]) <= 0 {
io.WriteString(w, "Animal not understood.\n")
return
}
io.WriteString(w, "Animal understood.\n")
input <- string(animal[0])
})
go func() {
if err := srv.ListenAndServe(); err != nil {
if err.Error() != "http: Server closed" {
fmt.Printf("getHTTPAnimal error with ListenAndServe: %s", err)
}
}
}()
return srv
}
func getTerminalInput(input chan string) {
scanner := bufio.NewScanner(os.Stdin)
for {
scanner.Scan()
input <- scanner.Text()
}
}
func main() {
input := make(chan string)
webinput := make(chan string)
srv := getHTTPAnimal(webinput)
fmt.Print("Enter animal type: ")
go getTerminalInput(input)
var animal string
select {
case ta := <-input:
animal = ta
case webta := <-webinput:
animal = webta
fmt.Printf("%s\n", animal)
}
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
srv.Shutdown(ctx)
close(webinput)
fmt.Printf("Enter animal name: ")
name := <-input
fmt.Printf("Congrats on getting %s the half-%s\n", name, animal)
}

You can stop goroutine with close statement.
Example:
package main
import "sync"
func main() {
var wg sync.WaitGroup
wg.Add(1)
ch := make(chan int)
go func() {
for {
foo, ok := <- ch
if !ok {
println("done")
wg.Done()
return
}
println(foo)
}
}()
ch <- 1
ch <- 2
ch <- 3
close(ch)
wg.Wait()
}

There is couple of ways to stop goroutine:
1) by closing the read channel:
msgCh := make(chan string)
go func(){
for msg, ok := range msgCh {
if !ok {
return
}
}
}()
msgCh <- "String"
close(msgCh)
2) passing some sort of kill switch
msgCh := make(chan string)
killSwitch := make(chan struct{})
go func(){
for {
select{
case msg <- msgCh:
log.Println(msg)
case <-killSwitch:
return
}
}
}
msgCh <- "String"
close(killSwitch)
With approach NR2 you are avoiding writing to closed chan that would end up as panic. Also close signal is populated as fanout pattern (all places where that chan is used will receive that signal)

Related

Channels terminate prematurely

I am prototyping a series of go routines for a pipeline that each perform a transformation. The routines are terminating before all the data has passed through.
I have checked Donavan and Kernighan book and Googled for solutions.
Here is my code:
package main
import (
"fmt"
"sync"
)
func main() {
a1 := []string{"apple", "apricot"}
chan1 := make(chan string)
chan2 := make(chan string)
chan3 := make(chan string)
var wg sync.WaitGroup
go Pipe1(chan2, chan1, &wg)
go Pipe2(chan3, chan2, &wg)
go Pipe3(chan3, &wg)
func (data []string) {
defer wg.Done()
for _, s := range data {
wg.Add(1)
chan1 <- s
}
go func() {
wg.Wait()
close(chan1)
}()
}(a1)
}
func Pipe1(out chan<- string, in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
out <- s + "s are"
}
}
func Pipe2(out chan<- string, in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
out <- s + " good for you"
}
}
func Pipe3(in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
fmt.Println(s)
}
}
My expected output is:
apples are good for you
apricots are good for you
The results of running main are inconsistent. Sometimes I get both lines. Sometimes I just get the apples. Sometimes nothing is output.
As Adrian already pointed out, your WaitGroup.Add and WaitGroup.Done calls are mismatched. However, in cases like this the "I am done" signal is typically given by closing the output channel. WaitGroups are only necessary if work is shared between several goroutines (i.e. several goroutines consume the same channel), which isn't the case here.
package main
import (
"fmt"
)
func main() {
a1 := []string{"apple", "apricot"}
chan1 := make(chan string)
chan2 := make(chan string)
chan3 := make(chan string)
go func() {
for _, s := range a1 {
chan1 <- s
}
close(chan1)
}()
go Pipe1(chan2, chan1)
go Pipe2(chan3, chan2)
// This range loop terminates when chan3 is closed, which Pipe2 does after
// chan2 is closed, which Pipe1 does after chan1 is closed, which the
// anonymous goroutine above does after it sent all values.
for s := range chan3 {
fmt.Println(s)
}
}
func Pipe1(out chan<- string, in <-chan string) {
for s := range in {
out <- s + "s are"
}
close(out) // let caller know that we're done
}
func Pipe2(out chan<- string, in <-chan string) {
for s := range in {
out <- s + " good for you"
}
close(out) // let caller know that we're done
}
Try it on the playground: https://play.golang.org/p/d2J4APjs_lL
You're calling wg.Wait in a goroutine, so main is allowed to return (and therefore your program exits) before the other routines have finished. This would cause the behavior your see, but taking out of a goroutine alone isn't enough.
You're also misusing the WaitGroup in general; your Add and Done calls don't relate to one another, and you don't have as many Dones as you have Adds, so the WaitGroup will never finish. If you're calling Add in a loop, then every loop iteration must also result in a Done call; as you have it now, you defer wg.Done() before each of your loops, then call Add inside the loop, resulting in one Done and many Adds. This code would need to be significantly revised to work as intended.

How to assign the values to struct while go routines are running?

I'm using goroutines in my project and I want to to assign the values to the struct fields but I don't know that how I will assign the values get by using mongodb quires to the struct fields I'm showing my struct and the query too.
type AppLoadNew struct{
StripeTestKey string `json:"stripe_test_key" bson:"stripe_test_key,omitempty"`
Locations []Locations `json:"location" bson:"location,omitempty"`
}
type Locations struct{
Id int `json:"_id" bson:"_id"`
Location string `json:"location" bson:"location"`
}
func GoRoutine(){
values := AppLoadNew{}
go func() {
data, err := GetStripeTestKey(bson.M{"is_default": true})
if err == nil {
values.StripeTestKey := data.TestStripePublishKey
}
}()
go func() {
location, err := GetFormLocation(bson.M{"is_default": true})
if err == nil {
values.Locations := location
}
}()
fmt.Println(values) // Here it will nothing
// empty
}
Can you please help me that I will assign all the values to the AppLoadNew struct.
In Go no value is safe for concurrent read and write (from multiple goroutines). You must synchronize access.
Reading and writing variables from multiple goroutines can be protected using sync.Mutex or sync.RWMutex, but in your case there is something else involved: you should wait for the 2 launched goroutines to complete. For that, the go-to solution is sync.WaitGroup.
And since the 2 goroutines write 2 different fields of a struct (which act as 2 distinct variables), they don't have to be synchronized to each other (see more on this here: Can I concurrently write different slice elements). Which means using a sync.WaitGroup is sufficient.
This is how you can make it safe and correct:
func GoRoutine() {
values := AppLoadNew{}
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
data, err := GetStripeTestKey(bson.M{"is_default": true})
if err == nil {
values.StripeTestKey = data.StripeTestKey
}
}()
wg.Add(1)
go func() {
defer wg.Done()
location, err := GetFormLocation(bson.M{"is_default": true})
if err == nil {
values.Locations = location
}
}()
wg.Wait()
fmt.Println(values)
}
See a (slightly modified) working example on the Go Playground.
See related / similar questions:
Reading values from a different thread
golang struct concurrent read and write without Lock is also running ok?
How to make a variable thread-safe
You can use sync package with WaitGroup, here is an example:
package main
import (
"fmt"
"sync"
"time"
)
type Foo struct {
One string
Two string
}
func main() {
f := Foo{}
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// Perform long calculations
<-time.After(time.Second * 1)
f.One = "foo"
}()
wg.Add(1)
go func() {
defer wg.Done()
// Perform long calculations
<-time.After(time.Second * 2)
f.Two = "bar"
}()
fmt.Printf("Before %+v\n", f)
wg.Wait()
fmt.Printf("After %+v\n", f)
}
The output:
Before {One: Two:}
After {One:foo Two:bar}

Can I reopen a closed channel?

I am trying to figure out whether I can reopen a channel after closing it.
Test case:
I have a channel with some things in it
I want to range over them therefore I need to close the channel beforehand
I want to put more stuff in the channel and iterate through it once more
go func() {
queue <- "1"
queue <- "2"
close(queue)
}()
for i := range queue {
go func(i string) {
fmt.Println("From queue: ", i)
}(i)
}
go func() {
open(queue)
queue <- "3"
queue <- "4"
close(queue)
}()
for i := range queue {
go func(i string) {
fmt.Println("From queue: ", i)
}(i)
}
Of course open does not exist. How can I implement something like what I need in Go?
I don't want to use the Sleep function
I want to range over them therefore I need to close the channel beforehand
Nope, no need to close the channel. It'll resume iteration when another item is pushed thru the channel.
The below code accepts console input and pushes it to the channel:
main.go
package main
import (
"log"
"bufio"
"os"
"fmt"
)
func process(c chan string) {
for s := range c {
log.Println("processed", s)
}
}
func main() {
c := make(chan string, 10)
go process(c)
// get from console and process
reader := bufio.NewReader(os.Stdin)
fmt.Println("INPUT STUFF. TYPE #quit TO EXIT.")
for {
input, _, _ := reader.ReadLine()
if string(input) == "#quit" {
break
}
c <- string(input)
}
log.Println("BYE!")
}
output
INPUT STUFF. TYPE #quit TO EXIT.
hello
2018/10/23 10:43:52 processed hello
world
2018/10/23 10:43:54 processed world
#quit
2018/10/23 10:43:57 BYE!
The below sample uses Sleep() and is runnable as a Go Playground snippet
package main
import (
"log"
"time"
)
func process(c chan string) {
for s := range c {
log.Println("processed", s)
}
}
func main() {
c := make(chan string, 10)
go process(c)
// push some data
c <- "barry allen"
c <- "iris west"
time.Sleep(time.Second * 2)
// push more data
c <- "joe west"
c <- "caitlin snow"
time.Sleep(time.Second * 3)
}
output
2009/11/10 23:00:00 processed barry allen
2009/11/10 23:00:00 processed iris west
2009/11/10 23:00:02 processed joe west
2009/11/10 23:00:02 processed caitlin snow
Hope this helps. Cheers,
You cannot reopen a closed channel but you can send a channel on a channel, perhaps this is what you're looking for?
package main
import (
"fmt"
"time"
)
func main() {
queue := make(chan chan int)
defer close(queue)
go func() { // reader
for {
ch := <-queue
for i := range ch {
fmt.Println(i)
}
fmt.Println("Done with this channel")
}
}()
go func() { // writer-1
ch := make(chan int)
defer close(ch)
queue <- ch
ch <- 4
ch <- 2
}()
go func() { // writer-2
ch := make(chan int)
defer close(ch)
queue <- ch
ch <- 4
ch <- 20
}()
time.Sleep(time.Second)
}
Tho you cannot reopen the channel, it is possible to create a new one and assign to the variable. The previously closed channel will be garbage collected. In this case I reused the queue variable, but you can also create a new queue2 variable and pass it to the goroutine.
package main
import (
"context"
"fmt"
"time"
)
func main() {
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
queue := make(chan int)
go printFormat(nil, queue)
queue <- 1
queue <- 2
close(queue)
// fake some actions in between
time.Sleep(2 * time.Second)
queue = make(chan int)
go printFormat(cancel, queue)
queue <- 3
queue <- 4
close(queue)
<-ctx.Done()
}
func printFormat(c context.CancelFunc, q chan int) {
for i := range q {
fmt.Printf("Data %d \n", i)
}
if c != nil {
c()
}
}

What's the idiomatic solution to embarassingly parallel tasks in Go?

I'm currently staring at a beefed up version of the following code:
func embarrassing(data []string) []string {
resultChan := make(chan string)
var waitGroup sync.WaitGroup
for _, item := range data {
waitGroup.Add(1)
go func(item string) {
defer waitGroup.Done()
resultChan <- doWork(item)
}(item)
}
go func() {
waitGroup.Wait()
close(resultChan)
}()
var results []string
for result := range resultChan {
results = append(results, result)
}
return results
}
This is just blowing my mind. All this is doing can be expressed in other languages as
results = parallelMap(data, doWork)
Even if it can't be done quite this easily in Go, isn't there still a better way than the above?
If you need all the results, you don't need the channel (and the extra goroutine to close it) to communicate the results, you can write directly into the results slice:
func cleaner(data []string) []string {
results := make([]string, len(data))
wg := &sync.WaitGroup{}
wg.Add(len(data))
for i, item := range data {
go func(i int, item string) {
defer wg.Done()
results[i] = doWork(item)
}(i, item)
}
wg.Wait()
return results
}
This is possible because slice elements act as distinct variables, and thus can be written individually without synchronization. For details, see Can I concurrently write different slice elements. You also get the results in the same order as your input for free.
Anoter variation: if doWork() would not return the result but get the address where the result should be "placed", and additionally the sync.WaitGroup to signal completion, that doWork() function could be executed "directly" as a new goroutine.
We can create a reusable wrapper for doWork():
func doWork2(item string, result *string, wg *sync.WaitGroup) {
defer wg.Done()
*result = doWork(item)
}
If you have the processing logic in such format, this is how it can be executed concurrently:
func cleanest(data []string) []string {
results := make([]string, len(data))
wg := &sync.WaitGroup{}
wg.Add(len(data))
for i, item := range data {
go doWork2(item, &results[i], wg)
}
wg.Wait()
return results
}
Yet another variation could be to pass a channel to doWork() on which it is supposed to deliver the result. This solution doesn't even require a sync.Waitgroup, as we know how many elements we want to receive from the channel:
func cleanest2(data []string) []string {
ch := make(chan string)
for _, item := range data {
go doWork3(item, ch)
}
results := make([]string, len(data))
for i := range results {
results[i] = <-ch
}
return results
}
func doWork3(item string, res chan<- string) {
res <- "done:" + item
}
"Weakness" of this last solution is that it may collect the result "out-of-order" (which may or may not be a problem). This approach can be improved to retain order by letting doWork() receive and return the index of the item. For details and examples, see How to collect values from N goroutines executed in a specific order?
You can also use reflection to achieve something similar.
In this example it distribute the handler function over 4 goroutines and returns the results in a new instance of the given source slice type.
package main
import (
"fmt"
"reflect"
"strings"
"sync"
)
func parralelMap(some interface{}, handle interface{}) interface{} {
rSlice := reflect.ValueOf(some)
rFn := reflect.ValueOf(handle)
dChan := make(chan reflect.Value, 4)
rChan := make(chan []reflect.Value, 4)
var waitGroup sync.WaitGroup
for i := 0; i < 4; i++ {
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
for v := range dChan {
rChan <- rFn.Call([]reflect.Value{v})
}
}()
}
nSlice := reflect.MakeSlice(rSlice.Type(), rSlice.Len(), rSlice.Cap())
for i := 0; i < rSlice.Len(); i++ {
dChan <- rSlice.Index(i)
}
close(dChan)
go func() {
waitGroup.Wait()
close(rChan)
}()
i := 0
for v := range rChan {
nSlice.Index(i).Set(v[0])
i++
}
return nSlice.Interface()
}
func main() {
fmt.Println(
parralelMap([]string{"what", "ever"}, strings.ToUpper),
)
}
Test here https://play.golang.org/p/iUPHqswx8iS

How to exit from my go code using go routines and term ui

I recently started learning go and I am really impressed with all the features. I been playing with go routines and term-ui and facing some trouble. I am trying to exit this code from console after I run it but it just doesn't respond. If I run it without go-routine it does respond to my q key press event.
Any help is appreciated.
My code
package main
import (
"fmt"
"github.com/gizak/termui"
"time"
"strconv"
)
func getData(ch chan string) {
i := 0
for {
ch <- strconv.Itoa(i)
i++
time.Sleep(time.Second)
if i == 20 {
break
}
}
}
func Display(ch chan string) {
err := termui.Init()
if err != nil {
panic(err)
}
defer termui.Close()
termui.Handle("/sys/kbd/q", func(termui.Event) {
fmt.Println("q captured")
termui.Close()
termui.StopLoop()
})
for elem := range ch {
par := termui.NewPar(elem)
par.Height = 5
par.Width = 37
par.Y = 4
par.BorderLabel = "term ui example with chan"
par.BorderFg = termui.ColorYellow
termui.Render(par)
}
}
func main() {
ch := make(chan string)
go getData(ch)
Display(ch)
}
This is possibly the answer you are looking for. First off, you aren't using termui correctly. You need to call it's Loop function to start the Event loop so that it can actually start listening for the q key. Loop is called last because it essentially takes control of the main goroutine from then on until StopLoop is called and it quits.
In order to stop the goroutines, it is common to have a "stop" channel. Usually it is a chan struct{} to save memory because you don't ever have to put anything in it. Wherever you want the goroutine to possibly stop and shutoff (or do something else perhaps), you use a select statement with the channels you are using. This select is ordered, so it will take from them in order unless they block, in which case it tries the next one, so the stop channel usually goes first. The stop channel normally blocks, but to get it to take this path, simply close()ing it will cause this path to be chosen in the select. So we close() it in the q keyboard handler.
package main
import (
"fmt"
"github.com/gizak/termui"
"strconv"
"time"
)
func getData(ch chan string, stop chan struct{}) {
i := 0
for {
select {
case <-stop:
break
case ch <- strconv.Itoa(i):
}
i++
time.Sleep(time.Second)
if i == 20 {
break
}
}
}
func Display(ch chan string, stop chan struct{}) {
for {
var elem string
select {
case <-stop:
break
case elem = <-ch:
}
par := termui.NewPar(elem)
par.Height = 5
par.Width = 37
par.Y = 4
par.BorderLabel = "term ui example with chan"
par.BorderFg = termui.ColorYellow
termui.Render(par)
}
}
func main() {
ch := make(chan string)
stop := make(chan struct{})
err := termui.Init()
if err != nil {
panic(err)
}
defer termui.Close()
termui.Handle("/sys/kbd/q", func(termui.Event) {
fmt.Println("q captured")
close(stop)
termui.StopLoop()
})
go getData(ch, stop)
go Display(ch, stop)
termui.Loop()
}

Resources