Slice append from channels - go

I want to create slice and add values returned from channel.
Below is the code I tried but could not able to solve it.
I have to send address of the slice, but I am not able to figure out how :(
package main
import "fmt"
import "time"
func sendvalues(cs chan int){
for i:=0;i<10;i++{
cs<-i
}
}
func appendInt(cs chan int, aINt []int)[]*int{
for {
select {
case i := <-cs:
aINt = append(aINt,i)//append returns new type right ?
fmt.Println("slice",aINt)
}
}
}
func main() {
cs := make(chan int)
intSlice := make([]int, 0,10)
fmt.Println("Before",intSlice)
go sendvalues(cs)
go appendInt(cs,intSlice)// I have to pass address here
time.Sleep(999*999999)
fmt.Println("After",intSlice)
}

Your code won't work for two (in fact three) reasons:
append returns a new slice as soon as the capacity is reached.
Thus, the assignment in appendInt will do nothing.
appendInt runs concurrently, therefore:
As long as appendInt does not message main that it is finished,
main does not know when the intSlice has all the values you want.
You have to wait for all goroutines to return at the end of main
Problem 1: Modifying slices in functions
You may know that in Go every value you pass to a function is copied. Reference values,
such as slices, are copied too, but have pointers internally which then point to the original memory location. That means you can modify the elements of a slice in a function. What you
can't do is reassigning this value with a new slice as the internal pointer would point to somewhere different. You need pointers for that. Example (Play):
func modify(s *[]int) {
for i:=0; i < 10; i++ {
*s = append(*s, i)
}
}
func main() {
s := []int{1,2,3}
modify(&s)
fmt.Println(s)
}
Problem 2: Synchronizing goroutines
To wait for started goroutines, you can use a sync.WaitGroup. Example (Play):
func modify(wg *sync.WaitGroup, s *[]int) {
defer wg.Done()
for i:=0; i < 10; i++ {
*s = append(*s, i)
}
}
func main() {
wg := &sync.WaitGroup{}
s := []int{1,2,3}
wg.Add(1)
go modify(wg, &s)
wg.Wait()
fmt.Println(s)
}
The example above waits (using wg.Wait()) for modify to finish
(modify calls wg.Done() when finished). If you remove the wg.Wait() call, you will
see why not synchronizing is a problem. Comparison of outputs:
With wg.Wait(): [1 2 3 0 1 2 3 4 5 6 7 8 9]
Without wg.Wait(): [1 2 3]
The main goroutine returns earlier than the modify goroutine which is why you will never
see the modified results. Therefore synchronizing is absolutely necessary.
A good way to communicate the new slice would be to use a channel. You would not need to
use pointers and you would have synchronization. Example (Play):
func modify(res chan []int) {
s := []int{}
for i:=0; i < 10; i++ {
s = append(s, i)
}
res <- s
}
func main() {
c := make(chan []int)
go modify(c)
s := <-c
fmt.Println(s)
}

Related

Add items to channel in one goroutine and process in another

Really new to the world of go and trying to learn by doing. I want to see how I would use go routines to add items to a "queue" in one go routine while another go routine listens for the queue and processes things as they enter. In this case I have a func that adds a defined amount of items to a int[] while the other attempts to print them as they are added.. I assume a flag will need to be sent to signal that one go routine has stopped adding items to the queue. I apologize for the noob question but am struggling understanding the new terminology and syntax.
package main
import "fmt"
func printFromSlice(c chan []int){
i := <-c
// wait for item to be added to channel
fmt.Print(i)
}
func addToSlice (c chan []int, i int) {
for n := 0; n < i; n++{
c <- n //I know this isnt right
}
// How do we signal this is complete??
}
func main(){
c := make(chan []int)
//add things to []i int
go addToSlice(c, 25)
//do something from []int
go printFromSlice(c)
}
**UPDATE**
modified to use the following code however now it will only execute to a single print form the ~printFromSlice` function before it closes out...
package main
func printFromSlice(c chan int){
i := <-c
// wait for item to be added to channel
println("read")
println(i)
}
func addToSlice (c chan int) {
for n := 0; n < 100; n++{
println("add")
println(n)
c <- n
}
close(c)
}
func main(){
println("starting...")
c := make(chan int)
//add things to []i int
go addToSlice(c)
//do something from []int
go printFromSlice(c)
<-c
}
You need to add a sync.WaitGroup to block the main thread until the two goroutine could finish. You could refer to this Go by Example.
package main
import "sync"
func printFromSlice(c chan int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement the waitgroup counter by 1 after `printFromSlice` returns
// wait for item to be added to channel
// i := <-c // this only waits / blocks 1 time
// For each message arriving at c, read and print
for i := range c { // this would read all messages until channel is closed
println("read")
println(i)
}
}
func addToSlice(c chan int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement the waitgroup counter by 1 after `addToSlice` returns
for n := 0; n < 100; n++ {
println("add")
println(n)
c <- n
}
close(c)
}
func main() {
var wg sync.WaitGroup
println("starting...")
c := make(chan int)
wg.Add(2) // Adds 2 to the waitgroup counter
//add things to []i int
go addToSlice(c, &wg) // Pass the wait group as reference so they can call wg.Done()
//do something from []int
go printFromSlice(c, &wg) // Pass the wait group as reference so they can call wg.Done()
// <-c // No need for this to block the code
wg.Wait() // Waits / blocks until waitgroup counter is 0
}

Issues with chans and waitgroups

While I've seen similar issues, none of the answers found on SO have helped me, but I hope a kind soul with an eagle eye can help me pinpoint the issue here. I know it's not the best use of goroutines but I wanted to do it this way as an exercise, which has obviously failed.
my code
package main
import (
"fmt"
"sort"
"sync"
)
func main() {
X := []int{1, 2, 3, 4, 0}
Y := []int{2, 3, 6, 8, 4}
solution := Solution(X, Y)
fmt.Println(solution)
}
//Solution solution
func Solution(X []int, Y []int) int {
size := len(X)
resultChan := make(chan int)
results := make(map[int]int)
ParseDivision(size, X, Y, resultChan)
for val := range resultChan {
results[val] = results[val] + 1
}
close(resultChan)
return FindGreatest(results)
}
//Divide divide
func Divide(a int, b int, resultChan chan<- int, wg *sync.WaitGroup) {
defer wg.Done()
result := float64(a) / float64(b)
resultChan <- int(result * 1000)
}
//FindGreatest find greatest in map
func FindGreatest(myMap map[int]int) int {
values := make([]int, 0, len(myMap))
for _, val := range myMap {
values = append(values, val)
}
sort.Ints(values)
return values[len(values)-1]
}
//ParseDivision int
func ParseDivision(lenth int, X []int, Y []int, resultChan chan<- int) {
var wg sync.WaitGroup
wg.Add(lenth)
for i := 0; i < lenth; i++ {
go Divide(X[i], Y[i], resultChan, &wg)
}
wg.Wait()
}
Result: fatal error: all goroutines are asleep - deadlock!
I am not at all sure why as I've followed a number of examples as well as answers from SO, with respect to passing the waitGroup by reference, as well as using a channel to get the result of the operations performed in goroutines.
Some things to note:
the go-routine that writes to a channel should typically close the channel;
and following from above, the "reader" go-routine should wait for the channel to close (via range et.c) - i.e. it should never have to close the channel its reading from.
since your results channel is unbuffered (buffer size of 0), the main go-routine blocks waiting as the workers try to write, and the main go-routine is not (yet) reading from the results channel (as the pool manager go-routine must complete for this to happen - DEADLOCK!). Solution:
ensure the work pool go-routine runs independent of the main (reader) go-routine.
3-line fix: https://play.golang.org/p/9hYuyDgMjGi

Is copy concurrency safe in golang?

In some circumstance, i would copy some content to different piece of the slice.
Like this
a := make([]int, 10)
for i := 0; i < 10; i++ {
b := []int{i}
go func(i int) {
copy(a[i:i+1], b)
}(i)
}
time.Sleep(time.Second)
fmt.Println(a)
It leads DATA RACE. But it always behave right in product environment.
So my question is:
Any data race cloud be undefined behavior?
Can i always get right result in such a practice?
To avoid the data race, which will have undefined results, synchronize. For example,
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
a := make([]int, 10)
for i := 0; i < 10; i++ {
b := []int{i}
wg.Add(1)
go func(i int) {
defer wg.Done()
copy(a[i:i+1], b)
}(i)
}
wg.Wait()
fmt.Println(a)
}
Playground: https://play.golang.org/p/rYCBMV7wuNn
Output:
$ go run -race norace.go
[0 1 2 3 4 5 6 7 8 9]
$
Package sync
import "sync"
type WaitGroup
A WaitGroup waits for a collection of goroutines to finish. The main
goroutine calls Add to set the number of goroutines to wait for. Then
each of the goroutines runs and calls Done when finished. At the same
time, Wait can be used to block until all goroutines have finished.
A WaitGroup must not be copied after first use.
type WaitGroup struct {
// contains filtered or unexported fields
}

Go: transformed channel

Let's say I have an int channel in Go:
theint := make(chan int)
I want to wrap this channel in a new channel called incremented
incremented := make(chan int)
Such that:
go func() { theint <- 1 }
<- incremented // 2
appended can be assumed to be the only one that reads from the int.
It will work if a run a goroutine in the background
go func() {
for num := range theint {
incremented <- num + 1
}
}
However, I prefer to do it without a goroutine since I can't control it in my context.
Is there a simpler way to do it?
One thing that came to mind is python's yield:
for num in theint:
yield num + 1
Is something like this possible in go?
Generator pattern
What you are trying to implement is generator pattern. To use channels and goroutines for implementation of this pattern is totally common practice.
However, I prefer to do it without a goroutine since I can't control it in my context.
I believe the problem is deadlock
fatal error: all goroutines are asleep - deadlock!
To avoid deadlocks and orphaned (not closed) channels use sync.WaitGroup. This is an idiomatic way to control goroutines.
Playground
package main
import (
"fmt"
"sync"
)
func incGenerator(n []int) chan int {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(len(n))
for _, i := range n {
incremented := i + 1
go func() {
wg.Done()
ch <- incremented
}()
}
go func() {
wg.Wait()
close(ch)
}()
return ch
}
func main() {
n := []int{1, 2, 3, 4, 5}
for x := range incGenerator(n) {
fmt.Println(x)
}
}
One thing you can also consider is having a select on the int channel and an exit channel - in an infinite for loop. You can choose a variable increment value too. Please see code below:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var accum int //accumulator of incremented values
var wg sync.WaitGroup
c1 := make(chan int)
exChan := make(chan bool)
wg.Add(1)
go func() {
time.Sleep(time.Second * 1)
c1 <- 1
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 2
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 5
wg.Done()
}()
go func() {
wg.Wait()
close(exChan)
}()
for {
var done bool
select {
case incBy := <-c1: //Increment by value in channel
accum += incBy
fmt.Println("Received value to increment:", incBy, "; Accumulated value is", accum)
case d := <-exChan:
done = !(d)
}
if done == true {
break
}
}
fmt.Println("Final accumulated value is", accum)
}
Playground: https://play.golang.org/p/HmdRmMCN7U
Exit channel not needed, if we are having non-zero increments always. I like #I159 's approach too!
Anyways, hope this helps.

How to break out of select gracefuly in golang

I have a program in golang that counts SHA1s and prints ones that start with two zeros. I want to use goroutines and channels. My problem is that I don't know how to gracefully exit select clause if I don't know how many results it will produce.
Many tutorials know that in advance and exit when counter hits. Other suggest using WaitGroups, but I don't want to do that: I want to print results in main thread as soon it appears in channel. Some suggest to close a channel when goroutines are finished, but I want to close it after asynchronous for finishes, so I don't know how.
Please help me to achieve my requirements:
package main
import (
"crypto/sha1"
"fmt"
"time"
"runtime"
"math/rand"
)
type Hash struct {
message string
hash [sha1.Size]byte
}
var counter int = 0
var max int = 100000
var channel = make(chan Hash)
var source = rand.NewSource(time.Now().UnixNano())
var generator = rand.New(source)
func main() {
nCPU := runtime.NumCPU()
runtime.GOMAXPROCS(nCPU)
fmt.Println("Number of CPUs: ", nCPU)
start := time.Now()
for i := 0 ; i < max ; i++ {
go func(j int) {
count(j)
}(i)
}
// close channel here? I can't because asynchronous producers work now
for {
select {
// how to stop receiving if there are no producers left?
case hash := <- channel:
fmt.Printf("Hash is %v\n ", hash)
}
}
fmt.Printf("Count of %v sha1 took %v\n", max, time.Since(start))
}
func count(i int) {
random := fmt.Sprintf("This is a test %v", generator.Int())
hash := sha1.Sum([]byte(random))
if (hash[0] == 0 && hash[1] == 0) {
channel <- Hash{random, hash}
}
}
Firstly: if you don't know when your computation ends, how could you even model it? Make sure you know exactly when and under what circumstances your program terminates. If you're done you know how to write it in code.
You're basically dealing with a producer-consumer problem. A standard case. I would model
that this way (on play):
Producer
func producer(max int, out chan<- Hash, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < max; i++ {
random := fmt.Sprintf("This is a test %v", rand.Int())
hash := sha1.Sum([]byte(random))
if hash[0] == 0 && hash[1] == 0 {
out <- Hash{random, hash}
}
}
close(out)
}
Obviously you're brute-forcing hashes, so the end is reached when the loop is finished.
We can close the channel here and signal the other goroutines that there is nothing more to listen for.
Consumer
func consumer(max int, in <-chan Hash, wg *sync.WaitGroup) {
defer wg.Done()
for {
hash, ok := <-in
if !ok {
break
}
fmt.Printf("Hash is %v\n ", hash)
}
}
The consumer takes all the incoming messages from the in channel and checks if it was closed (ok).
If it is closed, we're done. Otherwise print the received hashes.
Main
To start this all up we can write:
wg := &sync.WaitGroup{}
c := make(chan Hash)
wg.Add(1)
go producer(max, c, wg)
wg.Add(1)
go consumer(max, c, wg)
wg.Wait()
The WaitGroup's purpose is to wait until the spawned goroutines finished, signalled by
the call of wg.Done in the goroutines.
Sidenote
Also note that the Rand you're using is not safe for concurrent access. Use the one initialized
globally in math/rand. Example:
rand.Seed(time.Now().UnixNano())
rand.Int()
The structure of your program should probably be re-examined.
Here is a working example of what I presume you are looking for.
It can be run on the Go playground
package main
import (
"crypto/sha1"
"fmt"
"math/rand"
"runtime"
"time"
)
type Hash struct {
message string
hash [sha1.Size]byte
}
const Max int = 100000
func main() {
nCPU := runtime.NumCPU()
runtime.GOMAXPROCS(nCPU)
fmt.Println("Number of CPUs: ", nCPU)
hashes := Generate()
start := time.Now()
for hash := range hashes {
fmt.Printf("Hash is %v\n ", hash)
}
fmt.Printf("Count of %v sha1 took %v\n", Max, time.Since(start))
}
func Generate() <-chan Hash {
c := make(chan Hash, 1)
go func() {
defer close(c)
source := rand.NewSource(time.Now().UnixNano())
generator := rand.New(source)
for i := 0; i < Max; i++ {
random := fmt.Sprintf("This is a test %v", generator.Int())
hash := sha1.Sum([]byte(random))
if hash[0] == 0 && hash[1] == 0 {
c <- Hash{random, hash}
}
}
}()
return c
}
Edit: This does not fire up a separate routine for each Hash computation,
but to be honest, I fail to see the value on doing so. The scheduling of all those routines will likely cost you far more than running the code in a single routine.
If need be, you can split it up into chunks of N routines, but a 1:1 mapping is not the way to go with this.

Resources