For a personal project, I'm spawning an external process which hosts a REST API.
I only want to pass control back to the main thread once the external process has been initialized, which can be known by reading Stdout.
So I have created an io.Writer implementation which closes a channel once a certain criteria has been met.
type channelIOWriter struct {
InitializedChannel chan bool
isInitialized bool
buffer string
}
func newChannelIOWriter() *channelIOWriter {
retVal := new(channelIOWriter)
retVal.InitializedChannel = make(chan bool)
return retVal
}
func (writer *channelIOWriter) Write(data []byte) (int, error) {
for _, b := range data {
if b == '\n' {
writer.buffer = *bytes.NewBuffer(make([]byte, 0))
continue
}
writer.buffer.WriteByte(b)
if strings.HasPrefix(writer.buffer.String(), "ChromeDriver was started successfully.") {
if !writer.isInitialized {
close(writer.InitializedChannel)
}
writer.isInitialized = true
}
}
return len(data), nil
}
Next, I have the main function which, in a seperate goroutine, spawns the external process.
A wait is performed on the channel that's suposed to be closed by the io.Writer implementation.
func main() {
writer := newChannelIOWriter()
go func() {
cmd := exec.Command("chromedriver", "--port=9009")
cmd.Stdout = writer
cmd.Start()
}()
<-writer.InitializedChannel
fmt.Println("Control is passed back to the MAIN thread.")
}
According to the comments on the question, it seems to be better to use a context.
Anyone who can explain on how to use that?
I stripped out all unnecessary detail and demo how context with deadline works
package main
import (
"context"
"fmt"
"os/exec"
"time"
)
type example struct {
cancel context.CancelFunc
}
func (e *example) Write(data []byte) (int, error) {
defer e.cancel()
return len(data), nil
}
func main() {
deadline := time.Now().Add(5 * time.Second)
ctx, cancel := context.WithDeadline(context.Background(), deadline)
writer := &example{
cancel: cancel,
}
go func() {
cmd := exec.Command("ls", ".")
cmd.Stdout = writer
_ = cmd.Start()
}()
<-ctx.Done()
fmt.Println("Control is passed back to the MAIN thread.", ctx.Err())
}
Related
I've written this code for education purposes, now I need to write some test for it.
First, I need to test method Worker, but I don't understand how can I do it properly?
I'm totally new to test and to Go generally.
package worker
import (
"context"
"fmt"
)
type Queue interface {
TakeMessage() (<-chan string, error)
}
type Download interface {
Download(url string) error
}
type Worker struct {
queue Queue
download Download
}
func NewWorker(queue Queue, download Download) *Worker {
newWorker := Worker{}
newWorker.queue = queue
newWorker.download = download
return &newWorker
}
func (w *Worker) Worker(ctx context.Context) error {
msgs, err := w.queue.TakeMessage()
if err != nil {
return fmt.Errorf("error while consume queue: %w", err)
}
for {
select {
case <-ctx.Done():
return nil
case msg := <-msgs:
fmt.Println(msg)
w.download.Download(msg)
}
}
}
I write some lines of testing code and I suppossed to check if Worker return nil when ctx.Done() is passed to channel Now this test doesn't work, it's stuck maybe because of infinite cycle in worker method. Then I need to check if method takes messages from queue and pass it to Download method.
package worker
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
)
type MockQueue struct {
mock.Mock
}
type MockDownloader struct {
mock.Mock
}
func (m *MockQueue) TakeMessage() (<-chan string, error) {
strCh := make(chan string)
strCh <- "some_url/some.txt"
return strCh, nil
}
func (d *MockDownloader) Download(url string) error {
if url == "some_url/some.txt" {
return nil
} else {
return errors.New(url)
}
}
func TestWorkerCloseContext(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
resultCh := make(chan error)
newQueue := &MockQueue{}
newDownload := &MockDownloader{}
newWorker := Worker{newQueue, newDownload}
go func() {
resultCh <- newWorker.Worker(ctx)
}()
cancel()
assert.Nil(t, <-resultCh)
}
func TestWorkerMessageReceive(t \*testing.T) {
ctx, cancel := context.WithCancel(context.Background())
resultCh := make(chan error)
newQueue := &MockQueue{}
newDownload := &MockDownloader{}
newWorker := Worker{newQueue, newDownload}
go func() {
resultCh \<- newWorker.Worker(ctx)
}()
//some code here
}
Go comes with a pretty extensive built-in testing library. See https://pkg.go.dev/testing
I would recommend looking at the below tutorials:
https://go.dev/blog/cover
https://blog.alexellis.io/golang-writing-unit-tests/
https://dev.to/quii/learn-go-by-writing-tests-structs-methods-interfaces--table-driven-tests-1p01
I am playing around with worker pools and i want to consolidate all errors from worker pools before i return the error. I've written a sample code but i am entering a deadlock.
What am i trying to achieve?
a client send 100 requests,i want to first add those requests to a job queue and dispatch it to n number of go routines that does tasks in background , if at all there are errors i want to accumulate all these errors before i send all errors to the client. I have written a snippet, can someone explain what's wrong and how to mitigate the deadlock.
package main
import (
"context"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/apex/log"
"github.com/hashicorp/go-multierror"
)
type Manager struct {
taskChan chan int
wg *sync.WaitGroup
QuitChan chan bool
ErrorChan chan error
busyWorkers int64
}
func main() {
fmt.Println("Hello, 世界")
m := New()
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
//defer cancel()
for i := 0; i < 3; i++ {
m.wg.Add(1)
go m.run(ctx, test)
}
for i := 1; i < 5; i++ {
m.taskChan <- i
}
close(m.taskChan)
go func(*Manager) {
if len(m.taskChan) == 0 {
m.QuitChan <- true
}
}(m)
var errors error
for {
select {
case err := <-m.ErrorChan:
errors = multierror.Append(errors, err)
if m.busyWorkers == int64(0) {
break
}
default:
fmt.Println("hello")
}
}
m.wg.Wait()
fmt.Println(errors)
}
func New() *Manager {
return &Manager{taskChan: make(chan int),
wg: new(sync.WaitGroup),
QuitChan: make(chan bool),
ErrorChan: make(chan error),
}
}
func (m *Manager) run(ctx context.Context, fn func(a, b int) error) {
defer m.wg.Done()
defer fmt.Println("finished working")
for {
select {
case t, ok := <-m.taskChan:
if ok {
atomic.AddInt64(&m.busyWorkers, 1)
err := fn(t, t)
if err != nil {
m.ErrorChan <- err
}
atomic.AddInt64(&m.busyWorkers, -1)
}
case <-ctx.Done():
log.Infof("closing channel %v", ctx.Err())
return
case <-m.QuitChan:
return
}
}
}
// this can return error or not, this is the main driver func, but i'm propagating
//errors so that i can understand where i am going wrong
func test(a, b int) error {
fmt.Println(a, b)
return fmt.Errorf("dummy error %v", a)
}
You have 3 workers who all return errors.
Your main thread tries to put 5 jobs in the queue. Once the first 3 has been taken by your workers, the main thread is stuck waiting for a new worker to receive on taskChan and all your 3 workers are stuck trying to send data on ErrorChan.
In other words, deadlock.
Maybe you wanted to make taskChan a buffered channel? That way you can send data on it until the buffer is full without blocking.
taskChan: make(chan int, 10)
I am new to golang tried but not getting in my mind. I wanted to execute a shell command and then return the error (if exists) as a channel or stdoutput as a channel.
So far I have done this:
package main
import (
"bufio"
"io"
"os"
"os/exec"
)
type CommandRequest struct {
Command string
Args string
}
type CommandResponse struct {
StdOut chan string
StdErr chan string
}
func main() {
ExecuteCommand();
}
func ExecuteCommand() CommandResponse {
cmd := exec.Command("ping", "192.168.0.1")
_, err := cmd.Output()
var returnValue CommandResponse
if err != nil {
output := make(chan string)
go func() { output <- read(os.Stdout) }()
returnValue = CommandResponse{output, nil} //Reading from Stdin
}
return returnValue
}
func read(r io.Reader) <-chan string {
lines := make(chan string)
go func() {
defer close(lines)
scan := bufio.NewScanner(r)
for scan.Scan() {
lines <- scan.Text()
}
}()
return lines
}
Playground Link https://play.golang.org/p/pJ2R6fzK8gR
I tried as much as possible to reduce the error and what is left same i am getting in my workspace as well
There are a couple of ways to read from a command output.
1. Output Method
This is the easiest one. Get directly from Output
func main() {
out, _ := exec.Command("./ping", "192.168.0.1").Output()
fmt.Printf("Stdout: %s\n", string(out))
}
2. Redirect Outputs to bytes.Buffer
func main() {
var stdout, stderr bytes.Buffer
cmd := exec.Command("./ping", "192.168.0.1")
cmd.Stdout = &stdout
cmd.Stderr = &stderr
_ := cmd.Run()
fmt.Printf("Stdout: %s\n", stdout.String())
fmt.Printf("Stderr: %s\n", stderr.String())
}
2. Stdout Pipe with Channel
We generally do not return channel as in your code because their usage to hold and move data is different then variables. We pass them as argument to functions and read/write those channels from different processes (goroutines).
You can use StdoutPipe() method to get output from a pipe as io.Readcloser. Then read it with bufio.Scanner. As you can see I used channel here.
func main() {
cmd := exec.Command("./ping", "192.168.0.1")
cmdReader, _ := cmd.StdoutPipe()
scanner := bufio.NewScanner(cmdReader)
out := make(chan string)
go reader(scanner, out)
done := make(chan bool)
go func() {
value := <-out
println(value)
done <- true
}()
_ = cmd.Run()
<-done
}
func reader(scanner *bufio.Scanner, out chan string) {
for scanner.Scan() {
out <- scanner.Text()
}
}
Channels are the pipes that connect concurrent goroutines. You can send values into channels from one goroutine and receive those values into another goroutine.
If buffered channel is not used, sends and receives block until the other side is ready.
Returning Channel may not be appropriate channel usage.
There are two options to solve your question
Option1 : Not to use the channel and just simply return CommandResponse.
package main
import (
"fmt"
"os/exec"
)
type CommandRequest struct {
Command string
Args string
}
func main() {
cr := CommandRequest{"ls", "-la"}
fmt.Println(ExecuteCommand(cr))
}
func ExecuteCommand(cr CommandRequest) string{
cmd := exec.Command(cr.Command, cr.Args)
stdout, err := cmd.Output()
if err != nil {
return err.Error()
}
return string(stdout)
}
Option2: Use channel but without return. Create a channel and pass it as argument and keep blocked on the receive in main(). So the ExecuteCommand() could execute the shell command and send the return status through the channel.
package main
import (
"fmt"
"os/exec"
)
type CommandRequest struct {
Command string
Args string
}
func main() {
cr := CommandRequest{"ls", "-la"}
retc := make(chan string)
go ExecuteCommand(cr, retc)
ret := <-retc
fmt.Println(ret)
}
func ExecuteCommand(cr CommandRequest, retc chan string) {
cmd := exec.Command(cr.Command, cr.Args)
stdout, err := cmd.Output()
if err != nil {
retc <- err.Error()
return
}
retc <- string(stdout)
}
I need to update values from a struct and return (read-only) not writing from an HTTP handler, to avoid having race conditions I am using sync.Mutex this is a basic example:
http://play.golang.org/p/21IimsdKP6e
package main
import (
"encoding/json"
"log"
"net/http"
"sync"
"time"
)
type Counter struct {
count uint
flag bool
mu sync.Mutex
quit chan struct{}
time time.Time
wg sync.WaitGroup
}
func (c *Counter) Start() {
c.count = 1
c.time = time.Now()
c.flag = true
}
func (c *Counter) Listen() {
srv := &http.Server{
Addr: ":8080",
Handler: http.DefaultServeMux,
}
http.HandleFunc("/", c.HandleStatus)
c.wg.Add(1)
go func() {
defer c.wg.Done()
log.Println(srv.ListenAndServe())
}()
go func(quit chan struct{}) {
<-quit
if err := srv.Close(); err != nil {
log.Printf("HTTP error: %v", err)
}
}(c.quit)
}
func (c *Counter) HandleStatus(w http.ResponseWriter, r *http.Request) {
c.mu.Lock()
defer c.mu.Unlock()
status := struct {
Count uint `json:"count"`
Flag bool `json:"flag"`
Time string `json:"time"`
}{
Count: c.count,
Time: c.time.UTC().Format(time.RFC3339),
Flag: c.flag,
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(status); err != nil {
log.Println(err)
}
}
func main() {
c := &Counter{
quit: make(chan struct{}),
}
c.Start()
c.Listen()
timeout := time.After(time.Minute)
for {
select {
case <-time.After(time.Second):
c.mu.Lock()
c.count += 1
c.flag = !c.flag
c.mu.Unlock()
case <-timeout:
close(c.quit)
c.wg.Wait()
return
}
}
}
I have more handlers but does where I need to read from the Counter struct I need to add on each:
c.mu.Lock()
defer c.mu.Unlock()
And where the variables are modified, do something like:
c.mu.Lock()
c.count += 1
c.flag = !c.flag
c.mu.Unlock()
Therefore wondering how could I better syncronize/arrange the code to prevent adding on each handler Lock/Unlock
You have a couple of options:
Use a mutex, as in your existing example (though a RWMutex will be more efficient if most of the accesses are reads).
Wrap the same in a helper function, so that rather than having to operate the lock directly, your handlers can just call the function to get the value, and the function will handle the lock/read/unlock logic.
Turn the logic on its head, letting each routine have its own local copy of the value, and using a channel to notify routines of changes in the authoritative value (sharing by communicating instead of communicating by sharing).
I got a problem using sync.WaitGroup and select together. If you take a look at following http request pool you will notice that if an error occurs it will never be reported as wg.Done() will block and there is no read from the channel anymore.
package pool
import (
"fmt"
"log"
"net/http"
"sync"
)
var (
MaxPoolQueue = 100
MaxPoolWorker = 10
)
type Pool struct {
wg *sync.WaitGroup
queue chan *http.Request
errors chan error
}
func NewPool() *Pool {
return &Pool{
wg: &sync.WaitGroup{},
queue: make(chan *http.Request, MaxPoolQueue),
errors: make(chan error),
}
}
func (p *Pool) Add(r *http.Request) {
p.wg.Add(1)
p.queue <- r
}
func (p *Pool) Run() error {
for i := 0; i < MaxPoolWorker; i++ {
go p.doWork()
}
select {
case err := <-p.errors:
return err
default:
p.wg.Wait()
}
return nil
}
func (p *Pool) doWork() {
for r := range p.queue {
fmt.Printf("Request to %s\n", r.Host)
p.wg.Done()
_, err := http.DefaultClient.Do(r)
if err != nil {
log.Fatal(err)
p.errors <- err
} else {
fmt.Printf("no error\n")
}
}
}
Source can be found here
How can I still use WaitGroup but also get errors from go routines?
Just got the answer my self as I wrote the question and as I think it is an interesting case I would like to share it with you.
The trick to use sync.WaitGroup and chan together is that we wrap:
select {
case err := <-p.errors:
return err
default:
p.wg.Done()
}
Together in a for loop:
for {
select {
case err := <-p.errors:
return err
default:
p.wg.Done()
}
}
In this case select will always check for errors and wait if nothing happens :)
It looks a bit like the fail-fast mechanism enabled by the Tomb library (Tomb V2 GoDoc):
The tomb package handles clean goroutine tracking and termination.
If any of the tracked goroutines returns a non-nil error, or the Kill or Killf method is called by any goroutine in the system (tracked or not), the tomb Err is set, Alive is set to false, and the Dying channel is closed to flag that all tracked goroutines are supposed to willingly terminate as soon as possible.
Once all tracked goroutines terminate, the Dead channel is closed, and Wait unblocks and returns the first non-nil error presented to the tomb via a result or an explicit Kill or Killf method call, or nil if there were no errors.
You can see an example in this playground:
(extract)
// start runs all the given functions concurrently
// until either they all complete or one returns an
// error, in which case it returns that error.
//
// The functions are passed a channel which will be closed
// when the function should stop.
func start(funcs []func(stop <-chan struct{}) error) error {
var tomb tomb.Tomb
var wg sync.WaitGroup
allDone := make(chan struct{})
// Start all the functions.
for _, f := range funcs {
f := f
wg.Add(1)
go func() {
defer wg.Done()
if err := f(tomb.Dying()); err != nil {
tomb.Kill(err)
}
}()
}
// Start a goroutine to wait for them all to finish.
go func() {
wg.Wait()
close(allDone)
}()
// Wait for them all to finish, or one to fail
select {
case <-allDone:
case <-tomb.Dying():
}
tomb.Done()
return tomb.Err()
}
A simpler implementation would be like below. (Check in play.golang: https://play.golang.org/p/TYxxsDRt5Wu)
package main
import "fmt"
import "sync"
import "time"
type Error struct {
message string
}
func (e Error) Error() string {
return e.message
}
func main() {
var wg sync.WaitGroup
waitGroupLength := 8
errChannel := make(chan error, 1)
// Setup waitgroup to match the number of go routines we'll launch off
wg.Add(waitGroupLength)
finished := make(chan bool, 1) // this along with wg.Wait() are why the error handling works and doesn't deadlock
for i := 0; i < waitGroupLength; i++ {
go func(i int) {
fmt.Printf("Go routine %d executed\n", i+1)
time.Sleep(time.Duration(waitGroupLength - i))
time.Sleep(0) // only here so the time import is needed
if i%4 == 1 {
errChannel <- Error{fmt.Sprintf("Errored on routine %d", i+1)}
}
// Mark the wait group as Done so it does not hang
wg.Done()
}(i)
}
go func() {
wg.Wait()
close(finished)
}()
L:
for {
select {
case <-finished:
break L // this will break from loop
case err := <-errChannel:
if err != nil {
fmt.Println("error ", err)
// handle your error
}
}
}
fmt.Println("Executed all go routines")
}