Synchronize value (counter) across goroutines - go

Question

I have a golang application that goes through pages of a website, and is supposed to download every link on the website. It looks something a little like this (I don't know the number of pages beforehand, so that is done synchronously):
page := 0
results := getPage(page)
c := make(chan *http.Response)
for len(results) > 0 {
for result := range results {
go myProxySwitcher.downloadChan(result.URL, c)
fmt.Println(myProxySwitcher.counter)
}
page++
results = getPage(page)
myProxySwitcher.counter++
}
The twist is, every 10 requests, I want to change the Proxy I use to connect to the website. To do this, I made a struct with a counter member:
type ProxySwitcher struct {
proxies []string
client *http.Client
counter int
}
And then I have incremented the counter each time a request is made from downloadChan.
func (p *ProxySwitcher) downloadChan(url string, c chan *http.Response) {
p.counter++
proxy := p.proxies[int(p.counter/10)%len(p.proxies]
res := p.client.Get(url, proxy)
c <- res
}
When it does the downloads, it doesn't appear the the counter is synchronized between goroutines. How can I sync the value of the counter between goroutines?
The result I get from those printlns are:
1
1
1
1
1
1
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
And I am expecting
1
2
3
4
5
...

Answer 1

You have a race condition in your code.
In the first snippet, you're modifying the counter field from the "main" goroutine:
// ...
myProxySwitcher.counter++
In the third snippet, you're also modifying that counter from a different goroutine:
// ...
p.counter++
This is illegal code in Go. By definition, the results are undefined. To understand why, you'll have to go over the Go Memory Model. Hint: it won't likely be an easy read.
To fix it, you need to ensure synchronization. There are many ways to do it.
One way, as suggested in a comment on your question, to do it is to use a mutex. Below is an example, kinda messy, as it would need some refactoring on the main loop. But this is how you would synchronize access to the counter:
type ProxySwitcher struct {
proxies []string
client *http.Client
mu sync.Mutex
counter int
}
func (p *ProxySwitcher) downloadChan(url string, c chan *http.Response) {
p.mu.Lock()
p.counter++
// gotta read it from p while holding
// the lock to use it below
counter := p.counter
p.mu.Unlock()
// here you use counter rather than p.counter,
// since you don't hold the lock anymore
proxy := p.proxies[int(counter/10)%len(p.proxies)]
res := p.client.Get(url, proxy)
c <- res
}
// ... the loop ...
for len(results) > 0 {
for result := range results {
go myProxySwitcher.downloadChan(result.URL, c)
// this is kinda messy, would need some heavier
// refactoring, but this should fix the race:
myProxySwitcher.mu.Lock()
fmt.Println(myProxySwitcher.counter)
myProxySwitcher.mu.Unlock()
}
page++
results = getPage(page)
// same... it's messy, needs refactoring
myProxySwitcher.mu.Lock()
myProxySwitcher.counter++
myProxySwitcher.mu.Unlock()
}
Alternatively, you could change that counter to e.g. uint64, and then use the atomic/sync package to perform goroutine-safe operations:
type ProxySwitcher struct {
proxies []string
client *http.Client
counter uint64
}
func (p *ProxySwitcher) downloadChan(url string, c chan *http.Response) {
counter := atomic.AddUint64(&p.counter, 1)
// here you use counter rather than p.counter, since that's your local copy
proxy := p.proxies[int(counter/10)%len(p.proxies)]
res := p.client.Get(url, proxy)
c <- res
}
// ... the loop ...
for len(results) > 0 {
for result := range results {
go myProxySwitcher.downloadChan(result.URL, c)
counter := atomic.LoadUint64(&myProxySwitcher.counter)
fmt.Println(counter)
}
page++
results = getPage(page)
atomic.AddUint64(&myProxySwitcher.counter, 1)
}
I'd probably use this last version, as it's cleaner and we don't really need a mutex.