I'm pretty new in Go and need the answers to some of dilemmas while implementing small HTTP notifier than runs concurrently, sending message to the configured HTTP endpoint. For that purpose, I use the following structure:
type Notifier struct {
url string
waitingQueue *list.List
progress *list.List
errs chan NotificationError
active sync.WaitGroup
mu sync.Mutex
}
Here is the notify method:
func (hn *Notifier) Notify(message string) {
if hn.ProcessingCount() < hn.config.MaxActiveRequestsCount() && hn.checkIfOlderThanOldestWaiting(time.Now().Unix()) {
element := hn.addToProgressQueue(message)
hn.active.Add(1)
go hn.sendNotification(element)
} else {
hn.waitingQueue.PushBack(QueueItem{message, time.Now().UnixNano()})
}
}
And addToInProgressQueue:
func (hn *Notifier) addToProgressQueue(message string) *list.Element {
hn.mu.Lock()
defer hn.mu.Unlock()
queueItem := QueueItem{message, time.Now().UnixNano()}
element := hn.progress.PushBack(queueItem)
return element
}
I guess this won't work as expected for concurrent reads/writes of queue? Is it enought to use RWMutex instead of Mutex to ensure the locks are working properly?
The code of ProcessingCount
func (hn *Notifier) ProcessingCount() int {
hn.mu.Lock()
defer hn.mu.Unlock()
return hn.inProgress.Len()
}
Can there be a data race here?
Also, if you have some good resources on data race examples in Golang, it would be well appreciated.
Thanks in advance
Related
Do i need mutex in this case ?
I have a simple type with map of net.Conn
type ConnectionsHandler struct {
mutex sync.Mutex
connections map[string]net.Conn
authorizedConnections []string
}
And when i need to add or remove connection from different goroutines, i think i need to use mutex for thread safety
func (c *ConnectionsHandler) addConnection(conn net.Conn, id string) {
c.mutex.Lock()
c.connections[id] = conn
c.mutex.Unlock()
}
func (c *ConnectionsHandler) removeConnection(id string) {
c.mutex.Lock()
if conn, ok := c.connections[id]; ok {
conn.Close()
delete(c.connections, id)
}
c.mutex.Unlock()
}
But i also need for iterate over all connections and write to them, or write to single one
func (c *ConnectionsHandler) sendMessage(msg GameMessage) {
// write msg to specific single one
}
func (c *ConnectionsHandler) sendMessageToAll(msg GameMessage) {
// iterate over all connections and write msg to them
}
And the question is, do i need mutex lock in sendMessageToAll() when i call Write() on each net.Conn stored in map ??
A Race Condition occures when i'am running my code. It is a simple implementation of a concurrent safe storage. The Race Condition disappears when i change the reciever in get() method to (p *storageType). I'm confused. I need someone who could explain to me such a behaivior.
package main
type storageType struct {
fc chan func()
value int
}
func newStorage() *storageType {
p := storageType{
fc: make(chan func()),
}
go p.run()
return &p
}
func (p storageType) run() {
for {
(<-p.fc)()
}
}
func (p *storageType) set(s int) {
p.fc <- func() {
p.value = s
}
}
func (p storageType) get() int {
res := make(chan int)
p.fc <- func() {
res <- p.value
}
return <-res
}
func main() {
storage := newStorage()
for i := 0; i < 1000; i++ {
go storage.set(i)
go storage.get()
}
}
In main() the storage variable is of type *storageType. If storageType.Get() has value receiver, then storage.get() means (*storage).get().
The get() call has storageType as the reciver, so the storage pointer variable has to be dereferenced to make a copy (that will be used as the receiver value). This copying means the value of the pointed storageType struct must be read. But this read is not synchronized with the run() method which reads and writes the struct (its value field).
If you change the receiver of get() to be a pointer (of type *storageType), then the receiver again will be a copy, but this time it will be a copy of the pointer, not the pointed struct. So no unsynchronized read of the struct happens.
See possible duplicate: Why does the method of a struct that does not read/write its contents still cause a race case?
First one: your main function doesn't wait for all goroutines to finish. All goroutines are forced to return when main does.
Look into using a sync.WaitGroup
I have a chat application using 2 go routines. I would like to add/remove records to/from the list in one thread and read the same list from the other thread.
As I am pretty new in Go, I am a bit puzzled about what data structure should be used. I thought of slices, but not sure that I use it the right way
func listener(addr *net.UDPAddr, clients *[] *net.UDPAddr, messages chan clientMessage) {
for {
*clients=append(*clients,otherAddr)
}
}
func sender(messages chan clientMessage,clients *[] *net.UDPAddr) {
for {
message :=<- messages
for _,client := range *clients {
fmt.Printf("Message %s sent to %s\n", message.message, client.String())
}
}
}
func main() {
var clients [] *net.UDPAddr
go listener(s,&clients,messageCh)
go sender(messageCh,&clients)
}
Since listener only needs to write, and sender only needs to read - this is a good example of using channels to communicate. The flow would look like the following:
Listener would post the new client to the channel.
Sender will receive the new client and will update its local slice
of clients.
It will be a lot cleaner and safer this way - since listener will not be able to "accidentally" read and sender will not be able to "accidentally" write. Listener can also close the channel to indicate to the sender that it's done.
A slice is looks OK for the scenario, but a mutex is needed to prevent concurrent read and write to the slice.
Let's bundle the slice and mutex together in a struct and add methods for the two operations: add and enumerate.
type clients struct {
mu sync.Mutex
values []*net.UDPAddr
}
// add adds a new client
func (c *clients) add(value *net.UDPAddr) {
c.mu.Lock()
c.values = append(c.values, value)
c.mu.Unlock()
}
// do calls fn for each client
func (c *clients) do(fn func(*net.UDPAddr) error) error {
c.mu.Lock()
defer c.mu.Unlock()
for _, value := range c.values {
if err := fn(value); err != nil {
return err
}
}
return nil
}
Use it like this:
func listener(addr *net.UDPAddr, clients *clients, messages chan clientMessage) {
for {
clients.add(otherAddr)
}
}
func sender(messages chan clientMessage, clients *clients) {
for {
message := <-messages
clients.do(func(client *net.UDPAddr) error {
fmt.Printf("Message %s sent to %s\n", message.message, client.String())
return nil
})
}
}
func main() {
var clients clients
go listener(s, &clients, messageCh)
go sender(messageCh, &clients)
}
I'm trying to implement graceful channel closing from receiver side.
Yes, I'm aware that this violates the channel closing rule:
...don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders.
But I want to implement such logic. Unfortunately, I fail into deadlock issue in the number of cases: the application just hangs for the unlimited time, trying to lock same locked Mutex again.
So, I have 2 goroutines:
one that will write into a channel and
another that will receive data + will close channel from the receiver side.
My channel wrapped in the struct with sync.Mutex and closed boolean flag:
type Chan struct {
sync.Mutex // can be replaced with deadlock.Mutex from "github.com/sasha-s/go-deadlock"
data chan int
closed bool
}
All Send(), Close(), IsClosed() operations on this struct are guarded with Mutex and to prevent duplicate locking have the non-threadsafe method versions (send(), close(), isClosed()).
The full source code:
package main
import (
"log"
"net/http"
"sync"
)
func main() {
log.Println("Start")
ch := New(0) // unbuffered channel to expose problem faster
wg := sync.WaitGroup{}
wg.Add(2)
// send data:
go func(ch *Chan) {
for i := 0; i < 100; i++ {
ch.Send(i)
}
wg.Done()
}(ch)
// receive data and close from receiver side:
go func(ch *Chan) {
for data := range ch.data {
log.Printf("Received %d data", data)
// Bad practice: I want to close the channel from receiver's side:
if data > 50 {
ch.Close()
break
}
}
wg.Done()
}(ch)
wg.Wait()
log.Println("End")
}
type Chan struct {
deadlock.Mutex //sync.Mutex
data chan int
closed bool
}
func New(size int) *Chan {
defer func() {
log.Printf("Channel was created")
}()
return &Chan{
data: make(chan int, size),
}
}
func (c *Chan) Send(data int) {
c.Lock()
c.send(data)
c.Unlock()
}
func (c *Chan) Close() {
c.Lock()
c.close()
c.Unlock()
}
func (c *Chan) IsClosed() bool {
c.Lock()
defer c.Unlock()
return c.isClosed()
}
// send is internal non-threadsafe api.
func (c *Chan) send(data int) {
if !c.closed {
c.data <- data
log.Printf("Data %d was sent", data)
}
}
// close is internal non-threadsafe api.
func (c *Chan) close() {
if !c.closed {
close(c.data)
c.closed = true
log.Println("Channel was closed")
} else {
log.Println("Channel was already closed")
}
}
// isClosed is internal non-threadsafe api.
func (c *Chan) isClosed() bool {
return c.closed
}
You can run this program in the sandbox.
On the local machine, in small number of runs, after 30 seconds the output will be (using deadlock.Mutex instead of sync.Mutex):
2018/04/01 11:26:22 Data 50 was sent
2018/04/01 11:26:22 Received 50 data
2018/04/01 11:26:22 Data 51 was sent
2018/04/01 11:26:22 Received 51 data
POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 35 lock 0xc42015a040
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:71 main.(*Chan).Send { c.Lock() } <<<<<
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:30 main.main.func1 { ch.Send(i) }
Have been trying to lock it again for more than 30s
goroutine 36 lock 0xc42015a040
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:77 main.(*Chan).Close { c.Lock() } <<<<<
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:44 main.main.func2 { ch.Close() }
Why this deadlock happened and how to fix this implementation to avoid deadlocks?
Closing the channel on the sender's side is not the answer. So, this is not the fix for my question: Example of closing channel from sender side.
Send grabs the lock, then attempts to sends data down the channel. This may happen just after the 50th receive operation. There will be no more receives, so c.data <- data blocks forever and consequently the Mutex is held forever.
For cancellation, use another channel (instead of the boolean) and a select statement in Send. You can also leverage the context package.
You can try as hard as you like: you have to close the channel from sender side.
You might be able to get it working without a complete lockdown but you will leak goroutines. The sender will block forever and cannot be shut down. If the receiver wants to trigger a shutdown it has to tell the sender to shut the channel down. How you could tell the sender to shut down:
A boolean as you suggest (needs another mutex)
A stop-channel that when closed signals the sender to close the data channel (cannot be closed multiple times)
a ctx.Context: calling the cancel() function will signal the sender to stop. (Can be cancelled multiple times without worry)
(Only elaborating on Peters correct answer)
I'm learning go, and I would like to explore some patterns.
I would like to build a Registry component which maintains a map of some stuff, and I want to provide a serialized access to it:
Currently I ended up with something like this:
type JobRegistry struct {
submission chan JobRegistrySubmitRequest
listing chan JobRegistryListRequest
}
type JobRegistrySubmitRequest struct {
request JobSubmissionRequest
response chan Job
}
type JobRegistryListRequest struct {
response chan []Job
}
func NewJobRegistry() (this *JobRegistry) {
this = &JobRegistry{make(chan JobRegistrySubmitRequest, 10), make(chan JobRegistryListRequest, 10)}
go func() {
jobMap := make(map[string] Job)
for {
select {
case sub := <- this.submission:
job := MakeJob(sub.request) // ....
jobMap[job.Id] = job
sub.response <- job.Id
case list := <- this.listing:
res := make([]Job, 0, 100)
for _, v := range jobMap {
res = append(res, v)
}
list.response <- res
}
/// case somechannel....
}
}()
return
}
Basically, I encapsulate each operation inside a struct, which carries
the parameters and a response channel.
Then I created helper methods for end users:
func (this *JobRegistry) List() ([]Job, os.Error) {
res := make(chan []Job, 1)
req := JobRegistryListRequest{res}
this.listing <- req
return <-res, nil // todo: handle errors like timeouts
}
I decided to use a channel for each type of request in order to be type safe.
The problem I see with this approach are:
A lot of boilerplate code and a lot of places to modify when some param/return type changes
Have to do weird things like create yet another wrapper struct in order to return errors from within the handler goroutine. (If I understood correctly there are no tuples, and no way to send multiple values in a channel, like multi-valued returns)
So, I'm wondering whether all this makes sense, or rather just get back to good old locks.
I'm sure that somebody will find some clever way out using channels.
I'm not entirely sure I understand you, but I'll try answering never the less.
You want a generic service that executes jobs sent to it. You also might want the jobs to be serializable.
What we need is an interface that would define a generic job.
type Job interface {
Run()
Serialize(io.Writer)
}
func ReadJob(r io.Reader) {...}
type JobManager struct {
jobs map[int] Job
jobs_c chan Job
}
func NewJobManager (mgr *JobManager) {
mgr := &JobManager{make(map[int]Job),make(chan Job,JOB_QUEUE_SIZE)}
for {
j,ok := <- jobs_c
if !ok {break}
go j.Run()
}
}
type IntJob struct{...}
func (job *IntJob) GetOutChan() chan int {...}
func (job *IntJob) Run() {...}
func (job *IntJob) Serialize(o io.Writer) {...}
Much less code, and roughly as useful.
About signaling errors with an axillary stream, you can always use a helper function.
type IntChanWithErr struct {
c chan int
errc chan os.Error
}
func (ch *IntChanWithErr) Next() (v int,err os.Error) {
select {
case v := <- ch.c // not handling closed channel
case err := <- ch.errc
}
return
}