So, I'm trying to construct a websocket server in go. And i ran into this interesting bug, which i cant for the life of me figure out why its happening.
NOTE: The comments in the code snippets are there only for this post. Read them.
Ive got this function:
func Join(ws *websocket.Conn) {
Log.Connection(ws)
enc := json.NewEncoder(ws)
dec := json.NewDecoder(ws)
var dJ g.DiscussionJoin
var disc g.Discussion
Log.Err(dec.Decode(&dJ), "dec.Decode")
ssD := g.FindDiscussionByID(dJ.DiscussionID)
ssDJ := dJ.Convert(ws)
g.DiscHandle <- &ssDJ
disc = ssD.Convert()
Log.Err(enc.Encode(disc), "enc.Encode")
Log.Activity("Discussion", "Joined", disc.DiscussionID.Subject)
fmt.Println("Listening") //This gets called
g.Listen(dec)
fmt.Println("Stoped Listening") //This DOESN'T get called [IT SHOULD]
ssDJ.SSDiscussion.Leave(ssDJ.SSUserID)
Log.Disconnection(ws)
}
The function thats causing this is (in my opinion) g.Listen(...):
func Listen(dec *json.Decoder) {
timeLastSent := time.Now().Second()
in := Message{}
for ((timeLastSent + ConnTimeout) % 60) != time.Now().Second() {
if err := dec.Decode(&in); err != nil {
continue
} else if in == Ping {
timeLastSent = time.Now().Second()
continue
}
timeLastSent = time.Now().Second()
Messages <- in
in = Message{}
}
fmt.Println("Client timed out!") //This gets called
return
}
Ive tried both with and without the return on the last row of Listen
As response to #SimoEndre, Ive left the main method out of the code example, but since you mentioned it, this is the function that takes g.Messege{} out of the Messeges channel.
NOTE: MessageHandler() runs on its own go routine.
func MessageHandler() {
for msg := range Messages {
for _, disc := range LivingDiscussions {
if disc.DiscussionID.UDID == msg.UDID {
go disc.Push(msg)
break
}
}
}
}
Looking at the Listen function you will remark that it has a Messages channel which receive the the Message{} struct, but in the main goroutine it does not get outputted. Remember that goroutines are two way communication channels, which means that if a channel does receive an input value it must have an output value the channel to not block.
So you need to create a channel with the same struct type as the Message{}
message := make(chan Message{})
Then in the Join function you have to pop out the value pushed to channel:
func Join(ws *websocket.Conn) {
...
<-message
}
Update after new inputs:
It's not enough to iterate over the values coming from a channel, you need to do this inside a go func().
Getting the values out of different concurrently executing goroutines can be accomplished with the select keyword, which closely resembles the switch control statement and is sometimes called the communications switch.
go func() {
for msg := range Messages {
for _, disc := range LivingDiscussions {
if disc.DiscussionID.UDID == msg.UDID {
select {
case disc.Push <- msg: // push the channel value to the stack
default :
// default action
}
}
}
}
}()
I don't know how your disc.Push method is implemented, but if the idea is to push the received channel values to a stack you have to modify your code in a way to send back the channel value to the array. In the code snippet above i've just wanted to emphasize that it's important to get the values back pushed into the channel.
Related
To give you context,
The variable elementInput is dynamic. I do not know the exact length of it.
It can be 10, 5, or etc.
The *Element channel type is struct
My example is working. But my problem is this implementation is still synchronized, because I am waiting for the channel return so that I can append it to my result
Can you pls help me how to concurrent call GetElements() function and preserve the order defined in elementInput (based on index)
elementInput := []string{FB_FRIENDS, BEAUTY_USERS, FITNESS_USERS, COMEDY_USERS}
wg.Add(len(elementInput))
for _, v := range elementInput {
//create channel
channel := make(chan *Element)
//concurrent call
go GetElements(ctx, page, channel)
//Preserve the order
var elementRes = *<-channel
if len(elementRes.List) > 0 {
el = append(el, elementRes)
}
}
wg.Wait()
Your implementation is not concurrent.
Reason after every subroutine call you are waiting for result, that is making this serial
Below is Sample implementation similar to your flow
calling Concurreny method which calls function concurrently
afterwards we loop and collect response from every above call
main subroutine sleep for 2 seconds
Go PlayGround with running code -> Sample Application
func main() {
Concurrency()
time.Sleep(2000)
}
func response(greeter string, channel chan *string) {
reply := fmt.Sprintf("hello %s", greeter)
channel <- &reply
}
func Concurrency() {
events := []string{"ALICE", "BOB"}
channels := make([]chan *string, 0)
// start concurrently
for _, event := range events {
channel := make(chan *string)
go response(event, channel)
channels = append(channels, channel)
}
// collect response
response := make([]string, len(channels))
for i := 0; i < len(channels); i++ {
response[i] = *<-channels[i]
}
// print response
log.Printf("channel response %v", response)
}
I am attempting to loop through an Array and copy each value within an array. I would like to but spin each loop off in a separate goroutine. When I do run it with goroutines then will loop one less than the size of the array (len(Array) -1) but if I get rid of the goroutine then it processes just fine.
Am I missing something about how this should work? It seems very odd that it is always one less when running goroutines. Below is my code.
func createEventsForEachWorkoutReference(plan *sharedstructs.Plan, user *sharedstructs.User, startTime time.Time, timeZoneKey *string, transactionID *string, monitoringChannel chan interface{}) {
//Set the activity type as these workouts are coming from plans
activityType := "workout"
for _, workoutReference := range plan.WorkoutReferences {
go func(workoutReference sharedstructs.WorkoutReference) {
workout, getWorkoutError := workout.GetWorkoutByName(workoutReference.WorkoutID.ID, *transactionID)
if getWorkoutError == nil && workout != nil {
//For each workout, create a reference to be inserted into the event
reference := sharedstructs.Reference{ID: workout.WorkoutID, Type: activityType, Index: 0}
referenceArray := make([]sharedstructs.Reference, 0)
referenceArray = append(referenceArray, reference)
event := sharedstructs.Event{
EventID: uuidhelper.GenerateUUID(),
Description: workout.Description,
Type: activityType,
UserID: user.UserID,
IsPublic: false,
References: referenceArray,
EventDateTime: startTime,
PlanID: plan.PlanID}
//Insert the Event into the databse, I don't handle errors intentionally as it will be async
creationError := eventdomain.CreateNewEvent(&event, transactionID)
if creationError != nil {
redFalconLogger.LogCritical("plan.createEventsForEachWorkoutReference() Error Creating a workout"+creationError.Error(), *transactionID)
}
//add to the outputchannel
monitoringChannel <- event
//Calculate the next start time for the next loop
startTime = calculateNextEventTime(&startTime, &workoutReference.RestTime, timeZoneKey, transactionID)
}
}(workoutReference)
}
return
}
After a bit deeper dive, I think that I figured the root cause but not the (elegant) solution yet.
What appears to be happening is that my calling function is running in an async goroutine as well and using a "chan interface{}" to monitor and stream progress back to the client. On the last item in the array, it is completing the calling goroutine before the chan can be processed upstream.
What is the proper way to wait for the channel processing to complete. Below is a portion of my unit test that I am using to provide context.
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
createEventsForEachWorkoutReference(plan, &returnedUser, startDate, &timeZone, &transactionID, monitoringChan)
}()
var userEventArrayList []sharedstructs.Event
go func() {
for result := range monitoringChan {
switch result.(type) {
case sharedstructs.Event:
counter++
event := result.(sharedstructs.Event)
userEventArrayList = append(userEventArrayList, event)
fmt.Println("Channel Picked Up New Event: " + event.EventID + " with counter " + strconv.Itoa(counter))
default:
fmt.Println("No Match")
}
}
}()
wg.Wait()
//I COULD SLEEP HERE BUT THAT SEEMS HACKY
close(monitoringChan)
Wanted to add one more example (without my custom code). You can comment out the sleep line to see it work with sleep in there.
https://play.golang.org/p/t6L_C4zScP-
Finally figured the answer...
The problem was that I needed to close my monitoringChan in the first goroutine and then monitor (Defer wg.close()) in the second goroutine. Worked great when I did that!
https://play.golang.org/p/fEaZXiWCLt-
I'm looking for a solution to multiplex some channel output in go.
I have a source of data which is a read from an io.Reader that I send to a single channel. On the other side I have a websocket request handler that reads from the channel. Now it happens that two clients create a websocket connection, both reading from the same channel but each of them only getting a part of the messages.
Code example (simplified):
func (b *Bootloader) ReadLog() (<-chan []byte, error) {
if b.logCh != nil {
logrus.Warn("ReadLog called while channel already exists!")
return b.logCh, nil // This is where we get problems
}
b.logCh = make(chan []byte, 0)
go func() {
buf := make([]byte, 1024)
for {
n, err := b.p.Read(buf)
if err == nil {
msg := make([]byte, n)
copy(msg, buf[:n])
b.logCh <- msg
} else {
break
}
}
close(b.logCh)
b.logCh = nil
}()
return b.logCh, nil
}
Now when ReadLog() is called twice, the second call just returns the channel created in the first call, which leads to the problem explained above.
The question is: How to do proper multiplexing?
Is it better/easier/more ideomatic to care about the multiplexing on the sending or receiving site?
Should I hide the channel from the receiver and work with callbacks?
I'm a little stuck at the moment. Any hints are welcome.
Mutiplexing is pretty straightforward: make a slice of channels you want to multiplex to, start up a goroutine that reads from the original channel and copies each message to each channel in the slice:
// Really this should be in Bootloader but this is just an example
var consumers []chan []byte
func (b *Bootloader) multiplex() {
// We'll use a sync.once to make sure we don't start a bunch of these.
sync.Once(func(){
go func() {
// Every time a message comes over the channel...
for v := range b.logCh {
// Loop over the consumers...
for _,cons := range consumers {
// Send each one the message
cons <- v
}
}
}()
})
}
I'm using channels in Go to process a data pipeline of sorts. The code looks something like this:
type Channels struct {
inputs chan string
errc chan error
quit chan struct{}
}
func (c *Channels) doSomethingWithInput() {
defer close(c.quit)
defer close(c.errc)
for input := range p.inputs {
_, err := doSomethingThatSometimesErrors(input)
if err != nil {
c.errc <- err
return
}
}
doOneFinalThingThatCannotError()
return
}
func (c *Channels) inputData(s string) {
// This function implementation is my question
}
func StartProcessing(c *Channels, data ...string) error {
go c.doSomethingWithInput()
go func() {
defer close(c.inputs)
for _, i := range data {
select {
case <-c.quit:
break
default:
}
inputData(i)
}
}()
// Block until the quit channel is closed.
<-c.quit
if err := <-c.errc; err != nil {
return err
}
return nil
}
This seems like a reasonable way to communicate a quit signal between channel processors and is based on this blog post about concurrency patterns in Go.
The thing I struggle with using this pattern is the inputData function. Adding strings to the input channel needs to wait for doSomethingWithInput() to read the channel, but it also might error. inputData needs to try and feed the inputs channel but give up if told to quit. The best I could do was this:
func (c *Channels) inputData(s string) {
for {
select {
case <-c.quit:
return
case c.inputs <- s:
return
}
}
}
Essentially, "oscillate between your options until one of them sticks." To be clear, I don't think it's a bad design. It just feels... wasteful. Like I'm missing something clever. How can I tell a channel sender to quit in Go when a channel consumer errors?
Your inputData() is fine, that's the way to do it.
In your use case, your channel consumer, the receiver, aka doSomethingWithInput() is the one which should have control over the "quit" channel. As it is, if an error occurs, just return from doSomethingWithInput(), which will in turn close the quit channel and make the sender(s) quit (will trigger case <-quit:). That is in fact the clever bit.
Just watch out with your error channel that's not buffered and closed when doSomethingWithInput() exits. You cannot read it afterwards to collect errors. You need to close it in your main function and initialize it with some capacity (make(chan int, 10) for example), or create a consumer goroutine for it. You may also want to try reading it with a select statement: your error checking code, as it is, will block forever if there are no errors.
I am trying to write a function in Go which monitors a channel and logs what is sent through it.
func monitorChannel(inChannel, outChannel reflect.Value, fid int64, cond *sync.Cond) {
for {
cond.L.Lock()
var toLog reflect.Value
var ok bool
for toLog, ok = inChannel.TryRecv() ; !toLog.IsValid(); { // while no value received
if !ok {
cond.L.Unlock()
return
}
cond.Wait()
}
outChannel.Send(toLog)
logMessage("a", "b", inChannel.Interface(), toLog.Interface(), fid)
cond.L.Unlock()
}
}
This function is supposed to receive from inChannel, log the message sent and send it through outChannel. Since I want to be able to log bi-directional channels, I call this function twice for each channel I want to log, swapping inChannel and outChannel. The lock is to keep the two goroutines from passing messages between each other. "fid" is just the id of the log file.
But when I run the following test code, I get a deadlock :
errsIn := make(chan int64)
errsOut := make(chan int64)
cond := sync.NewCond(&sync.Mutex{})
go monitorChannel(reflect.ValueOf(errsIn), reflect.ValueOf(errsOut), fid, cond)
go monitorChannel(reflect.ValueOf(errsOut), reflect.ValueOf(errsIn), fid, cond)
errsIn <- 1
if <-errsOut != 1 {
t.Fatal("lost value through channel send")
}
errsOut <- 1
if <-errsIn != 1 {
t.Fatal("lost value through channel send")
}
It seems as if TryRecv is returning false on its second return value even though I haven't closed the channel. Why is this? What should I do about it?
I am running go 1.0.3 on Windows 8 64 bit.
EDIT
I later discovered that TryRecv has a somewhat confusing behaviour and managed to make a generalized version of the function using the reflect package and two sync.Locker's. I still think that jnml's solution is more elegant, but if anyone has experienced similar problems with TryRecv, take a look at the comment in the middle of the function.
func passOnAndLog(in, out reflect.Value, l1, l2 sync.Locker) {
for {
l1.Lock()
val, ok := in.TryRecv()
for !val.IsValid() { // while nothing received
l1.Unlock()
time.Sleep(time.Nanosecond) // pausing current thread
l1.Lock()
val, ok = in.TryRecv()
}
// if val.IsValid() == true and ok == false ,the channel is closed
// if val.IsValid() == false and ok == false ,the channel is open but we received nothing
// if val.IsValid() == true and ok == true ,we received an actual value from the open channel
// if val.IsValid() == false and ok == true ,we have no idea what happened
if !ok {
return
}
l1.Unlock()
l2.Lock() // don't want the other thread to receive while I am sending
out.Send(val)
LogValue(val) // logging
l2.Unlock()
}
}
The reflection based solution is too convoluted for me to figure out, being lazy, if it is correct and or feasible at all. (I suspect it is not, but only by intuition.)
I would approach the task in a simpler, although non-generic way. Let's have a channel which will be used by some producer(s) to write to it and will be used by some consumer(s) to read from it.
c := make(chan T, N)
It's possible to monitor this channel using a small helper function, like for example:
func monitored(c chan T) chan T {
m := make(chan T, M)
go func() {
for v := range c {
m <- v
logMessage(v)
}
close(m)
}()
return m
}
Now it is enough to:
mc := monitored(c)
and
Pass c to producers(s), but mc to consumers(s).
Close c when done to not leak goroutines.
Warning: Above code was not tested at all.