currently im implementing a caching system using std lib http/net.
An endpoint parses a key and validates the request using the isOK(key) function. If it is not okay, one routine is send to makeSureNowOK(key,edpoint) to make sure, isOk(key) will return true at the next request.
My simplified solution looks as follows:
func (ep *Endpoint) Handler() func(...) {
for {
ep.mu.Lock()
// WAITINGROOM //
//lint:ignore SA2001 empty critical section
ep.mu.Unlock()
bytesBody, err := isOK(key)
if err != nil {
select {
case <-ep.pause:
go makeSureNowOK(key)
default:
}
} else {
...
return
}
}
}
func makeSureNowOK(key string, ep ...) {
ep.mu.Lock()
... do validation ..
ep.pause <- struct{}{}
ep.mu.Unlock()
}
So I'm using a mutex to block further executions and a channel using select to catch back routines that passed the isOK function.
Another Idea to not use mutex is to use a closed channel to allow routines to pass. But then I have to recreate it, to block routines. That feels somewhat hacky.
How would you approach this problem?
Edit: To make my question more clear: The code above is working like so. But I feel like creating a "Waitingroom" by calling .Unlock() immediately after .Lock() is not a clean way to achieve this. Do you have other suggestions?
An alternative way would be to use sync waitgroup, but then I'd have to call waitgroup.Wait (where right now im un/locking the mutex which will be before waitgroup.Add which is aswell bad.
Related
A common pattern I use is:
resource.open()
defer resource.close()
sometimes checking errors in between, which leads to:
err := resource.open()
if err != nil{
//do error stuff and return
}
defer resource.close()
Sometimes I will need multiple open/close resources in a row, leading to a variation of the previous 5 lines to be repeated one after another. This variation may be repeated verbatim several times in my code (where I need all the same resources).
It would be wonderful to wrap all this in a function. However doing so would close the resource as soon as the function call is over. Is there any way around this - either deferring to a "level up" the call stack or some other way?
One way to do this is using an "initializer" function with callback:
func WithResources(f func(Resource1, Resource2)) {
r1:=NewResource1()
defer r1.Close()
r2:=NewResource2()
defer r2.Close()
f(r1,r2)
}
func F() {
WithResources(func(r1 Resource1, r2 Resource2) {
// Use r1, r2
})
}
The signature of the function f depends on your exact use case.
Another way is to use a struct for a resource set:
type Resources struct {
R1 Resource1
R2 Resource2
...
}
func NewResources() *Resources {
r:=&Resources{}
r.R1=NewR1()
r.R2=NewR2()
return r
}
func (r *Resources) Close() {
r.R1.Close()
r.R2.Close()
}
func f() {
r:=NewResources()
defer r.Close()
...
}
It would be wonderful to wrap all this in a function.
Most probably a lot of people would hate reading such code. So "wonderful" might be very subjective.
However doing so would close the resource as soon as the function call is over.
Exactly.
Is there any way around this [...]?
No.
In my program I have several go-routines who are essentially running endless processes. Why? you may ask, long story short it is the purpose of my entire application so it's out of question to change that. I would like to give users the ability to stop a single go-routine. I understand that I can use channel to signal the go-routines to stop, however there may be cases where I have, say, 10 go-routines running and I only want to stop 1. The issue is that the number of go-routines I want to run is dynamic and based on user input. What is the best way for me to add the ability to stop a go-routine dynamically and allow for singles to be stopped without the rest?
You need design a map to manage contexts.
Assume you've already known usage of context. It might look like:
ctx, cancel := context.WithCancel(ctx.TODO())
go func(ctx){
for {
select {
case <-ctx.Done():
return
default:
// job
}
}
}(ctx)
cancel()
Ok, now you can convert your question to another, it might called 'how to manage contexts of many goroutine'
type GoroutineManager struct{
m sync.Map
}
func (g *GoroutineManager) Add(cancel context.CancelFunc, key string)) {
g.m.Store(key, cancel)
}
func (g *GoroutineManager) KillGoroutine(key string) {
cancel, exist := g.m.Load(key)
if exist {
cancel()
}
}
Ok, Now you can manage your goroutine like :
ctx, cancel := context.WithCancel(ctx.TODO())
manager.Add(cancel, "routine-job-1")
go func(ctx){
for {
select {
case <-ctx.Done():
return
default:
// job
}
}
}(ctx)
// kill it as your wish
manager.KillGoroutine("routine-job-1")
I have a go routine that keeps blocked until a channel receives new data. However, I need to stop the go routine whenever a condition is true. I wonder what is the best way
to do this.
I will illustrate the problem with an example code. The first solution I thought was using a select statement and check the condition constantly, like this:
func routine(c chan string, shouldStop func() bool) {
select {
case s := <-c:
doStuff(s)
default:
if shouldStop() {
return
}
}
}
However, this approach will force the routine to call shouldStop() every time and never block. I thought this could lead to performance problems, specially because there a lot others routines running.
Another option would be to use a sleep to at least block a little between shouldStop() calls. However, this would not be a perfect solution, since I'd like to call doStuff() in the exact time the channel receives with new data
Lastly, I thought about using a second channel just to achieve this, like:
func routine(c chan string, stop chan bool) {
select {
case s := <-c:
doStuff(s)
case b := <-stop:
return
}
}
While I thought that this might work, this would force me to have an extra channel along with the shouldStop flag. Maybe there is a better solution I'm not aware of.
Any suggestion is appreciated. Thanks.
I'm writing a package to control a Canon DSLR using their EDSDK DLL from Go.
This is a personal project for a photo booth to use at our wedding at my partners request, which I'll be happy to post on GitHub when complete :).
Looking at the examples of using the SDK elsewhere, it isn't threadsafe and uses thread-local resources, so I'll need to make sure I'm calling it from a single thread during usage. While not ideal, it looks like Go provides a "runtime.LockOSThread" function for doing just that, although this does get called by the core DLL interop code itself, so I'll have to wait and find out if that interferes or not.
I want the rest of the application to be able to call the SDK using a higher level interface without worrying about the threading, so I need a way to pass function call requests to the locked thread/Goroutine to execute there, then pass the results back to the calling function outside of that Goroutine.
So far, I've come up with this working example of using very broad function definitions using []interface{} arrays and passing back and forward via channels. This would take a lot of mangling of input/output data on every call to do type assertions back out of the interface{} array, even if we know what we should expect for each function ahead of time, but it looks like it'll work.
Before I invest a lot of time doing it this way for possibly the worst way to do it - does anyone have any better options?
package edsdk
import (
"fmt"
"runtime"
)
type CanonSDK struct {
FChan chan functionCall
}
type functionCall struct {
Function func([]interface{}) []interface{}
Arguments []interface{}
Return chan []interface{}
}
func NewCanonSDK() (*CanonSDK, error) {
c := &CanonSDK {
FChan: make(chan functionCall),
}
go c.BackgroundThread(c.FChan)
return c, nil
}
func (c *CanonSDK) BackgroundThread(fcalls <-chan functionCall) {
runtime.LockOSThread()
for f := range fcalls {
f.Return <- f.Function(f.Arguments)
}
runtime.UnlockOSThread()
}
func (c *CanonSDK) TestCall() {
ret := make(chan []interface{})
f := functionCall {
Function: c.DoTestCall,
Arguments: []interface{}{},
Return: ret,
}
c.FChan <- f
results := <- ret
close(ret)
fmt.Printf("%#v", results)
}
func (c *CanonSDK) DoTestCall([]interface{}) []interface{} {
return []interface{}{ "Test", nil }
}
For similar embedded projects I've played with, I tend to create a single goroutine worker that listens on a channel to perform all the work over that USB device. And any results sent back out on another channel.
Talk to the device with channels only in Go in a one-way exchange. LIsten for responses from the other channel.
Since USB is serial and polling, I had to setup a dedicated channel with another goroutine that justs picks items off the channel when they were pushed into it from the worker goroutine that just looped.
I have a struct called Hub with a Run() method which is executed in its own goroutine. This method sequentially handles incoming messages. Messages arrive concurrently from multiple producers (separate goroutines). Of course I use a channel to accomplish this task. But now I want to hide the Hub behind an interface to be able to choose from its implementations. So, using a channel as a simple Hub's field isn't appropriate.
package main
import "fmt"
import "time"
type Hub struct {
msgs chan string
}
func (h *Hub) Run() {
for {
msg, hasMore := <- h.msgs
if !hasMore {
return
}
fmt.Println("hub: msg received", msg)
}
}
func (h *Hub) SendMsg(msg string) {
h.msgs <- msg
}
func send(h *Hub, prefix string) {
for i := 0; i < 5; i++ {
fmt.Println("main: sending msg")
h.SendMsg(fmt.Sprintf("%s %d", prefix, i))
}
}
func main() {
h := &Hub{make(chan string)}
go h.Run()
for i := 0; i < 10; i++ {
go send(h, fmt.Sprintf("msg sender #%d", i))
}
time.Sleep(time.Second)
}
So I've introduced Hub.SendMsg(msg string) function that just calls h.msgs <- msg and which I can add to the HubInterface. And as a Go-newbie I wonder, is it safe from the concurrency perspective? And if so - is it a common approach in Go?
Playground here.
Channel send semantics do not change when you move the send into a method. Andrew's answer points out that the channel needs to be created with make to send successfully, but that was always true, whether or not the send is inside a method.
If you are concerned about making sure callers can't accidentally wind up with invalid Hub instances with a nil channel, one approach is to make the struct type private (hub) and have a NewHub() function that returns a fully initialized hub wrapped in your interface type. Since the struct is private, code in other packages can't try to initialize it with an incomplete struct literal (or any struct literal).
That said, it's often possible to create invalid or nonsense values in Go and that's accepted: net.IP("HELLO THERE BOB") is valid syntax, or net.IP{}. So if you think it's better to expose your Hub type go ahead.
Easy answer
Yes
Better answer
No
Channels are great for emitting data from unknown go-routines. They do so safely, however I would recommend being careful with a few parts. In the listed example the channel is created with the construction of the struct by the consumer (and not not by a consumer).
Say the consumer creates the Hub like the following: &Hub{}. Perfectly valid... Apart from the fact that all the invokes of SendMsg() will block for forever. Luckily you placed those in their own go-routines. So you're still fine right? Wrong. You are now leaking go-routines. Seems fine... until you run this for a period of time. Go encourages you to have valid zero values. In this case &Hub{} is not valid.
Ensuring SendMsg() won't block could be achieved via a select{} however you then have to decide what to do when you encounter the default case (e.g. throw data away). The channel could block for more reasons than bad setup too. Say later you do more than simply print the data after reading from the channel. What if the read gets very slow, or blocks on IO. You then will start pushing back on the producers.
Ultimately, channels allow you to not think much about concurrency... However if this is something of high-throughput, then you have quite a bit to consider. If it is production code, then you need to understand that your API here involves SendMsg() blocking.