I have a method that may be used in multiple goroutines and run concurrently.
Inside this method, I have a conditional statement. If the conditional statement is true, I want all other goroutines calling this method to wait for one and only one of the goroutines to execute this conditional statement before proceeding to the next section.
For example:
type SomeClass struct {
mu sync.Mutex
}
func (c *SomeClass) SomeFunc() {
//Do some calculation
if condition {
//This part should be executed by only one goroutine if the condition is true.
//All others must wait for this to finish
}
//Additional calculations
}
And I want to use it like this:
func main(){
//initilize
go someClass.SomeFunc()
//If the condition is true, the following will wait at the conditional statement until the first one finishes the code inside the conditional block
//Once it's done, they can run concurrently
go someClass.SomeFunc()
go someClass.SomeFunc()
}
Edit
This is perhaps not the right design for this so I'm looking for any suggestions on how to implement this.
Edit2:
Note that each routine will have its own condition. This value of condition is not shared between threads. However, the work inside the condition should run only once only if the condition in 2 or more routines happens to be true at the same time.
You'll want a mutex protecting the condition from concurrent read/writes, and then a method for resetting the condition when you wish to execute the synchronous code again.
type SomeClass struct {
conditionMu sync.Mutex
condition bool
}
func (c *SomeClass) SomeFunc() {
// Lock the mutex, so that concurrent calls to SomeFunc will wait here.
c.conditionMu.Lock()
if c.condition {
// Synchronous code goes here.
// Reset the condition to false so that any waiting goroutines won't run the code inside this block again.
c.condition = false
}
// Unlock the mutex, and any waiting goroutines.
c.conditionMu.Unlock()
}
// ResetCondition sets the stored condition to true in a thread-safe manner.
func (c *SomeClass) ResetCondition() {
c.conditionMu.Lock()
c.condition = true
c.conditionMu.Unlock()
}
The other answers to this question were incorrect because they do not satisfy the requirements of the question.
If the lock is added outside the conditional statement, then it will act as a barrier and will force all routines to synchronize at that spot. This is not the point of this question. Suppose resolving the condition value takes a long time, we do not want to check the value one routine at a time. We want to let every process check the condition at once so if the condition is false, we can move forward without stopping.
We want to ensure that the goroutines run in parallel if the condition is not true. Adding a lock inside the method and outside the conditional statement will not allow that to happen.
The following solutions are correct and passed all tests and performed well.
Solution 1:
Use 2 nested conditional statement such as this:
Note that in this case, if the condition is false, no lock will be called and no synchronization is needed. Everything can run in parallel.
type SomeClass struct {
conditionMu sync.Mutex
rwMu sync.RWMutex
additionalWorkRequired bool
}
func (c *SomeClass) SomeFunc() {
//Do some work ...
//Note: The condition is not shared, some routines can have false and some true at the same time, which is fine.
condition := true;
// All routines can check this condition and go inside the block if the condition is true
if condition {
c.rwMutex.Lock()
c.additionalWorkRequired = true
c.rwMutex.Unlock()
//Lock so other routines can wait here for the first one
c.conditionMu.Lock()
if c.additionalWorkRequired {
// Synchronous code goes here.
c.additionalWorkRequired = false
}
//Unlock so all other processors can move forward in parallel
c.conditionMu.unlock()
}
//Finish up the remaining work
}
Solution 2:
Use the do function from sync/singleflight which can handle this situation automatically.
From documentation:
Do executes and returns the results of the given function, making sure that only one execution is in-flight for a given key at a time. If a duplicate comes in, the duplicate caller waits for the original to complete and receives the same results. The return value shared indicates whether v was given to multiple callers.
Edit:
Since many seem to be confused by this question and answer, I'm adding a use case which might make things more clear:
1. Send a HTTP Request
2. If the server returns an error saying credentials are incorrect (This is condition):
2.1. Save current credentials in a local variable
2.2. Acquire the mutex lock
2.2.1. Compare the shared credentials with the ones in the local variable(This is the second condition)
If they are the same, then replace them with new ones
2.3. Unlock
2.4. Retry request
Related
I understand that launch is an extension function on CoroutineScope. But then I see it being used like this:
import kotlinx.coroutines.*
fun main() {
GlobalScope.launch { // launch a new coroutine in background and continue
delay(1000L) // non-blocking delay for 1 second (default time unit is ms)
println("World!") // print after delay
}
println("Hello,") // main thread continues while coroutine is delayed
Thread.sleep(2000L) // block main thread for 2 seconds to keep JVM alive
}
My understaning is that in kotlin one can define an infix function and then call it without any paranthesis. But from the documenation, I don't think launch is an infix function (in fact it has more than one parameter, so it can not be infix). It is also not a keyword in language. Then how is it called without any paranthesis?
The first two parameters are default parameters and the third one is High order function. When the last parameter is High order function then you can move Lamba out of parenthesis.
Suppose you have fun:
fun post(s:String="default", block:()->Unit){}
You call it in these ways:
post("1",{
})
You will get a suggestion Lamda should be moved out of parentheses
After moving out of parentheses:
post("1"){
}
Now you can remove the first parameter since it is default parameter
post {
}
https://kotlinlang.org/docs/reference/lambdas.html
I'm trying to reduce the amount of http requests my discord bot is making.
It's reading from an API.
With the fetched data it updates an internal database and outputs the changes.
Thing is: that database is different for every server the bot is in, and that's where I'm using the go routines. But, some servers need to fetch the same data, here is where I want to reduce the http requests. Right now I'm making requests regardless if I've already fetched a character. I want to create some sort of data that could be shared between the go routines and before making a request search within this data.
I was advised to use mutex. I'm trying. Original question: Working with unbuffered channels in golang
I made a skeleton of the real code I've tried: https://play.golang.org/p/mt229ns1R8m
In this example master := make([][]map[string]interface{}, 0) is simulating the discord servers.
Chars and Chars2 would be the tracked chars for each individual server.
The char "Test" is mutual to both of them, so it should be fetched from the API only once.
It's outputing this:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test]]Cache: [map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
I was expecting the output to be:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
But a new cache is being generated by every go routine. How can I fix this?
Thanks.
There are too many unknowns here for me to really write a proper design, but let's make a few notes:
Try not to use interface{} at all, if at all possible. In this case, it seems that it must be possible, though I'm not sure what the actual types will be.
Try to make your data as simple as possible, but no simpler. In this case, that probably means: have one data structure for "thing that talks to a Discord server" and a separate one for "thing that talks to the local database" (is this a caching database? if so, what are the criteria for invalidating a cache entry?). But if one "character" (whatever that is—apparently a string) can have different properties per Discord server, that means that your index into your local database is not just a character, but rather a pair of values: the string value itself plus a Discord-server-identifier.
This might give you a functional interface like this:
var cacheServer *CacheServer
func InitCacheServer() error {
cacheServer = ... // whatever it takes to initialize the cache server
}
(I've assumed lazy initialization of the cache server. If you can do up-front initialization, you can drop the next test below. Replace ValueType with the type of the result of a cached lookup of a name.)
func (DiscordServer ds) Get(name string) (ValueType, error) {
if cacheserver == nil {
if err := InitCacheServer(); err != nil {
return nil, err
}
}
// Do a cache lookup. Tell the cache server that if there
// is no entry, it should return a NoEntry error and we will
// fill the cache ourselves, so it should hold this slot as
// "will be filled, so wait for it".
slot, v, err := cacheServer.Lookup(name, ds.identity, CacheServer.IntentToFill)
if err == CacheServer.NoEntry {
// We have the slot held. Try to look up the right info
// directly in the Discord server, then cache it.
v, err = ds.UncachedGet(name)
// Tell cache server that this is the value, or that it should
// produce this error instead of NoCache.
cacheServer.FillSlot(slot, v, err)
}
}
You might only want to cache some error types, rather than all; that's another one of those design questions that needs an answer that I cannot provide here. There are other ways to do this that don't necessarily need a slot pointer return value, too; I've just chosen this one for this example.
Note that most of the "hard work" is now in the cache server, which definitely requires some fancy footwork. In particular you will want to lock the overall data structure for a little while, use that to find the correct slot, then hold the slot itself so that other users of the slot must wait, while releasing the overall lock so that other users of other entries need not wait. This introduces locking order constraints: be careful to avoid deadlock. One method that should work is:
type CacheServer struct {
lock sync.Mutex
data map[string]map[string]*Entry
// more fields
}
type Entry {
lock sync.Mutex
cachedValue ValueType
cachedError error
}
(You'll need some more types, like Intent—just two enumerated integers for now—below, and probably more fields in the above; this is just a skeleton.)
func (cs *CacheServer) Lookup(name, srv string, flags Intent) (*Entry, ValueType, error) {
cs.lock.Lock()
defer cs.lock.Unlock()
// first, look up the server - if it does not exist, create one
smap := cs.data[srv]
if smap == nil {
cs.data[server] = make(map[string]*Entry)
}
entry := smap[name]
if entry == nil {
// no cached entry - if this is a pure lookup, just error,
// but if not, make a locked entry
if flags == CacheServer.IntentToFill {
// make a new entry and return with it locked
entry = &Entry{}
smap[name] = entry
entry.lock.Lock() // and do not unlock
}
return entry, nil, NoEntry
}
entry.lock.Lock() // wait for someone to fill it, if needed
defer entry.lock.Unlock()
return nil, entry.cachedValue, entry.cachedError
}
You need a routine to fill and release the entry as well, but it's pretty simple. You could, if you choose, make this a method on the Entry type rather than on the CacheServer type, as at least in this particular prototype, there is no need to use the cache server data structures directly. If you start getting fancier with cache invalidation, though, it might be nice to have access to the CacheServer object.
Note: I've designed this so that you can do a cache lookup without an intent-to-fill, if that's useful. If not, there's no reason to bother with the Intent argument.
Is there a way to run some goroutines until one of them returns valid value(integer higher than zero)?
inside a goroutine i'm to guess a number that i need to put inside math formula
You have to write this yourself.
A goroutine runs until it returns. Each goroutine decides for itself when to return.
Suppose you have functions/procedures A, B, and C, each of which does a long computation and one of the three might find a useful answer first and the other two should stop if so. In that case, you'll want to spin off three goroutines which do the computing:
func doA(args) {
... do computing for A ...
... deliver a result ...
return // this line is redundant, and here only for illustration
}
func doB(args) {
... do computing for B ...
... deliver a result ...
}
and so on.
What goes in the args? Well, that's up to you, but it's a good idea to give all three functions some way to find out that one of the others has delivered a useful answer and they should stop. A pretty clever way to do this is to have a channel that someone—whoever decides that a result is "useful", for instance—closes to indicate that everyone else should stop working. All of the do functions can then look like this:
func doA(done chan struct{}, other_args) {
var result_ready bool
for !result_ready {
select {
case <-done: // someone else delivered a good result
return // so stop working now
default:
... work a bit more ...
}
}
... deliver result ...
}
That still leaves the "deliver result" part. Where does the result go? That's up to you too, but a good way to handle this is to have a channel into each routine can put a result when they have one.
There are some tricks to consider here. Suppose that the main driver wants to look at multiple results and pick one that is "good enough" by some measure we don't want to encode into each worker. Each worker should do some work and send an answer-so-far, and then keep working until main says "I like one of the answers I got" by main closing the done channel. Then we end up with a code structure like this:
func doA(done chan struct{}, resultChan chan resulttype, args) {
for {
select {
case <-done:
return
default:
... do a little work ...
... try to deliver result-so-far ...
}
}
}
The try to deliver result-so-far should read like this:
select {
case <-done:
return
case resultChan <- result:
}
This might well eliminate the need for the earlier select: we have the thing that computes A compute until it has a result ready, then handle either "you can quit now" or "your result is being sent", whichever occurs first. If "you can quit" occurs first, it quits. If "your result is being sent", it gets right back to work on the next problem.
At worst, doA runs a little longer than it needs to: however long it takes to get one result.
Note that the concept of a "done channel", along with a lot of other scaffolding that is useful in many real programs, is contained in the idea of a context. See Go Concurrency Patterns: Context and, relating more directly to your own problem here, Go Concurrency Patterns: Pipelines and cancellation.
I am working on refactoring some legacy code that suffers from deadlocks. There are two main root causes:
1) the same thread locking the same mutex multiple times, which should not difficult to resolve, and
2) the code occasionally calls into user defined functions which can enter the same code at the top level. I need to lock the mutex before calling user defined functions, but I might end up executing the same code again which will result in a deadlock situation. So, I need some mechanism to tell me that the mutex has already been locked and I should not lock it again. Any suggestions?
Here is a (very) brief summary of what the code does:
class TreeNode {
public:
// Assign a new value to this tree node
void set(const boost::any& value, boost::function<void, const TreeNode&> validator) {
boost::upgrade_lock<boost::shared_mutex> lock(mutexToTree_);
// call validator here
boost::upgrade_to_unique_lock<boost::shared_mutex> ulock(lock);
// set this TreeNode to value
}
// Retrieve the value of this tree node
boost::any get() {
boost::shared_lock<boost::shared_mutex> lock(mutexToTree_);
// get value for this tree node
}
private:
static boost::shared_mutex mutexToRoot_;
};
The problem is that the validator function can call into get(), which locks mutexToRoot_ on the same thread. I could modify mutexToRoot_ to be a recursive mutex but that would prevent other threads from reading the tree during get() operation, which is unwanted behavior.
Since C++11 you can use std::recursive_mutex, which allows the owning thread to call lock or try_lock without blocking/reporting failure, whereas the other threads will block on lock/receive false on try_lock until the owning thread calls unlock as many times as it called lock/try_lock before.
I'm processing some user session data inside of a goroutine and creating a map to keep track of user id -> session data inside of it. The goroutine loops through a slice and if a SessionEnd event is found, the map key is deleted inside the same iteration. This doesn't seem to always be the case, as I can still retrieve some of the data as well as the 'key exists' bool variable sometimes in the following iterations. It's as if some variables haven't yet been zeroed.
Each map has only one goroutine writing/reading from it. From my understanding there shouldn't be a race condition, but it definitely seems that there is with the map and delete().
The code works fine if the garbage collector is run on every iteration. Am I using a map for the wrong purpose?
Pseudocode (a function that is run inside a single goroutine, lines is passed as a variable):
active := make(ActiveSessions) // map[int]UserSession
for _, l := range lines { // lines is a slice of a parsed log
u = l.EventData.(parser.User)
s, exists = active[u.SessionID]
switch l.Event {
// Contains cases which can check if exists is true or false
// errors if contains an event that can't happen,
// for example UserDisconnect before UserConnect,
// or UserConnect while a session is already active
case "UserConnect":
if exists {
// error, can't occur
// The same session id can occur in the log after a prior session has completed,
// which is exactly when the problems occur
}
case "UserDisconnect":
sessionFinished = true
}
// ...
if sessionFinished {
// <add session to finished sessions>
delete(active, u.SessionID)
// Code works only if runtime.GC() is executed here, could just be a coincidence
}
}