Is there a way to run some goroutines until one of them returns valid value(integer higher than zero)?
inside a goroutine i'm to guess a number that i need to put inside math formula
You have to write this yourself.
A goroutine runs until it returns. Each goroutine decides for itself when to return.
Suppose you have functions/procedures A, B, and C, each of which does a long computation and one of the three might find a useful answer first and the other two should stop if so. In that case, you'll want to spin off three goroutines which do the computing:
func doA(args) {
... do computing for A ...
... deliver a result ...
return // this line is redundant, and here only for illustration
}
func doB(args) {
... do computing for B ...
... deliver a result ...
}
and so on.
What goes in the args? Well, that's up to you, but it's a good idea to give all three functions some way to find out that one of the others has delivered a useful answer and they should stop. A pretty clever way to do this is to have a channel that someone—whoever decides that a result is "useful", for instance—closes to indicate that everyone else should stop working. All of the do functions can then look like this:
func doA(done chan struct{}, other_args) {
var result_ready bool
for !result_ready {
select {
case <-done: // someone else delivered a good result
return // so stop working now
default:
... work a bit more ...
}
}
... deliver result ...
}
That still leaves the "deliver result" part. Where does the result go? That's up to you too, but a good way to handle this is to have a channel into each routine can put a result when they have one.
There are some tricks to consider here. Suppose that the main driver wants to look at multiple results and pick one that is "good enough" by some measure we don't want to encode into each worker. Each worker should do some work and send an answer-so-far, and then keep working until main says "I like one of the answers I got" by main closing the done channel. Then we end up with a code structure like this:
func doA(done chan struct{}, resultChan chan resulttype, args) {
for {
select {
case <-done:
return
default:
... do a little work ...
... try to deliver result-so-far ...
}
}
}
The try to deliver result-so-far should read like this:
select {
case <-done:
return
case resultChan <- result:
}
This might well eliminate the need for the earlier select: we have the thing that computes A compute until it has a result ready, then handle either "you can quit now" or "your result is being sent", whichever occurs first. If "you can quit" occurs first, it quits. If "your result is being sent", it gets right back to work on the next problem.
At worst, doA runs a little longer than it needs to: however long it takes to get one result.
Note that the concept of a "done channel", along with a lot of other scaffolding that is useful in many real programs, is contained in the idea of a context. See Go Concurrency Patterns: Context and, relating more directly to your own problem here, Go Concurrency Patterns: Pipelines and cancellation.
Related
I am working in Golang, which I am new to, and I have come across two interesting articles:
https://hackernoon.com/in-memory-caching-in-golang
The one from hackernoon is really good and the first example (Simple Map) is precisely what I am for to create a cache as it gives an example for expiring values in a cache. Where I am struggling to understand, is that it does not say whether the implementation creates just one instantation of the cache and not multiple copies, which would conflict or you have one value in one copy and one in another, and the look ups won't work properly.
In another link https://thedevelopercafe.com/articles/singleton-in-golang-839d8610958b it talks about instantation of one cache.
So, my question in both they use sync and so can I ask someone who has experience in Golang to advise me whether the example from Hackernoon in the function called newlocalcache sets up a singleton and if not what do I need to do to add it?
the function called newlocalcache sets up a singleton
No, it constructs and returns a new local cache every time it's called.
if not what do I need to do to add it?
Call it just once. For example:
var localCacheSingleton *localCache
var newLocalCacheOnce sync.Once
func newLocalCache(cleanupInterval time.Duration) *localCache {
newLocalCacheOnce.Do(func() {
lc := &localCache{
users: make(map[int64]cachedUser),
stop: make(chan struct{}),
}
lc.wg.Add(1)
go func(cleanupInterval time.Duration) {
defer lc.wg.Done()
lc.cleanupLoop(cleanupInterval)
}(cleanupInterval)
localCacheSingleton = lc
})
return localCacheSingleton
}
I am trying to download some json, parse it, check some information in the json and depending one the result continue processing or not.
What's the most RxSwift idiomatic way of doing this?
URLSession.shared.rx
.data(request:request)
.observe(on: ConcurrentDispatchQueueScheduler(qos: .background))
.flatMap(parseJson) // into ModelObject
.flatMap(checkModel) // on some condition is there any way to jump into the onCompleted block? if the condition is false then execute processObject
.map(processObject)
.subscribe(
onError: { error in
print("error: \(error)")
}, onCompleted: {
print("Completed with no error")
})
.disposed(by: disposeBag)
where parseJsonis something like:
func parseJson(_ data: Data) -> Single<ModelObject>
checkModel does some checking and if some conditions are fullfilled should complete the sequence without ending in processObject
func checkModel(_ modelObject: ModelObject) -> Single<ModelObject> {
//probably single is not what I want here
}
And finally processObject
func processObject(_ modelObject: ModelObject) -> Completable {
}
This is a bit of a tough question to answer because on the one hand you ask a bog simple question about skipping a map while on the other hand you ask for "most RxSwift idiomatic way of doing this," which would require more changes than simply jumping the map.
If I just answer the basic question. The solution would be to have checkModel return a Maybe rather than a Single.
Looking at this code from a "make it more idiomatic" perspective, a few more changes need to take place. A lot of what I'm about to say comes from assumptions based on the names of the functions and expectations as to what you are trying to accomplish. I will try to call out those assumptions as I go along...
The .observe(on: ConcurrentDispatchQueueScheduler(qos: .background)) is likely not necessary. URLSession already emits on the background.
The parseJson function probably should not return an Observable type at all. It should just return a ModelObject. This assumes that the function is pure; that it doesn't perform any side effect and merely transforms a Data into a ModelObject.
func parseJson(_ data: Data) throws -> ModelObject
The checkModel function should probably not return an Observable type. This really sounds like it should return a Bool and be used to filter the model objects that don't need further processing out. Here I'm assuming again that the function is pure, it doesn't perform any side-effect, it just checks the model.
func checkModel(_ modelObject: ModelObject) -> Bool
Lastly, the processObject function presumably has side effects. It's likely a consumer of data and therefore shouldn't return anything at all (i.e., it should return Void.)
func processObject(_ modelObject: ModelObject)
Udpdate: In your comments you say you want to end with a Completable. Even so, I would not want this function to return a completable because that would make it lazy and thus require you to subscribe even when you just want to call it for its effects.
You can create a generic wrap operator to make any side-effecting function into a Completable:
extension Completable {
static func wrap<T>(_ fn: #escaping (T) -> Void) -> (T) -> Completable {
{ element in
fn(element)
return Completable.empty()
}
}
}
If the above functions are adjusted as discussed above, then the Observable chain becomes:
let getAndProcess = URLSession.shared.rx.data(request:request)
.map(parseJson)
.filter(checkModel)
.flatMap(Completable.wrap(processObject))
.asCompletable()
The above will produce a Completable that will execute the flow every time it's subscribed to.
By setting things up this way, you will find that your base functions are far easier to test. You don't need any special infrastructure, not even RxText to make sure they are correct. Also, it is clear this way that parseJson and checkModel aren't performing any side effects.
The idea is to have a "Functional Core, Imperative Shell". The imperative bits (in this case the data request and the processing) are moved out to the edges while the core of the subscription is kept purely functional and easy to test/understand.
I have a method that may be used in multiple goroutines and run concurrently.
Inside this method, I have a conditional statement. If the conditional statement is true, I want all other goroutines calling this method to wait for one and only one of the goroutines to execute this conditional statement before proceeding to the next section.
For example:
type SomeClass struct {
mu sync.Mutex
}
func (c *SomeClass) SomeFunc() {
//Do some calculation
if condition {
//This part should be executed by only one goroutine if the condition is true.
//All others must wait for this to finish
}
//Additional calculations
}
And I want to use it like this:
func main(){
//initilize
go someClass.SomeFunc()
//If the condition is true, the following will wait at the conditional statement until the first one finishes the code inside the conditional block
//Once it's done, they can run concurrently
go someClass.SomeFunc()
go someClass.SomeFunc()
}
Edit
This is perhaps not the right design for this so I'm looking for any suggestions on how to implement this.
Edit2:
Note that each routine will have its own condition. This value of condition is not shared between threads. However, the work inside the condition should run only once only if the condition in 2 or more routines happens to be true at the same time.
You'll want a mutex protecting the condition from concurrent read/writes, and then a method for resetting the condition when you wish to execute the synchronous code again.
type SomeClass struct {
conditionMu sync.Mutex
condition bool
}
func (c *SomeClass) SomeFunc() {
// Lock the mutex, so that concurrent calls to SomeFunc will wait here.
c.conditionMu.Lock()
if c.condition {
// Synchronous code goes here.
// Reset the condition to false so that any waiting goroutines won't run the code inside this block again.
c.condition = false
}
// Unlock the mutex, and any waiting goroutines.
c.conditionMu.Unlock()
}
// ResetCondition sets the stored condition to true in a thread-safe manner.
func (c *SomeClass) ResetCondition() {
c.conditionMu.Lock()
c.condition = true
c.conditionMu.Unlock()
}
The other answers to this question were incorrect because they do not satisfy the requirements of the question.
If the lock is added outside the conditional statement, then it will act as a barrier and will force all routines to synchronize at that spot. This is not the point of this question. Suppose resolving the condition value takes a long time, we do not want to check the value one routine at a time. We want to let every process check the condition at once so if the condition is false, we can move forward without stopping.
We want to ensure that the goroutines run in parallel if the condition is not true. Adding a lock inside the method and outside the conditional statement will not allow that to happen.
The following solutions are correct and passed all tests and performed well.
Solution 1:
Use 2 nested conditional statement such as this:
Note that in this case, if the condition is false, no lock will be called and no synchronization is needed. Everything can run in parallel.
type SomeClass struct {
conditionMu sync.Mutex
rwMu sync.RWMutex
additionalWorkRequired bool
}
func (c *SomeClass) SomeFunc() {
//Do some work ...
//Note: The condition is not shared, some routines can have false and some true at the same time, which is fine.
condition := true;
// All routines can check this condition and go inside the block if the condition is true
if condition {
c.rwMutex.Lock()
c.additionalWorkRequired = true
c.rwMutex.Unlock()
//Lock so other routines can wait here for the first one
c.conditionMu.Lock()
if c.additionalWorkRequired {
// Synchronous code goes here.
c.additionalWorkRequired = false
}
//Unlock so all other processors can move forward in parallel
c.conditionMu.unlock()
}
//Finish up the remaining work
}
Solution 2:
Use the do function from sync/singleflight which can handle this situation automatically.
From documentation:
Do executes and returns the results of the given function, making sure that only one execution is in-flight for a given key at a time. If a duplicate comes in, the duplicate caller waits for the original to complete and receives the same results. The return value shared indicates whether v was given to multiple callers.
Edit:
Since many seem to be confused by this question and answer, I'm adding a use case which might make things more clear:
1. Send a HTTP Request
2. If the server returns an error saying credentials are incorrect (This is condition):
2.1. Save current credentials in a local variable
2.2. Acquire the mutex lock
2.2.1. Compare the shared credentials with the ones in the local variable(This is the second condition)
If they are the same, then replace them with new ones
2.3. Unlock
2.4. Retry request
I'm trying to reduce the amount of http requests my discord bot is making.
It's reading from an API.
With the fetched data it updates an internal database and outputs the changes.
Thing is: that database is different for every server the bot is in, and that's where I'm using the go routines. But, some servers need to fetch the same data, here is where I want to reduce the http requests. Right now I'm making requests regardless if I've already fetched a character. I want to create some sort of data that could be shared between the go routines and before making a request search within this data.
I was advised to use mutex. I'm trying. Original question: Working with unbuffered channels in golang
I made a skeleton of the real code I've tried: https://play.golang.org/p/mt229ns1R8m
In this example master := make([][]map[string]interface{}, 0) is simulating the discord servers.
Chars and Chars2 would be the tracked chars for each individual server.
The char "Test" is mutual to both of them, so it should be fetched from the API only once.
It's outputing this:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test]]Cache: [map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
I was expecting the output to be:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
But a new cache is being generated by every go routine. How can I fix this?
Thanks.
There are too many unknowns here for me to really write a proper design, but let's make a few notes:
Try not to use interface{} at all, if at all possible. In this case, it seems that it must be possible, though I'm not sure what the actual types will be.
Try to make your data as simple as possible, but no simpler. In this case, that probably means: have one data structure for "thing that talks to a Discord server" and a separate one for "thing that talks to the local database" (is this a caching database? if so, what are the criteria for invalidating a cache entry?). But if one "character" (whatever that is—apparently a string) can have different properties per Discord server, that means that your index into your local database is not just a character, but rather a pair of values: the string value itself plus a Discord-server-identifier.
This might give you a functional interface like this:
var cacheServer *CacheServer
func InitCacheServer() error {
cacheServer = ... // whatever it takes to initialize the cache server
}
(I've assumed lazy initialization of the cache server. If you can do up-front initialization, you can drop the next test below. Replace ValueType with the type of the result of a cached lookup of a name.)
func (DiscordServer ds) Get(name string) (ValueType, error) {
if cacheserver == nil {
if err := InitCacheServer(); err != nil {
return nil, err
}
}
// Do a cache lookup. Tell the cache server that if there
// is no entry, it should return a NoEntry error and we will
// fill the cache ourselves, so it should hold this slot as
// "will be filled, so wait for it".
slot, v, err := cacheServer.Lookup(name, ds.identity, CacheServer.IntentToFill)
if err == CacheServer.NoEntry {
// We have the slot held. Try to look up the right info
// directly in the Discord server, then cache it.
v, err = ds.UncachedGet(name)
// Tell cache server that this is the value, or that it should
// produce this error instead of NoCache.
cacheServer.FillSlot(slot, v, err)
}
}
You might only want to cache some error types, rather than all; that's another one of those design questions that needs an answer that I cannot provide here. There are other ways to do this that don't necessarily need a slot pointer return value, too; I've just chosen this one for this example.
Note that most of the "hard work" is now in the cache server, which definitely requires some fancy footwork. In particular you will want to lock the overall data structure for a little while, use that to find the correct slot, then hold the slot itself so that other users of the slot must wait, while releasing the overall lock so that other users of other entries need not wait. This introduces locking order constraints: be careful to avoid deadlock. One method that should work is:
type CacheServer struct {
lock sync.Mutex
data map[string]map[string]*Entry
// more fields
}
type Entry {
lock sync.Mutex
cachedValue ValueType
cachedError error
}
(You'll need some more types, like Intent—just two enumerated integers for now—below, and probably more fields in the above; this is just a skeleton.)
func (cs *CacheServer) Lookup(name, srv string, flags Intent) (*Entry, ValueType, error) {
cs.lock.Lock()
defer cs.lock.Unlock()
// first, look up the server - if it does not exist, create one
smap := cs.data[srv]
if smap == nil {
cs.data[server] = make(map[string]*Entry)
}
entry := smap[name]
if entry == nil {
// no cached entry - if this is a pure lookup, just error,
// but if not, make a locked entry
if flags == CacheServer.IntentToFill {
// make a new entry and return with it locked
entry = &Entry{}
smap[name] = entry
entry.lock.Lock() // and do not unlock
}
return entry, nil, NoEntry
}
entry.lock.Lock() // wait for someone to fill it, if needed
defer entry.lock.Unlock()
return nil, entry.cachedValue, entry.cachedError
}
You need a routine to fill and release the entry as well, but it's pretty simple. You could, if you choose, make this a method on the Entry type rather than on the CacheServer type, as at least in this particular prototype, there is no need to use the cache server data structures directly. If you start getting fancier with cache invalidation, though, it might be nice to have access to the CacheServer object.
Note: I've designed this so that you can do a cache lookup without an intent-to-fill, if that's useful. If not, there's no reason to bother with the Intent argument.
I found this example here:
func(*myHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
// What does this line do??
if h, ok := route[request.URL.String()]; ok {
h(writer, request)
return
}
io.WriteString(writer, "my server: " + request.URL.String())
}
I am extremely puzzled by this line:
if h, ok := route[request.URL.String()]; ok { h(writer, request) }
first off, how can it be valid syntactically to declare ok after it is assigned to the result of route()?
second, if h is returned by route(), how can it be used in the definition of ok???
I am thoroughly confused by this. Gophers, please help.
Take a look at this page in Tour of Go. It explains that:
Like for, the if statement can start with a short statement to execute before the condition.
Given your example:
h, ok := route[request.URL.String()]; ok { ... }
The short statement is h, ok := route[request.URL.String()]. The ; after marks the end of the short statement. The ok that follows is the condition. If true, then the code block {...} is executed. In that code block you can use all the variables that were assigned in the short statement.
first off, how can it be valid syntactically to declare ok after it is assigned to the result of route()?
ok is not declared after it is assigned. It is a boolean value, used in the condition.
second, if h is returned byroute(), how can it be used in the definition ofok`???
It's not definition of ok. It's the code block of the if statement.
Consider this alternative, almost equivalent writing style:
h, ok := route[request.URL.String()]
if ok {
h(writer, request)
return
}
This is perfectly clear, right? We just split the original code to two steps. But this is not the same. When writing this way, the h and ok variables are visible after the if statement. This is not desirable. It's a good practice to minimize the visibility of variables (also known as the live time). The longer a variable is visible, the longer the window of vulnerability during which it can be accidentally misused. This syntax of Go is truly great, genius, and I don't know a single other language where this kind of if statement is possible.
See also the section on the if statement in Effective Go.
Firstly: you should run through the Go Tour - specifically the section on Maps: https://tour.golang.org/moretypes/16
The "comma, ok" idiom here is designed to check that the key exists in the map. Doing a lookup on a non-existent key will panic (and likely crash your application).
if h, ok := route[request.URL.String()]; ok { h(writer, request) }
If request.URL.String() exists, set h and set ok to true.
If ok is true, then call the function h.
Otherwise, write the URL to the response (and do nothing further).