A common pattern I use is:
resource.open()
defer resource.close()
sometimes checking errors in between, which leads to:
err := resource.open()
if err != nil{
//do error stuff and return
}
defer resource.close()
Sometimes I will need multiple open/close resources in a row, leading to a variation of the previous 5 lines to be repeated one after another. This variation may be repeated verbatim several times in my code (where I need all the same resources).
It would be wonderful to wrap all this in a function. However doing so would close the resource as soon as the function call is over. Is there any way around this - either deferring to a "level up" the call stack or some other way?
One way to do this is using an "initializer" function with callback:
func WithResources(f func(Resource1, Resource2)) {
r1:=NewResource1()
defer r1.Close()
r2:=NewResource2()
defer r2.Close()
f(r1,r2)
}
func F() {
WithResources(func(r1 Resource1, r2 Resource2) {
// Use r1, r2
})
}
The signature of the function f depends on your exact use case.
Another way is to use a struct for a resource set:
type Resources struct {
R1 Resource1
R2 Resource2
...
}
func NewResources() *Resources {
r:=&Resources{}
r.R1=NewR1()
r.R2=NewR2()
return r
}
func (r *Resources) Close() {
r.R1.Close()
r.R2.Close()
}
func f() {
r:=NewResources()
defer r.Close()
...
}
It would be wonderful to wrap all this in a function.
Most probably a lot of people would hate reading such code. So "wonderful" might be very subjective.
However doing so would close the resource as soon as the function call is over.
Exactly.
Is there any way around this [...]?
No.
Related
currently im implementing a caching system using std lib http/net.
An endpoint parses a key and validates the request using the isOK(key) function. If it is not okay, one routine is send to makeSureNowOK(key,edpoint) to make sure, isOk(key) will return true at the next request.
My simplified solution looks as follows:
func (ep *Endpoint) Handler() func(...) {
for {
ep.mu.Lock()
// WAITINGROOM //
//lint:ignore SA2001 empty critical section
ep.mu.Unlock()
bytesBody, err := isOK(key)
if err != nil {
select {
case <-ep.pause:
go makeSureNowOK(key)
default:
}
} else {
...
return
}
}
}
func makeSureNowOK(key string, ep ...) {
ep.mu.Lock()
... do validation ..
ep.pause <- struct{}{}
ep.mu.Unlock()
}
So I'm using a mutex to block further executions and a channel using select to catch back routines that passed the isOK function.
Another Idea to not use mutex is to use a closed channel to allow routines to pass. But then I have to recreate it, to block routines. That feels somewhat hacky.
How would you approach this problem?
Edit: To make my question more clear: The code above is working like so. But I feel like creating a "Waitingroom" by calling .Unlock() immediately after .Lock() is not a clean way to achieve this. Do you have other suggestions?
An alternative way would be to use sync waitgroup, but then I'd have to call waitgroup.Wait (where right now im un/locking the mutex which will be before waitgroup.Add which is aswell bad.
My experience working with Go is recent and in reviewing some code, I have seen that while it is write-protected, there is a problem with reading the data. Not with the reading itself, but with possible modifications that can occur between the reading and the modification of the slice.
type ConcurrentSlice struct {
sync.RWMutex
items []Item
}
type Item struct {
Index int
Value Info
}
type Info struct {
Name string
Labels map[string]string
Failure bool
}
As mentioned, the writing is protected in this way:
func (cs *ConcurrentSlice) UpdateOrAppend(item ScalingInfo) {
found := false
i := 0
for inList := range cs.Iter() {
if item.Name == inList.Value.Name{
cs.items[i] = item
found = true
}
i++
}
if !found {
cs.Lock()
defer cs.Unlock()
cs.items = append(cs.items, item)
}
}
func (cs *ConcurrentSlice) Iter() <-chan ConcurrentSliceItem {
c := make(chan ConcurrentSliceItem)
f := func() {
cs.Lock()
defer cs.Unlock()
for index, value := range cs.items {
c <- ConcurrentSliceItem{index, value}
}
close(c)
}
go f()
return c
}
But between collecting the content of the slice and modifying it, modifications can occur.It may be that another routine modifies the same slice and when it is time to assign a value, it no longer exists: slice[i] = item
What would be the right way to deal with this?
I have implemented this method:
func GetList() *ConcurrentSlice {
if list == nil {
denylist = NewConcurrentSlice()
return denylist
}
return denylist
}
And I use it like this:
concurrentSlice := GetList()
concurrentSlice.UpdateOrAppend(item)
But I understand that between the get and the modification, even if it is practically immediate, another routine may have modified the slice. What would be the correct way to perform the two operations atomically? That the slice I read is 100% the one I modify. Because if I try to assign an item to a index that no longer exists, it will break the execution.
Thank you in advance!
The way you are doing the blocking is incorrect, because it does not ensure that the items you return have not been removed. In case of an update, the array would still be at least the same length.
A simpler solution that works could be the following:
func (cs *ConcurrentSlice) UpdateOrAppend(item ScalingInfo) {
found := false
i := 0
cs.Lock()
defer cs.Unlock()
for _, it := range cs.items {
if item.Name == it.Name{
cs.items[i] = it
found = true
}
i++
}
if !found {
cs.items = append(cs.items, item)
}
}
Use a sync.Map if the order of the values is not important.
type Items struct {
m sync.Map
}
func (items *Items) Update(item Info) {
items.m.Store(item.Name, item)
}
func (items *Items) Range(f func(Info) bool) {
items.m.Range(func(key, value any) bool {
return f(value.(Info))
})
}
Data structures 101: always pick the best data structure for your use case. If you’re going to be looking up objects by name, that’s EXACTLY what map is for. If you still need to maintain the order of the items, you use a treemap
Concurrency 101: like transactions, your mutex should be atomic, consistent, and isolated. You’re failing isolation here because the data structure read does not fall inside your mutex lock.
Your code should look something like this:
func {
mutex.lock
defer mutex.unlock
check map or treemap for name
if exists update
else add
}
After some tests, I can say that the situation you fear can indeed happen with sync.RWMutex. I think it could happen with sync.Mutex too, but I can't reproduce that. Maybe I'm missing some informations, or maybe the calls are in order because they all are blocked and the order they redeem the right to lock is ordered in some way.
One way to keep your two calls safe without other routines getting in 'conflict' would be to use an other mutex, for every task on that object. You would lock that mutex before your read and write, and release it when you're done. You would also have to use that mutex on any other call that write (or read) to that object. You can find an implementation of what I'm talking about here in the main.go file. In order to reproduce the issue with RWMutex, you can simply comment the startTask and the endTask calls and the issue is visible in the terminal output.
EDIT : my first answer was wrong as I misinterpreted a test result, and fell in the situation described by OP.
tl;dr;
If ConcurrentSlice is to be used from a single goroutine, the locks are unnecessary, because the way algorithm written there is not going to be any concurrent read/writes to slice elements, or the slice.
If ConcurrentSlice is to be used from multiple goroutines, existings locks are not sufficient. This is because UpdateOrAppend may modify slice elements concurrently.
A safe version woule need two versions of Iter:
This can be called by users of ConcurrentSlice, but it cannot be called from `UpdateOrAppend:
func (cs *ConcurrentSlice) Iter() <-chan ConcurrentSliceItem {
c := make(chan ConcurrentSliceItem)
f := func() {
cs.RLock()
defer cs.RUnlock()
for index, value := range cs.items {
c <- ConcurrentSliceItem{index, value}
}
close(c)
}
go f()
return c
}
and this is only to be called from UpdateOrAppend:
func (cs *ConcurrentSlice) internalIter() <-chan ConcurrentSliceItem {
c := make(chan ConcurrentSliceItem)
f := func() {
// No locking
for index, value := range cs.items {
c <- ConcurrentSliceItem{index, value}
}
close(c)
}
go f()
return c
}
And UpdateOrAppend should be synchronized at the top level:
func (cs *ConcurrentSlice) UpdateOrAppend(item ScalingInfo) {
cs.Lock()
defer cs.Unlock()
....
}
Here's the long version:
This is an interesting piece of code. Based on my understanding of the go memory model, the mutex lock in Iter() is only necessary if there is another goroutine working on this code, and even with that, there is a possible race in the code. However, UpdateOrAppend only modifies elements of the slice with lower indexes than what Iter is working on, so that race never manifests itself.
The race can happen as follows:
The for-loop in iter reads element 0 of the slice
The element is sent through the channel. Thus, the slice receive happens after the first step.
The receiving end potentially updates element 0 of the slice. There is no problem up to here.
Then the sending goroutine reads element 1 of the slice. This is when a race can happen. If step 3 updated index 1 of the slice, the read at step 4 is a race. That is: if step 3 reads the update done by step 4, it is a race. You can see this if you start with i:=1 in UpdateOrAppend, and running it with the -race flag.
But UpdateOrAppend always modifies slice elements that are already seen by Iter when i=0, so this code is safe, even without the lock.
If there will be other goroutines accessing and modifying the structure, you need the Mutex, but you need it to protect the complete UpdateOrAppend method, because only one goroutine should be allowed to run that. You need the mutex to protect the potential updates in the first for-loop, and that mutex has to also include the slice append case, because that may actually modify the slice of the underlying object.
If Iter is only called from UpdateOrAppend, then this single mutex should be sufficient. If however Iter can be called from multiple goroutines, then there is another race possibility. If one UpdateOrAppend is running concurrently with multiple Iter instances, then some of those Iter instances will read from the modified slice elements concurrently, causing a race. So, it should be such that multiple Iters can only run if there are no UpdateOrAppend calls. That is a RWMutex.
But Iter can be called from UpdateOrAppend with a lock, so it cannot really call RLock, otherwise it is a deadlock.
Thus, you need two versions of Iter: one that can be called outside UpdateOrAppend, and that issues RLock in the goroutine, and another that can only be called from UpdateOrAppend and does not call RLock.
I'm coming across a few situations where I would like to use routing to change some Is_Active fields in my database but I'm curious about performance.
Let's have a route handler as so:
func testHandler(r *mux.Router) {
r.HandleFunc("/test/{status}" statusHandler).Methods("GET")
}
Now that parameter will only ever be 0 or 1, unless the user tries something else but either way nothing will happen unless it's 0 or 1. My question is, is it faster to parse the string into a boolean which would involve bringing in the strconv package or would it be faster to just do a switch on the string?
Example of both:
func statusHandler(w http.ResponseWriter, r *http.Request) {
v := mux.Vars(r)
active, err := strconv.ParseBool(v["status"])
// Handle err
if active {
// Do something
} else {
// Do something else
}
}
or
func statusHandler(w http.ResponseWriter, r *http.Request) {
v := mux.Vars(r)
switch v["status"] {
case "0":
// Do something
case "1":
// Do something else
default:
// Throw 500 Error
}
}
You could see the source code of ParseBool here. It uses switch too but with more cases. If the compiler inlines its code it, speed should be very similar to your code. If you want a definite answer you should benchmark different cases.
In general I discourage you from paying attention to this small details. It's just matter of some nano seconds but it make your codes more difficult to understand. Begin optimizations with profiling your code to find hotspots that take a lot of time and fix them.
I have started learning Go today.
One thing that makes me crazy, it's the err returned parameter.
Let's assume I need to nest few functions. like this:
return string(json.Marshal(MyData))
or more complex example:
return func1(func2(func3(MyData)))
Is it really necessary to write:
tmp1 , _ = func3(MyData)
tmp2 , _ = func2(tmp1)
tmp3 , _ = func1(tmp2)
return tmp3
That's annoying!
Is there any way to make the code looks cleaner?
It is possible to define a function to ignore errors, but Go's lack of generics make it so you'd have to use interface{} and typecasts all over the place, losing a lot of static guarantees from the typechecker in the process. It is extremely ugly. Don't do this.
func ignoreError(val interface {}, err error) interface {} {
return val
}
At every call to ignoreError() you would have to make a type cast to the expected return type.
Playground example
One possible abstraction pattern you will often see is to use a generic error handler.
This doesn't prevent you from having to deal with error values, but it does abstract the handling of errors away from the rest of your code.
Note that abstractions like these are considered "non-idiomatic" Go, the "pure" way is to explicitly handle errors in-place. This panic-driven alternative can still be very useful though, especially for quickly prototyping a script where you just want to dump all the errors in a console or logfile.
For reusable packages, I would stick to the verbose explicit way though, because others will expect error-producing functions to actually return error values, rather than using a panic-recover mechanism.
package main
import (
utils
)
func main() {
defer func() {
utils.Handle(func(err error) {
// Handle errors in a generic way,
// for example using println, or writing to http
})
}()
var result, err := someFragileFunction()
Check(err)
}
package utils
func Check(err error) {
if err != nil {
panic(err)
}
}
func Handle(handler func(err error)) {
if r := recover(); r != nil {
if err, ok := r.(error); ok {
handler(err)
} else {
panic(r)
}
}
}
The real answer is: Don't.
Never just ignore the errors.
Seriously. The errors are there for a reason. If a function returns an error,
it almost always means that it's possible, during the operation of your program,
even if it's 100% bug-free, for the function to fail. And if it does,
you don't usually want to just keep going as if nothing happened.
If you're absolutely sure that you're using a function in a way that ensures that it will never return a non-nil error (unless there's a bug in your program, and there always is), you might want to write a Must-style function like in the template package which panics with the returned error value.
Error handling is not noise. It's not clutter. It's not something you want
to get rid of. If it looks like 50% of your program is error
handling, that's because 50% of your program is, and should be, error handling.
I wonder if there is any idiomatic way to represent scoped semantics. By scoped I mean things like:
scoped mutex (oneliner instead of explicit Lock + deffered Unlock),
logging function (or any code block) entrance and exit,
measuring execution time.
Example code for first two bullets:
package main
import "log"
import "sync"
func Scoped(m *sync.Mutex) func() {
m.Lock()
return func() {
m.Unlock()
}
}
func Log(what string) func() {
log.Println(what, "started")
return func() {
log.Println(what, "done")
}
}
func main() {
defer Log("testing")()
m := &sync.Mutex{} // obviously mutex should be from other source in real life
defer Scoped(m)()
// use m
}
https://play.golang.org/p/33j-GrBWSq
Basically we need to make one function call just now (eg mutex lock), and one call should be postponed to defer (eg mutex unlock). I propose just returning unnamed function here, but it can be easily named (return struct with function field).
There is only one problem: user can forget to 'call' result of first call.
This code is (can be) idiomatic?
Take anonymous function as a scope:
func() {
Entrance()
defer Exit()
// anything you want to do in this scope
}()
Your proposed solution is already nice. You return a value of func type which you also have to call at the end of the defer.
You can avoid that (returning a func value), but there have to be 2 function calls, one that logs the start event and another one that logs the end event.
The alternative is to make a function call which produces the parameter value of the function that is deferred (rather than returning a function) which is evaluated with the defer statement, and this way it still can remain one line.
You can also try it on the Go Playground:
func start(s string) string {
fmt.Println("Started", s)
return s
}
func end(name string) {
fmt.Println("Ended", name)
}
func main() {
defer end(start("main"))
fmt.Println("Doing main's work...")
}
Output:
Started main
Doing main's work...
Ended main
I do not believe there is an idiomatic way to do this. I'm not sure why you'd want to either, is it really so bad to write
m.Lock()
defer m.Unlock()
?
I think question isn't relevant to Go idiomaticity, Seems it's generally better to reason about code when function behave identically either call. To keep state I'd better make an object and define function as method on that object. Means something like
type message string
func (foo message) Log(bar string){
if bar==nil{doSomethingSpecial()}
switch foo{
case something: doSomething()
...
case nil: doSomethingInitial()
default: doDefault()
}
log.Println(bar, "started")
foo=bar
}