For example:
package package
// Dear user, CleanUp must only be used with defer: defer CleanUp()
func CleanUp() {
// some logic to check if call was deferred
// do tear down
}
And in userland code:
func main() {
package.CleanUp() // PANIC, CleanUp must be deferred!
}
But all should be fine if user runs:
func main() {
defer package.CleanUp() // good job, no panic
}
Things I already tried:
func DeferCleanUp() {
defer func() { /* do tear down */ }()
// But then I realized this was exactly the opposite of what I needed
// user doesn't need to call defer CleanUp anymore but...
}
// now if the APi is misused it can cause problems too:
defer DeferCleanUp() // a defer inception xD, question remains.
Alright, per OPs request and just for laughs, I'm posting this hacky approach to solving this by looking at the call stack and applying some heuristics.
DISCLAIMER: Do not use this in real code. I don't think checking deferred is even a good thing.
Also Note: this approach will only work if the executable and the source are on the same machine.
Link to gist: https://gist.github.com/dvirsky/dfdfd4066c70e8391dc5 (this doesn't work in the playground because you can't read the source file there)
package main
import(
"fmt"
"runtime"
"io/ioutil"
"bytes"
"strings"
)
func isDeferred() bool {
// Let's get the caller's name first
var caller string
if fn, _, _, ok := runtime.Caller(1); ok {
caller = function(fn)
} else {
panic("No caller")
}
// Let's peek 2 levels above this - the first level is this function,
// The second is CleanUp()
// The one we want is who called CleanUp()
if _, file, line, ok := runtime.Caller(2); ok {
// now we actually need to read the source file
// This should be cached of course to avoid terrible performance
// I copied this from runtime/debug, so it's a legitimate thing to do :)
data, err := ioutil.ReadFile(file)
if err != nil {
panic("Could not read file")
}
// now let's read the exact line of the caller
lines := bytes.Split(data, []byte{'\n'})
lineText := strings.TrimSpace(string(lines[line-1]))
fmt.Printf("Line text: '%s'\n", lineText)
// Now let's apply some ugly rules of thumb. This is the fragile part
// It can be improved with regex or actual AST parsing, but dude...
return lineText == "}" || // on simple defer this is what we get
!strings.Contains(lineText, caller) || // this handles the case of defer func() { CleanUp() }()
strings.Contains(lineText, "defer ")
} // not ok - means we were not clled from at least 3 levels deep
return false
}
func CleanUp() {
if !isDeferred() {
panic("Not Deferred!")
}
}
// This should not panic
func fine() {
defer CleanUp()
fmt.Println("Fine!")
}
// this should not panic as well
func alsoFine() {
defer func() { CleanUp() }()
fmt.Println("Also Fine!")
}
// this should panic
func notFine() {
CleanUp()
fmt.Println("Not Fine!")
}
// Taken from the std lib's runtime/debug:
// function returns, if possible, the name of the function containing the PC.
func function(pc uintptr) string {
fn := runtime.FuncForPC(pc)
if fn == nil {
return ""
}
name := fn.Name()
if lastslash := strings.LastIndex(name, "/"); lastslash >= 0 {
name = name[lastslash+1:]
}
if period := strings.Index(name, "."); period >= 0 {
name = name[period+1:]
}
name = strings.Replace(name, "·", ".", -1)
return name
}
func main(){
fine()
alsoFine()
notFine()
}
Related
I have the following code in Go using the semaphore library just as an example:
package main
import (
"fmt"
"context"
"time"
"golang.org/x/sync/semaphore"
)
// This protects the lockedVar variable
var lock *semaphore.Weighted
// Only one go routine should be able to access this at once
var lockedVar string
func acquireLock() {
err := lock.Acquire(context.TODO(), 1)
if err != nil {
panic(err)
}
}
func releaseLock() {
lock.Release(1)
}
func useLockedVar() {
acquireLock()
fmt.Printf("lockedVar used: %s\n", lockedVar)
releaseLock()
}
func causeDeadLock() {
acquireLock()
// calling this from a function that's already
// locked the lockedVar should cause a deadlock.
useLockedVar()
releaseLock()
}
func main() {
lock = semaphore.NewWeighted(1)
lockedVar = "this is the locked var"
// this is only on a separate goroutine so that the standard
// go "deadlock" message doesn't print out.
go causeDeadLock()
// Keep the primary goroutine active.
for true {
time.Sleep(time.Second)
}
}
Is there a way to get the acquireLock() function call to print a message after a timeout indicating that there is a potential deadlock but without unblocking the call? I would want the deadlock to persist, but a log message to be written in the event that a timeout is reached. So a TryAcquire isn't exactly what I want.
An example of what I want in psuedo code:
afterFiveSeconds := func() {
fmt.Printf("there is a potential deadlock\n")
}
lock.Acquire(context.TODO(), 1, afterFiveSeconds)
The lock.Acquire call in this example would call the afterFiveSeconds callback if the Acquire call blocked for more than 5 seconds, but it would not unblock the caller. It would continue to block.
I think I've found a solution to my problem.
func acquireLock() {
timeoutChan := make(chan bool)
go func() {
select {
case <-time.After(time.Second * time.Duration(5)):
fmt.Printf("potential deadlock while acquiring semaphore\n")
case <-timeoutChan:
break
}
}()
err := lock.Acquire(context.TODO(), 1)
close(timeoutChan)
if err != nil {
panic(err)
}
}
I need to use mutex to read a variable and if the variable is 0, return from the function. This would prevent the mutex from Unlocking though.
I know that I could simply put a mutex.Unlock just before the return but it does not seem nice / correct.
I can't even do a defer mutex.Unlock() at the beginning of the function because the the code after requires a lot of time to run.
Is there a correct way to do so?
This is the example:
func mutexfunc() {
mutex.Lock()
if variable == 0 {
return
}
mutex.Unlock()
// long execution time (mutex must be unlocked)
}
UPDATE:
this is the solution I prefer:
var mutex = &sync.Mutex{}
var mutexSensibleVar = 0
func main() {
if withLock(func() bool { return mutexSensibleVar == 1 }) {
fmt.Println("it's true")
} else {
fmt.Println("it's false")
}
fmt.Println("end")
}
func withLock(f func() bool) bool {
mutex.Lock()
defer mutex.Unlock()
return f()
}
If you can't use defer, which is something you can't do here, you have to do the obvious:
func mutexfunc() {
mutex.Lock()
if variable == 0 {
mutex.Unlock()
return
}
mutex.Unlock()
// long execution time (mutex must be unlocked)
}
If the mutex is there only to protect that variable (that is, there isn't other code you're not showing us), you can also use sync/atomic:
func f() {
if atomic.LoadInt64(&variable) ==0 {
return
}
...
}
You can separate the locked part into its own function.
func varIsZero() bool {
mutex.Lock()
defer mutex.Unlock()
return variable == 0
}
func mutexfunc() {
if varIsZero() { return }
...
}
An alternative would be to use an anonymous function inside mutexfunc rather than a completely independent function, but it's a matter of taste here.
Also consider the (clumsy but readable) variant with a "need to unlock" boolean:
func f(arg1 argtype1, arg2 argtype2) ret returntype {
var needToUnlock bool
defer func() {
if needToUnlock {
lock.Unlock()
}
}()
// arbitrary amount of code here that runs unlocked
lock.Lock()
needToUnlock = true
// arbitrary amount of code here that runs locked
lock.Unlock()
needToUnlock = false
// arbitrary amount of code here that runs unlocked
// repeat as desired
}
You can wrap such a thing up in a type:
type DeferableLock struct {
L Locker
isLocked bool
}
func (d *DeferableLock) Lock() {
d.L.Lock()
d.isLocked = true
}
func (d *DeferableLock) Unlock() {
d.L.Unlock()
d.isLocked = false
}
func (d *DeferableLock) EnsureUnlocked() {
if d.isLocked {
d.Unlock()
}
}
func NewDeferableLock(l Locker) *DeferableLock() {
return &DeferableLock{L: l}
}
You can now wrap any sync.Locker with a DeferableLock. At functions like f, use the deferable wrapper to wrap the lock, and call defer d.EnsureUnlock.
(Any resemblance to sync.Cond is entirely deliberate.)
I'm working on a golang web crawler that should parse the search results on some specific search engine. The main difficulty - parsing with concurrency, or rather, in processing pagination such as
← Previous 1 2 3 4 5 ... 34 Next →. All things work fine except recursive crawling of paginated results. Look at my code:
package main
import (
"bufio"
"errors"
"fmt"
"net"
"strings"
"github.com/antchfx/htmlquery"
"golang.org/x/net/html"
)
type Spider struct {
HandledUrls []string
}
func NewSpider(url string) *Spider {
// ...
}
func requestProvider(request string) string {
// Everything is good here
}
func connectProvider(url string) net.Conn {
// Also works
}
// getContents makes request to search engine and gets response body
func getContents(request string) *html.Node {
// ...
}
// CheckResult controls empty search results
func checkResult(node *html.Node) bool {
// ...
}
func (s *Spider) checkVisited(url string) bool {
// ...
}
// Here is the problems
func (s *Spider) Crawl(url string, channelDone chan bool, channelBody chan *html.Node) {
body := getContents(url)
defer func() {
channelDone <- true
}()
if checkResult(body) == false {
err := errors.New("Nothing found there")
ErrFatal(err)
}
channelBody <- body
s.HandledUrls = append(s.HandledUrls, url)
fmt.Println("Handled ", url)
newUrls := s.getPagination(body)
for _, u := range newUrls {
fmt.Println(u)
}
for i, newurl := range newUrls {
if s.checkVisited(newurl) == false {
fmt.Println(i)
go s.Crawl(newurl, channelDone, channelBody)
}
}
}
func (s *Spider) getPagination(node *html.Node) []string {
// ...
}
func main() {
request := requestProvider(*requestFlag)
channelBody := make(chan *html.Node, 120)
channelDone := make(chan bool)
var parsedHosts []*Host
s := NewSpider(request)
go s.Crawl(request, channelDone, channelBody)
for {
select {
case recievedNode := <-channelBody:
// ...
for _, h := range newHosts {
parsedHosts = append(parsedHosts, h)
fmt.Println("added", h.HostUrl)
}
case <-channelDone:
fmt.Println("Jobs finished")
}
break
}
}
It always returns the first page only, no pagination. Same GetPagination(...) works good. Please tell me, where is my error(s).
Hope Google Translate was correct.
The problem is probably that main exits before all goroutine finished.
First, there is a break after the select statement and it runs uncodintionally after first time a channel is read. That ensures the main func returns after the first time you send something over channelBody.
Secondly, using channelDone is not the right way here. The most idomatic approach would be using a sync.WaitGroup. Before starting each goroutine, use WG.Add(1) and replace the defer with defer WG.Done(); In main, use WG.Wait(). Please be aware that you should use a pointer to refer to the WaitGroup. You can read more here.
Consider the following code:
type foo struct {
bar string
}
func f() *foo {
ret := &foo{"before"}
defer func() { ret.bar = "after" }()
return ret
}
func main() {
fmt.Println(f()) // prints "&{after}"
}
The motivation is having to return a struct but some of the fields need to be set only before returning (e.g. a timestamp of when the function completed).
Is deferring the field assignation a race condition?
Is it idiomatic?
Is there a better way?
The main benefit of using defer statement versus invoking something before returning from a function or routine is defer will run the statement even when a panic occurs before returning.
Thus it is commonly used to clean up resources (such as closing a file or network connection) rather than to set state.
The function below will not print or return "hello"
func f() string {
panic("omg")
fmt.Println("hello")
return "hello"
}
This code will print but won't return "hello"
func f() string {
defer fmt.Println("ello")
panic("omg")
return "hello"
}
To answer your question: No, it won't result in a race. Apart from the above difference, it is equivalent to calling something before the return statement.
You mention timestamping when a function completes. In that case you can use defer like this:
package main
import (
"fmt"
"time"
)
func main() {
foo()
}
func foo() {
defer trace("foo")()
time.Sleep(1 * time.Second)
}
func trace(fn string) func() {
start := time.Now()
return func() {
layout := "15:04:05.000"
end := time.Now()
fmt.Printf("%s start at %s, end at %s, total %s", fn, start.Format(layout), end.Format(layout), end.Sub(start))
}
}
Output: foo start at 23:00:00.000, end at 23:00:01.000, total 1s
Following problem:
I have a function that only should allow one caller to execute.
If someone tries to call the function and it is already busy the second caller should immediatly return with an error.
I tried the following:
1. Use a mutex
Would be pretty easy. But the problem is, you cannot check if a mutex is locked. You can only block on it. Therefore it does not work
2. Wait on a channel
var canExec = make(chan bool, 1)
func init() {
canExec <- true
}
func onlyOne() error {
select {
case <-canExec:
default:
return errors.New("already busy")
}
defer func() {
fmt.Println("done")
canExec <- true
}()
// do stuff
}
What I don't like here:
looks really messi
if easy to mistakenly block on the channel / mistakenly write to the channel
3. Mixture of mutex and shared state
var open = true
var myMutex *sync.Mutex
func canExec() bool {
myMutex.Lock()
defer myMutex.Unlock()
if open {
open = false
return true
}
return false
}
func endExec() {
myMutex.Lock()
defer myMutex.Unlock()
open = true
}
func onlyOne() error {
if !canExec() {
return errors.New("busy")
}
defer endExec()
// do stuff
return nil
}
I don't like this either. Using a shard variable with mutex is not that nice.
Any other idea?
I'll throw my preference out there - use the atomic package.
var (
locker uint32
errLocked = errors.New("Locked out buddy")
)
func OneAtATime(d time.Duration) error {
if !atomic.CompareAndSwapUint32(&locker, 0, 1) { // <-----------------------------
return errLocked // All logic in these |
} // four lines |
defer atomic.StoreUint32(&locker, 0) // <-----------------------------
// logic here, but we will sleep
time.Sleep(d)
return nil
}
The idea is pretty simple. Set the initial value to 0 (0 value of uint32). The first thing you do in the function is check if the value of locker is currently 0 and if so it changes it to 1. It does all of this atomically. If it fails simply return an error (or however else you like to handle a locked state). If successful, you immediately defer replacing the value (now 1) with 0. You don't have to use defer obviously, but failing to set the value back to 0 before returning would leave you in a state where the function could no longer be run.
After you do those 4 lines of setup, you do whatever you would normally.
https://play.golang.org/p/riryVJM4Qf
You can make things a little nicer if desired by using named values for your states.
const (
stateUnlocked uint32 = iota
stateLocked
)
var (
locker = stateUnlocked
errLocked = errors.New("Locked out buddy")
)
func OneAtATime(d time.Duration) error {
if !atomic.CompareAndSwapUint32(&locker, stateUnlocked, stateLocked) {
return errLocked
}
defer atomic.StoreUint32(&locker, stateUnlocked)
// logic here, but we will sleep
time.Sleep(d)
return nil
}
You can use a semaphore for this (go get golang.org/x/sync/semaphore)
package main
import (
"errors"
"fmt"
"sync"
"time"
"golang.org/x/sync/semaphore"
)
var sem = semaphore.NewWeighted(1)
func main() {
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := onlyOne(); err != nil {
fmt.Println(err)
}
}()
time.Sleep(time.Second)
}
wg.Wait()
}
func onlyOne() error {
if !sem.TryAcquire(1) {
return errors.New("busy")
}
defer sem.Release(1)
fmt.Println("working")
time.Sleep(5 * time.Second)
return nil
}
You could use standard channel approach with select statement.
var (
ch = make(chan bool)
)
func main() {
i := 0
wg := sync.WaitGroup{}
for i < 100 {
i++
wg.Add(1)
go func() {
defer wg.Done()
err := onlyOne()
if err != nil {
fmt.Println("Error: ", err)
} else {
fmt.Println("Ok")
}
}()
go func() {
ch <- true
}()
}
wg.Wait()
}
func onlyOne() error {
select {
case <-ch:
// do stuff
return nil
default:
return errors.New("Busy")
}
}
Do you want a function to be executed exactly once or once at given time? In former case take a look at https://golang.org/pkg/sync/#Once.
If you want once at a time solution:
package main
import (
"fmt"
"sync"
"time"
)
// OnceAtATime protects function from being executed simultaneously.
// Example:
// func myFunc() { time.Sleep(10*time.Second) }
// func main() {
// once := OnceAtATime{}
// once.Do(myFunc)
// once.Do(myFunc) // not executed
// }
type OnceAtATime struct {
m sync.Mutex
executed bool
}
func (o *OnceAtATime) Do(f func()) {
o.m.Lock()
if o.executed {
o.m.Unlock()
return
}
o.executed = true
o.m.Unlock()
f()
o.m.Lock()
o.executed = false
o.m.Unlock()
}
// Proof of concept
func f(m int, done chan<- struct{}) {
for i := 0; i < 10; i++ {
fmt.Printf("%d: %d\n", m, i)
time.Sleep(250 * time.Millisecond)
}
close(done)
}
func main() {
done := make(chan struct{})
once := OnceAtATime{}
go once.Do(func() { f(1, done) })
go once.Do(func() { f(2, done) })
<-done
done = make(chan struct{})
go once.Do(func() { f(3, done) })
<-done
}
https://play.golang.org/p/nZcEcWAgKp
But the problem is, you cannot check if a mutex is locked. You can only block on it. Therefore it does not work
With possible Go 1.18 (Q1 2022), you will be able to test if a mutex is locked... without blocking on it.
See (as mentioned by Go 101) the issue 45435 from Tye McQueen :
sync: add Mutex.TryLock
This is followed by CL 319769, with the caveat:
Use of these functions is almost (but not) always a bad idea.
Very rarely they are necessary, and third-party implementations (using a mutex and an atomic word, say) cannot integrate as well with the race detector as implementations in package sync itself.
The objections (since retracted) were:
Locks are for protecting invariants.
If the lock is held by someone else, there is nothing you can say about the invariant.
TryLock encourages imprecise thinking about locks; it encourages making assumptions about the invariants that may or may not be true.
That ends up being its own source of races.
Thinking more about this, there is one important benefit to building TryLock into Mutex, compared to a wrapper:
failed TryLock calls wouldn't create spurious happens-before edges to confuse the race detector.
And:
A channel-based implementation is possible, but performs poorly in comparison.
There's a reason we have sync.Mutex rather than just using channel for locking.
I came up with the following generic solution for that:
Works for me, or do you see any problem with that?
import (
"sync"
)
const (
ONLYONECALLER_LOCK = "onlyonecaller"
ANOTHER_LOCK = "onlyonecaller"
)
var locks = map[string]bool{}
var mutex = &sync.Mutex{}
func Lock(lock string) bool {
mutex.Lock()
defer mutex.Unlock()
locked, ok := locks[lock]
if !ok {
locks[lock] = true
return true
}
if locked {
return false
}
locks[lock] = true
return true
}
func IsLocked(lock string) bool {
mutex.Lock()
defer mutex.Unlock()
locked, ok := locks[lock]
if !ok {
return false
}
return locked
}
func Unlock(lock string) {
mutex.Lock()
defer mutex.Unlock()
locked, ok := locks[lock]
if !ok {
return
}
if !locked {
return
}
locks[lock] = false
}
see: https://play.golang.org/p/vUUsHcT3L-
How about this package: https://github.com/viney-shih/go-lock . It use channel and semaphore (golang.org/x/sync/semaphore) to solve your problem.
go-lock implements TryLock, TryLockWithTimeout and TryLockWithContext functions in addition to Lock and Unlock. It provides flexibility to control the resources.
Examples:
package main
import (
"fmt"
"time"
"context"
lock "github.com/viney-shih/go-lock"
)
func main() {
casMut := lock.NewCASMutex()
casMut.Lock()
defer casMut.Unlock()
// TryLock without blocking
fmt.Println("Return", casMut.TryLock()) // Return false
// TryLockWithTimeout without blocking
fmt.Println("Return", casMut.TryLockWithTimeout(50*time.Millisecond)) // Return false
// TryLockWithContext without blocking
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
fmt.Println("Return", casMut.TryLockWithContext(ctx)) // Return false
// Output:
// Return false
// Return false
// Return false
}
Lets keep it simple:
package main
import (
"fmt"
"time"
"golang.org/x/sync/semaphore"
)
var sem *semaphore.NewWeighted(1)
func init() {
sem = emaphore.NewWeighted(1)
}
func doSomething() {
if !sem.TryAcquire(1) {
return errors.New("I'm busy")
}
defer sem.Release(1)
fmt.Println("I'm doing my work right now, then I'll take a nap")
time.Sleep(10)
}
func main() {
go func() {
doSomething()
}()
}