I'd like your advice on the correct way to test code that uses time.Ticker
For instance, let's say I have a countdown timer like below (just an example I thought up for the purposes of this question):
type TickFunc func(d time.Duration)
func Countdown(duration time.Duration, interval time.Duration, tickCallback TickFunc) {
ticker := time.NewTicker(interval)
for remaining := duration; remaining >= 0; remaining -= interval {
tickCallback(remaining)
<-ticker.C
}
ticker.Stop()
}
http://play.golang.org/p/WJisY52a5L
If I wanted to test this, I'd want to provide a mock so that I can have tests that run quickly and predictably, so I'd need to find a way to get my mock into the Countdown function.
I can think of a few ways to do this:
Create a Ticker interface and a first class function internal to the package that I can patch for the purposes of testing: http://play.golang.org/p/oSGY75vl0U
Create a Ticker interface and pass an implementation directly to the Countdown function:
http://play.golang.org/p/i67Ko5t4qk
If I do it the latter way, am I revealing too much information about how Countdown works and making it more difficult for potential clients to use this code? Instead of giving a duration and interval, they have to construct and pass in a Ticker.
I'm very interested in hearing what's the best approach when testing code like this? Or how you would change the code to preserve the behaviour, but make it more testable?
Thanks for your help!
Since this is a pretty simple function, I assume you are just using this as an example of how to mock non-trivial stuff. If you actually wanted to test this code, rather than mocking up ticker, why not just use really small intervals.
IMHO the 2nd option is the better of the two, making a user call:
foo(dur, NewTicker(interval)...
doesn't seem like much of a burden.
Also having the callback is serious code smell in Go:
func Countdown(ticker Ticker, duration time.Duration) chan time.Duration {
remainingCh := make(chan time.Duration, 1)
go func(ticker Ticker, dur time.Duration, remainingCh chan time.Duration) {
for remaining := duration; remaining >= 0; remaining -= ticker.Duration() {
remainingCh <- remaining
ticker.Tick()
}
ticker.Stop()
close(remainingCh)
}(ticker, duration, remainingCh)
return remainingCh
}
You could then use this code like:
func main() {
for d := range Countdown(NewTicker(time.Second), time.Minute) {
log.Printf("%v to go", d)
}
}
Here it is on the playground: http://play.golang.org/p/US0psGOvvt
This doesn't answer the how to inject the mock part, but it seems like you are trying too hard.
if the example is representative of what you actually are testing, then just use small numbers.
http://play.golang.org/p/b_1kqyIu-u
Countdown(5, 1, func(d time.Duration) {
log.Printf("%v to go", d)
})
Now, if you are testing code that calls Countdown (rather than testing Countdown), then I'd probably just create a flag you can set for your module that scales the numbers to be as fast possible with the same invocation count.
http://play.golang.org/p/KqCGnaR3vc
if testMode {
duration = duration/interval
interval = 1
}
Related
I have a go routine that keeps blocked until a channel receives new data. However, I need to stop the go routine whenever a condition is true. I wonder what is the best way
to do this.
I will illustrate the problem with an example code. The first solution I thought was using a select statement and check the condition constantly, like this:
func routine(c chan string, shouldStop func() bool) {
select {
case s := <-c:
doStuff(s)
default:
if shouldStop() {
return
}
}
}
However, this approach will force the routine to call shouldStop() every time and never block. I thought this could lead to performance problems, specially because there a lot others routines running.
Another option would be to use a sleep to at least block a little between shouldStop() calls. However, this would not be a perfect solution, since I'd like to call doStuff() in the exact time the channel receives with new data
Lastly, I thought about using a second channel just to achieve this, like:
func routine(c chan string, stop chan bool) {
select {
case s := <-c:
doStuff(s)
case b := <-stop:
return
}
}
While I thought that this might work, this would force me to have an extra channel along with the shouldStop flag. Maybe there is a better solution I'm not aware of.
Any suggestion is appreciated. Thanks.
I'm writing a package to control a Canon DSLR using their EDSDK DLL from Go.
This is a personal project for a photo booth to use at our wedding at my partners request, which I'll be happy to post on GitHub when complete :).
Looking at the examples of using the SDK elsewhere, it isn't threadsafe and uses thread-local resources, so I'll need to make sure I'm calling it from a single thread during usage. While not ideal, it looks like Go provides a "runtime.LockOSThread" function for doing just that, although this does get called by the core DLL interop code itself, so I'll have to wait and find out if that interferes or not.
I want the rest of the application to be able to call the SDK using a higher level interface without worrying about the threading, so I need a way to pass function call requests to the locked thread/Goroutine to execute there, then pass the results back to the calling function outside of that Goroutine.
So far, I've come up with this working example of using very broad function definitions using []interface{} arrays and passing back and forward via channels. This would take a lot of mangling of input/output data on every call to do type assertions back out of the interface{} array, even if we know what we should expect for each function ahead of time, but it looks like it'll work.
Before I invest a lot of time doing it this way for possibly the worst way to do it - does anyone have any better options?
package edsdk
import (
"fmt"
"runtime"
)
type CanonSDK struct {
FChan chan functionCall
}
type functionCall struct {
Function func([]interface{}) []interface{}
Arguments []interface{}
Return chan []interface{}
}
func NewCanonSDK() (*CanonSDK, error) {
c := &CanonSDK {
FChan: make(chan functionCall),
}
go c.BackgroundThread(c.FChan)
return c, nil
}
func (c *CanonSDK) BackgroundThread(fcalls <-chan functionCall) {
runtime.LockOSThread()
for f := range fcalls {
f.Return <- f.Function(f.Arguments)
}
runtime.UnlockOSThread()
}
func (c *CanonSDK) TestCall() {
ret := make(chan []interface{})
f := functionCall {
Function: c.DoTestCall,
Arguments: []interface{}{},
Return: ret,
}
c.FChan <- f
results := <- ret
close(ret)
fmt.Printf("%#v", results)
}
func (c *CanonSDK) DoTestCall([]interface{}) []interface{} {
return []interface{}{ "Test", nil }
}
For similar embedded projects I've played with, I tend to create a single goroutine worker that listens on a channel to perform all the work over that USB device. And any results sent back out on another channel.
Talk to the device with channels only in Go in a one-way exchange. LIsten for responses from the other channel.
Since USB is serial and polling, I had to setup a dedicated channel with another goroutine that justs picks items off the channel when they were pushed into it from the worker goroutine that just looped.
I need to build a data-structure like this:
map[string]SomeType
But it must store values for about 10 minutes and then clear it from memory.
Second condition is records amount - it must be huge. This data-structure must add at least 2-5K records per second.
So, what is the most correct way in Go to make it?
I'm trying to make goroutine with timeout for each new elemnt. And one(or more) garbage-collector goroutine with channel to receive timeouts and clear elements.
But I'm not sure it's the most clear way. Is it Ok to have millions of waiting goroutines with timeouts?
Thanks.
You will have to create a struct to hold your map and provide custom get/put/delete funcs to access it.
Note that 2-5k accesses per second is not really that much at all, so you don't have to worry about that.
Here's a simple implementation:
type item struct {
value string
lastAccess int64
}
type TTLMap struct {
m map[string]*item
l sync.Mutex
}
func New(ln int, maxTTL int) (m *TTLMap) {
m = &TTLMap{m: make(map[string]*item, ln)}
go func() {
for now := range time.Tick(time.Second) {
m.l.Lock()
for k, v := range m.m {
if now.Unix() - v.lastAccess > int64(maxTTL) {
delete(m.m, k)
}
}
m.l.Unlock()
}
}()
return
}
func (m *TTLMap) Len() int {
return len(m.m)
}
func (m *TTLMap) Put(k, v string) {
m.l.Lock()
it, ok := m.m[k]
if !ok {
it = &item{value: v}
m.m[k] = it
}
it.lastAccess = time.Now().Unix()
m.l.Unlock()
}
func (m *TTLMap) Get(k string) (v string) {
m.l.Lock()
if it, ok := m.m[k]; ok {
v = it.value
it.lastAccess = time.Now().Unix()
}
m.l.Unlock()
return
}
playground
note(2020-09-23): for some reason the time resolution on the current version of the playground is way off, this works fine, however to try on the playground you have to change the sleep to 3-5 seconds.
Take a look at buntdb.
tinykv is no longer being maintained.
Just for the record, I had the same problem and wrote tinykv package which uses a map internally.
It uses a heap of time.Time for timeouts, so it does not ranges over the whole map.
A max interval can be set when creating an instance. But actual intervals for checking the timeout can be any value of time.Duration greater than zero and less than max, based on the last item that timed out.
It provides CAS and Take functionality.
A callback (optional) can be set which notifies which key and value got timed out.
Timeouts can be explicit or sliding.
I suggest to use Map of golang's built-in package sync, it's very easy to use and already handles concurrency https://golang.org/pkg/sync/#Map
This code is from the most popular go matrix package https://github.com/skelterjohn/go.matrix/blob/go1/util.go
I googled this function and seems like it is for computing the fractal dimension. But in this package, this function is never used so I am having a hard time understanding this.
func countBoxes(start, cap int) chan box {
ints := make(chan box)
go func() {
for i := start; i < cap; i++ {
ints <- i
}
close(ints)
}()
return ints
}
Why do we need goroutine when we have only one anonymous function here?
And does anybody know what this function does in terms of matrix work?
Thanks in advance.
It returns a channel with cap - start queued integer events. (i.e. You can 'read'
start,start+1..,cap from the channel and then it closes ).
If you poke around in the code, it uses a similar kind of construct to create an iterator for the the indices of non-zero entries of sparse matrices. Look in sparse.go.
It's not used anywhere in the code that I can find, it may have been just to test
the idea.
I have a code like,
Routine 1 {
runtime.LockOSThread()
print something
send int to routine 2
runtime.UnlockOSThread
}
Routine 2 {
runtime.LockOSThread()
print something
send int to routine 1
runtime.UnlockOSThread
}
main {
go Routine1
go Routine2
}
I use run time lock-unlock because, I don't want that printing of
Routine 1 will mix with Routine 2. However, after execution of above
code, it outputs same as without lock-unlock (means printing outputs
mixed). Can anybody help me why this thing happening and how to force
this for happening.
NB: I give an example of print something, however there are lots of
printing and sending events.
If you want to serialize "print something", e.g. each "print something" should perform atomically, then just serialize it.
You can surround "print something" by a mutex. That'll work unless the code deadlock because of that - and surely it easily can in a non trivial program.
The easy way in Go to serialize something is to do it with a channel. Collect in a (go)routine everything which should be printed together. When collection of the print unit is done, send it through a channel to some printing "agent" as a "print job" unit. That agent will simply receive its "tasks" and atomically print each one. One gets that atomicity for free and as an important bonus the code can not deadlock easily no more in the simple case, where there are only non interdependent "print unit" generating goroutines.
I mean something like:
func printer(tasks chan string) {
for s := range tasks {
fmt.Printf(s)
}
}
func someAgentX(tasks chan string) {
var printUnit string
//...
tasks <- printUnit
//...
}
func main() {
//...
tasks := make(chan string, size)
go printer(tasks)
go someAgent1(tasks)
//...
go someAgentN(tasks)
//...
<- allDone
close(tasks)
}
What runtime.LockOSThread does is prevent any other goroutine from running on the same thread. It forces the runtime to create a new thread and run Routine2 there. They are still running concurrently but on different threads.
You need to use sync.Mutex or some channel magic instead.
You rarely need to use runtime.LockOSThread but it can be useful for forcing some higher priority goroutine to run on a thread of it's own.
package main
import (
"fmt"
"sync"
"time"
)
var m sync.Mutex
func printing(s string) {
m.Lock() // Other goroutines will stop here if until m is unlocked
fmt.Println(s)
m.Unlock() // Now another goroutine at "m.Lock()" can continue running
}
func main() {
for i := 0; i < 10; i++ {
go printing(fmt.Sprintf("Goroutine #%d", i))
}
<-time.After(3e9)
}
I think, this is because of runtime.LockOSThread(),runtime.UnlockOSThread does not work all time. It totaly depends on CPU, execution environment etc. It can't be forced by anyother way.