I'm writing a net/http server. My request handlers need access to some shared data that is updated by another go-routine. This go-routine keeps a single struct updated, and all of the net/http handlers need access to the same struct.
type MyData struct {
Expiration time.Time
A string
B string
C string
}
func update(data MyData) {
go func() {
for {
if data.Expiration >= time.Now() {
// set A, B, C
}
// sleep
}
}()
}
What's the best way to make a single, common MyData available to other go-routines? I was thinking a channel, but every reader of the channel should get the same MyData. The problem is that reading from a channel pops the item off the channel. I guess I'm looking for how to Peek() on the reader side, and Drain() on the writer side when it's time to update.
Mutex based solution
type MyData struct {
Expiration time.Time
A string
B string
C string
}
type MutexedMyData struct {
sync.RWMutex
MyData MyData
}
var m = &MutexedMyData{}
func (m *MutexedMyData) Update(my_data MyData) {
m.Lock()
m.MyData = my_data
m.Unlock()
}
func (m *MutexedMyData) Read() MyData {
u.RLock()
value := m.MyData
u.RUnlock()
return value
}
Related
Say I have an expensive function
func veryExpensiveFunction(int) int
and this function gets called a lot for the same number.
Is there a good way to allow this function to store previous results to use if the function gets called again that is perhaps even reusable for veryExpensiveFunction2?
Obviously, it would be possible to add an argument
func veryExpensiveFunctionCached(p int, cache map[int]int) int {
if val, ok := cache[p]; ok {
return val
}
result := veryExpensiveFunction(p)
cache[p] = result
return result
}
But now I have to create the cache somewhere, where I don't care about it. I would rather have it as a "static function member" if this were possible.
What is a good way to simulate a static member cache in go?
You can use closures; and let the closure manage the cache.
func InitExpensiveFuncWithCache() func(p int) int {
var cache = make(map[int]int)
return func(p int) int {
if ret, ok := cache[p]; ok {
fmt.Println("from cache")
return ret
}
// expensive computation
time.Sleep(1 * time.Second)
r := p * 2
cache[p] = r
return r
}
}
func main() {
ExpensiveFuncWithCache := InitExpensiveFuncWithCache()
fmt.Println(ExpensiveFuncWithCache(2))
fmt.Println(ExpensiveFuncWithCache(2))
}
output:
4
from cache
4
veryExpensiveFunctionCached := InitExpensiveFuncWithCache()
and use the wrapped function with your code.
You can try it here.
If you want it to be reusable, change the signature to InitExpensiveFuncWithCache(func(int) int) so it accept a function as a parameter. Wrap it in the closure, replacing the expensive computation part with it.
You need to be careful about synchronization if this cache will be used in http handlers. In Go standard lib, each http request is processed in a dedicated goroutine and at this moment we are at the domain of concurrency and race conditions. I would suggest a RWMutex to ensure data consistency.
As for the cache injection, you may inject it at a function where you create the http handler.
Here it is a prototype
type Cache struct {
store map[int]int
mux sync.RWMutex
}
func NewCache() *Cache {
return &Cache{make(map[int]int), sync.RWMutex{}}
}
func (c *Cache) Set(id, value int) {
c.mux.Lock()
c.store[id] = id
c.mux.Unlock()
}
func (c *Cache) Get(id int) (int, error) {
c.mux.RLock()
v, ok := c.store[id]
c.mux.RUnlock()
if !ok {
return -1, errors.New("a value with given key not found")
}
return v, nil
}
func handleComplexOperation(c *Cache) http.HandlerFunc {
return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request){
})
}
The Go standard library uses the following style for providing "static" functions (e.g. flag.CommandLine) but which leverage underlying state:
// "static" function is just a wrapper
func Lookup(p int) int { return expCache.Lookup(p) }
var expCache = NewCache()
func newCache() *CacheExpensive { return &CacheExpensive{cache: make(map[int]int)} }
type CacheExpensive struct {
l sync.RWMutex // lock for concurrent access
cache map[int]int
}
func (c *CacheExpensive) Lookup(p int) int { /*...*/ }
this design pattern not only allows for simple one-time use, but also allows for segregated usage:
var (
userX = NewCache()
userY = NewCache()
)
userX.Lookup(12)
userY.Lookup(42)
A Race Condition occures when i'am running my code. It is a simple implementation of a concurrent safe storage. The Race Condition disappears when i change the reciever in get() method to (p *storageType). I'm confused. I need someone who could explain to me such a behaivior.
package main
type storageType struct {
fc chan func()
value int
}
func newStorage() *storageType {
p := storageType{
fc: make(chan func()),
}
go p.run()
return &p
}
func (p storageType) run() {
for {
(<-p.fc)()
}
}
func (p *storageType) set(s int) {
p.fc <- func() {
p.value = s
}
}
func (p storageType) get() int {
res := make(chan int)
p.fc <- func() {
res <- p.value
}
return <-res
}
func main() {
storage := newStorage()
for i := 0; i < 1000; i++ {
go storage.set(i)
go storage.get()
}
}
In main() the storage variable is of type *storageType. If storageType.Get() has value receiver, then storage.get() means (*storage).get().
The get() call has storageType as the reciver, so the storage pointer variable has to be dereferenced to make a copy (that will be used as the receiver value). This copying means the value of the pointed storageType struct must be read. But this read is not synchronized with the run() method which reads and writes the struct (its value field).
If you change the receiver of get() to be a pointer (of type *storageType), then the receiver again will be a copy, but this time it will be a copy of the pointer, not the pointed struct. So no unsynchronized read of the struct happens.
See possible duplicate: Why does the method of a struct that does not read/write its contents still cause a race case?
First one: your main function doesn't wait for all goroutines to finish. All goroutines are forced to return when main does.
Look into using a sync.WaitGroup
I have a map (entities) where the key is a string and the value is a struct. Building that struct is an expensive operation because it has to go to the database to load the data.
type entityStateManagers struct {
entities map[string]*entityStateManager
mainLock *sync.Mutex
locks map[string]*sync.Mutex
}
The function below returns that struct for the provided key. If the struct is not in memory it will load it from the database.
I didn't want to have only one lock for everything because that would completely block access to the map while expensiveLoad(entityId) is running.
I tried having a separate struct (locks) with one lock per map key. With that, only access to entity_123 will have to wait for expensiveLoad(entity_123), for example.
I still have to do more testing, but it seems to be working.
func (handler entityStateManagers) getStateManager(entityId string) (*entityStateManager, error) {
handler.mainLock.Lock()
if handler.locks[entityId] == nil {
handler.locks[entityId] = &sync.Mutex{}
}
handler.mainLock.Unlock()
handler.locks[entityId].Lock()
defer handler.locks[entityId].Unlock()
if handler.entities[entityId] == nil {
handler.entities[entityId] = expensiveLoad(entityId)
}
return handler.entities[entityId], nil
}
Is this a reasonable approach? Is there something I am missing? Is there a better way for doing that?
i would say like this but i can be wrong, keeping two maps with same keys is useles, also pointers to mutexes are ehh.
type Wrapper struct {
*entityStateManager
sync.Mutex
}
type entityStateManagers struct {
entities map[string]*Wrapper
mainLock sync.Mutex
}
func (handler entityStateManagers) getStateManager(entityId string) (*entityStateManager, error) {
handler.mainLock.Lock()
val, ok := handler.entities[entityId]
if !ok {
val = &Wrapper{}
handler.entities[entityId] = val
}
val.Lock()
handler.mainLock.Unlock()
val.entityStateManager = expensiveLoad(entityId)
defer val.Unlock()
return val.entityStateManager, nil
}
Below I have an example of one structure which embeds another. I'm trying to figure out how to pass the more specific structure pointer to be stored in a less specific one. You can think of it as a collection. Wrapping in an interface doesn't seem to work, as doing so would make a copy, which isn't valid for structs with locks. Ideas?
package stackoverflow
import "sync"
type CoolerThingWithLock struct {
fancyStuff string
ThingWithLock
}
func NewCoolerThingWithLock() *CoolerThingWithLock {
coolerThingWithLock := &CoolerThingWithLock{}
coolerThingWithLock.InitThingWithLock()
return coolerThingWithLock
}
type ThingWithLock struct {
value int
lock sync.Mutex
children []*ThingWithLock
}
func (thingWithLock *ThingWithLock) InitThingWithLock() {
thingWithLock.children = make([]*ThingWithLock, 0)
}
func NewThingWithLock() *ThingWithLock {
newThingWithLock := &ThingWithLock{}
newThingWithLock.InitThingWithLock()
return newThingWithLock
}
func (thingWithLock *ThingWithLock) AddChild(newChild *ThingWithLock) {
thingWithLock.children = append(thingWithLock.children, newChild)
}
func (thingWithLock *ThingWithLock) SetValue(newValue int) {
thingWithLock.lock.Lock()
defer thingWithLock.lock.Unlock()
thingWithLock.value = newValue
for _, child := range thingWithLock.children {
child.SetValue(newValue)
}
}
func main() {
thingOne := NewThingWithLock()
thingTwo := NewCoolerThingWithLock()
thingOne.AddChild(thingTwo)
thingOne.SetValue(42)
}
Error: cannot use thingTwo (type *CoolerThingWithLock) as type
*ThingWithLock in argument to thingOne.AddChild
It's impossible to store the wrapping type in []*ThignWithLock since go has no notion of structural subtyping.
Your assertion that an interface will result in copying is incorrect, and you can get the desired effect by doing:
type InterfaceOfThingThatParticipatesInAHierarchy interface {
AddChild(InterfaceOfThingThatParticipatesInAHierarchy)
SetValue(int)
}
type ThingWithLock struct {
...
children []InterfaceOfThingThatParticipatesInAHierarchy
}
func (thingWithLock *ThingWithLock) AddChild(newChild InterfaceOfThingThatParticipatesInAHierarchy) { ... }
As long as the interface is implemented on a *ThingWithLock and not ThingWithLock, there will be no copying of the receiver struct itself, only the pointer to the struct will be copied on the stack.
I'm learning go, and I would like to explore some patterns.
I would like to build a Registry component which maintains a map of some stuff, and I want to provide a serialized access to it:
Currently I ended up with something like this:
type JobRegistry struct {
submission chan JobRegistrySubmitRequest
listing chan JobRegistryListRequest
}
type JobRegistrySubmitRequest struct {
request JobSubmissionRequest
response chan Job
}
type JobRegistryListRequest struct {
response chan []Job
}
func NewJobRegistry() (this *JobRegistry) {
this = &JobRegistry{make(chan JobRegistrySubmitRequest, 10), make(chan JobRegistryListRequest, 10)}
go func() {
jobMap := make(map[string] Job)
for {
select {
case sub := <- this.submission:
job := MakeJob(sub.request) // ....
jobMap[job.Id] = job
sub.response <- job.Id
case list := <- this.listing:
res := make([]Job, 0, 100)
for _, v := range jobMap {
res = append(res, v)
}
list.response <- res
}
/// case somechannel....
}
}()
return
}
Basically, I encapsulate each operation inside a struct, which carries
the parameters and a response channel.
Then I created helper methods for end users:
func (this *JobRegistry) List() ([]Job, os.Error) {
res := make(chan []Job, 1)
req := JobRegistryListRequest{res}
this.listing <- req
return <-res, nil // todo: handle errors like timeouts
}
I decided to use a channel for each type of request in order to be type safe.
The problem I see with this approach are:
A lot of boilerplate code and a lot of places to modify when some param/return type changes
Have to do weird things like create yet another wrapper struct in order to return errors from within the handler goroutine. (If I understood correctly there are no tuples, and no way to send multiple values in a channel, like multi-valued returns)
So, I'm wondering whether all this makes sense, or rather just get back to good old locks.
I'm sure that somebody will find some clever way out using channels.
I'm not entirely sure I understand you, but I'll try answering never the less.
You want a generic service that executes jobs sent to it. You also might want the jobs to be serializable.
What we need is an interface that would define a generic job.
type Job interface {
Run()
Serialize(io.Writer)
}
func ReadJob(r io.Reader) {...}
type JobManager struct {
jobs map[int] Job
jobs_c chan Job
}
func NewJobManager (mgr *JobManager) {
mgr := &JobManager{make(map[int]Job),make(chan Job,JOB_QUEUE_SIZE)}
for {
j,ok := <- jobs_c
if !ok {break}
go j.Run()
}
}
type IntJob struct{...}
func (job *IntJob) GetOutChan() chan int {...}
func (job *IntJob) Run() {...}
func (job *IntJob) Serialize(o io.Writer) {...}
Much less code, and roughly as useful.
About signaling errors with an axillary stream, you can always use a helper function.
type IntChanWithErr struct {
c chan int
errc chan os.Error
}
func (ch *IntChanWithErr) Next() (v int,err os.Error) {
select {
case v := <- ch.c // not handling closed channel
case err := <- ch.errc
}
return
}