store multiple values in maps - go

problem: Store both an IP address as well as a time to a counter. something like this
ip, time, count
and then i would like to be able to increment the count for each ip:
ip++
in my map, and then on a given interval, i would like to iterate all keys and find the keys with a time stamp older than N minutes.
I need this to make sure i don't "forget" a key in memory if for some reason, client disconnect and i dont delete the key properly.
http://play.golang.org/p/RiWWOCARq7
Question: How do i add a time stamp to every IP address i store in my map. Also i need this be be used across multiple go routines
Im new to both programming and Golang, so if this is not even the correct approach, if anyone could point me in the right direction, i would appreciate it.

For example,
package main
import (
"sync"
"time"
)
type IPCounter struct {
IPAddr string
Time time.Time
Count int
}
type ipCounterMap struct {
counters map[string]IPCounter
mutex sync.RWMutex
}
var ipCounters = ipCounterMap{counters: make(map[string]IPCounter)}
// Get IP address counter
func Counter(ipAddr string) IPCounter {
ipCounters.mutex.RLock()
defer ipCounters.mutex.RUnlock()
counter, found := ipCounters.counters[ipAddr]
if !found {
counter.IPAddr = ipAddr
}
return counter
}
// Increment IP address counter
func Incr(ipAddr string) {
now := time.Now().UTC()
ipCounters.mutex.Lock()
defer ipCounters.mutex.Unlock()
counter, found := ipCounters.counters[ipAddr]
if !found {
counter.IPAddr = ipAddr
}
counter.Time = now
counter.Count++
ipCounters.counters[ipAddr] = counter
}
// Delete IP address counter
func Delete(ipAddr string) {
ipCounters.mutex.Lock()
defer ipCounters.mutex.Unlock()
delete(ipCounters.counters, ipAddr)
}
// Get old IP address counters old durations ago
func OldIPCounters(old time.Duration) []IPCounter {
var counters []IPCounter
oldTime := time.Now().UTC().Add(-old)
ipCounters.mutex.RLock()
defer ipCounters.mutex.RUnlock()
for _, counter := range ipCounters.counters {
if counter.Time.Before(oldTime) {
counters = append(counters, counter)
}
}
return counters
}
func main() {}

You probably want a map of ip -> struct { ip, counter, lastTime }
that way, you can look up the counter by ip and then update it
var Counters = map[string]*Counter{}
type Counter struct {
ip string
count int
lastTime time.Time
}
Here's a working example on Play http://play.golang.org/p/TlCTc_4iq5
Adding the find older than is simply range over the values of the map comparing to now and do something when it's old enough.

It sounds like you need a struct. A map is for storing a collection where all the keys are of one type and all the elements are of another (or the same type).
For grouping a collection of variables of different data types, that are related, use a struct
e.g.
type IPCounter struct {
ip string
time time.Time
count int
}
Here's some sample code that creates one of these objects and increments the count:
package main
import (
"fmt"
"time"
)
type IPAddressCounter struct {
ip string
time time.Time
count int
}
func (i *IPAddressCounter) IncrementCount() {
i.count++
}
func makeCounter(ip string) IPAddressCounter {
return IPAddressCounter{
ip: ip,
time: time.Now(),
}
}
func main() {
mapOfIPCounters := make(map[string]IPAddressCounter)
mapOfIPCounters["192.168.1.1"] = makeCounter("192.168.1.1")
mapOfIPCounters["192.168.1.2"] = makeCounter("192.168.1.2")
mapOfIPCounters["192.168.1.3"] = makeCounter("192.168.1.3")
for key, value := range mapOfIPCounters {
value.IncrementCount()
mapOfIPCounters[key] = value
fmt.Println("The counter for "+key+" is", mapOfIPCounters[key].count)
}
}

Related

How to create a unique key for a map

I'm creating a structure where a developer can store a reference to something and retrieve it when needed using a reference key, but not delete the reference.
How to create a unique key for a map that is generated at the point of insertion?
So far I'm using a not exported pointer to an empty *struct{} , but wondering if there there is a better way.
package main
import "fmt"
type referenceKey **struct{}
type reference[K referenceKey, R any] struct {
s map[K]R
}
func (ref *reference[K, R]) Set(reference R) *K {
if ref.s == nil {
ref.s = make(map[K]R)
}
key := new(struct{})
refKey := K(&key)
ref.s[refKey] = reference
return &refKey
}
func (ref *reference[K, R]) Get(key *K) R {
return ref.s[*key]
}
func main() {
ref := &reference[referenceKey, int]{}
key1 := ref.Set(77)
key2 := ref.Set(15345351)
fmt.Println(ref.Get(key2))
fmt.Println(ref.Get(key1))
}
https://go.dev/play/p/SF6S5BNlP7N
EDIT:
With int I have to keep a reference to it and increment it. To reduce code I tried to use the pointer to a instance of a empty struct.
Basically I need to create a unique key that can't be created outside internal package nor can Get method be called with an invalid key.
I've also fixed the code to now allow anyone to create a new invalid key.
Based on the comments I would make it with a counter and protect the store with a mutex for concurrent use:
package main
import (
"fmt"
"sync"
)
type reference[T any] struct {
mutex sync.RWMutex
store map[uint64]T
counter uint64
}
func (r *reference[T]) Set(item T) uint64 {
r.mutex.Lock()
defer r.mutex.Unlock()
if r.store == nil {
r.store = make(map[uint64]T)
}
r.counter++
r.store[r.counter] = item
return r.counter
}
func (r *reference[T]) Get(key uint64) T {
r.mutex.RLock()
defer r.mutex.RUnlock()
return r.store[key]
}
func main() {
ref := reference[int]{}
key1 := ref.Set(77)
key2 := ref.Set(15345351)
fmt.Println(ref.Get(key2))
fmt.Println(ref.Get(key1))
}
But if in the future you need to be secure, and the keys should be random, then change the map's key to string, remove the counter and use uuid.NewString() for the new key value.

Prometheus counters: How to get current value with golang client?

I am using counters to count the number of requests. Is there any way to get current value of a prometheus counter?
My aim is to reuse existing counter without allocating another variable.
Golang prometheus client version is 1.1.0.
It's easy, have a function to fetch Prometheus counter value
import (
"github.com/prometheus/client_golang/prometheus"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/log"
)
func GetCounterValue(metric *prometheus.CounterVec) float64 {
var m = &dto.Metric{}
if err := metric.WithLabelValues("label1", "label2").Write(m); err != nil {
log.Error(err)
return 0
}
return m.Counter.GetValue()
}
Currently there is no way to get the value of a counter in the official Golang implementation.
You can also avoid double counting by incrementing your own counter and use an CounterFunc to collect it.
Note: use integral type and atomic to avoid concurrent access issues
// declare the counter as unsigned int
var requestsCounter uint64 = 0
// register counter in Prometheus collector
prometheus.MustRegister(prometheus.NewCounterFunc(
prometheus.CounterOpts{
Name: "requests_total",
Help: "Counts number of requests",
},
func() float64 {
return float64(atomic.LoadUint64(&requestsCounter))
}))
// somewhere in your code
atomic.AddUint64(&requestsCounter, 1)
It is possible to read the value of a counter (or any metric) in the official Golang implementation. I'm not sure when it was added.
This works for me for a simple metric with no vector:
func getMetricValue(col prometheus.Collector) float64 {
c := make(chan prometheus.Metric, 1) // 1 for metric with no vector
col.Collect(c) // collect current metric value into the channel
m := dto.Metric{}
_ = (<-c).Write(&m) // read metric value from the channel
return *m.Counter.Value
}
Update: here's a more general version that works with vectors and on histograms...
// GetMetricValue returns the sum of the Counter metrics associated with the Collector
// e.g. the metric for a non-vector, or the sum of the metrics for vector labels.
// If the metric is a Histogram then number of samples is used.
func GetMetricValue(col prometheus.Collector) float64 {
var total float64
collect(col, func(m dto.Metric) {
if h := m.GetHistogram(); h != nil {
total += float64(h.GetSampleCount())
} else {
total += m.GetCounter().GetValue()
}
})
return total
}
// collect calls the function for each metric associated with the Collector
func collect(col prometheus.Collector, do func(dto.Metric)) {
c := make(chan prometheus.Metric)
go func(c chan prometheus.Metric) {
col.Collect(c)
close(c)
}(c)
for x := range c { // eg range across distinct label vector values
m := dto.Metric{}
_ = x.Write(&m)
do(m)
}
}
While it is possible to obtain counter values in github.com/prometheus/client_golang as pointed at this answer, this looks too complicated. This can be greatly simplified by using an alternative library for exporing Prometheus metrics - github.com/VictoriaMetrics/metrics:
import (
"github.com/VictoriaMetrics/metrics"
)
var requestsTotal = metrics.NewCounter(`http_requests_total`)
//...
func getRequestsTotal() uint64 {
return requestsTotal.Get()
}
E.g. just call Get() function on the needed counter.

Do map of pointers is different with common way of using maps

I want to create cache with map. As map doesn't allow reference to its value, so it's not possible to change values in called functions.
After some search, I found, it's possible with creating map of pointer (of struct). It Almost solve problem and can work like variable by reference
But as i found a few using of this method for map. I worry about using it to be safe.
Is anyone has experience of using map of pointer? and is it right way to use it?
package main
import "fmt"
type Cache struct {
name string
counter int
}
func incr(c Cache) {
c.counter += 1
}
func incrp(c *Cache) {
c.counter += 2
}
func main() {
m := make(map[string]Cache)
m["james"] = Cache{name: "James", counter: 10}
c := m["james"]
incr(c)
fmt.Println(c.name, c.counter) // James 10
mp := make(map[string]*Cache)
mp["james"] = &Cache{name: "James", counter: 10}
cp := mp["james"]
incrp(cp)
fmt.Println(cp.name, cp.counter) // James 12
}
edited: My text had some confusing words and sentences, that caused to misunderstanding, so i tried to fixed it
You can accomplish this and still have a map of non-pointers, with a pointer receiver on the struct:
package main
import "fmt"
type Cache struct {
name string
counter int
}
func (c *Cache) incr() { // the '(c *Cache)' is the receiver;
c.counter += 1 // it makes incr() a method, not just a function
}
func main() {
m := make(map[string]Cache)
m["james"] = Cache{name: "James", counter: 10}
c := m["james"]
c.incr()
fmt.Println(c.name, c.counter)
}
Output:
James 11
If receivers and methods are new to you, here is where they are mentioned in the Tour of Go: https://tour.golang.org/methods/1
Note the page about pointer receivers a few steps later in the Tour: https://tour.golang.org/methods/4

How to access unexported struct fields

Is there a way to use reflect to access unexported fields in Go 1.8?
This no longer seems to work: https://stackoverflow.com/a/17982725/555493
Note that reflect.DeepEqual works just fine (that is, it can access unexported fields) but I can't make heads or tails of that function. Here's a go playarea that shows it in action: https://play.golang.org/p/vyEvay6eVG. The src code is below
import (
"fmt"
"reflect"
)
type Foo struct {
private string
}
func main() {
x := Foo{"hello"}
y := Foo{"goodbye"}
z := Foo{"hello"}
fmt.Println(reflect.DeepEqual(x,y)) //false
fmt.Println(reflect.DeepEqual(x,z)) //true
}
If the struct is addressable, you can use unsafe.Pointer to access the field (read or write) it, like this:
rs := reflect.ValueOf(&MyStruct).Elem()
rf := rs.Field(n)
// rf can't be read or set.
rf = reflect.NewAt(rf.Type(), unsafe.Pointer(rf.UnsafeAddr())).Elem()
// Now rf can be read and set.
See full example on the playground.
This use of unsafe.Pointer is valid according to the documentation and running go vet returns no errors.
If the struct is not addressable this trick won't work, but you can create an addressable copy like this:
rs = reflect.ValueOf(MyStruct)
rs2 := reflect.New(rs.Type()).Elem()
rs2.Set(rs)
rf = rs2.Field(0)
rf = reflect.NewAt(rf.Type(), unsafe.Pointer(rf.UnsafeAddr())).Elem()
// Now rf can be read. Setting will succeed but only affects the temporary copy.
See full example on the playground.
Based on cpcallen's work:
import (
"reflect"
"unsafe"
)
func GetUnexportedField(field reflect.Value) interface{} {
return reflect.NewAt(field.Type(), unsafe.Pointer(field.UnsafeAddr())).Elem().Interface()
}
func SetUnexportedField(field reflect.Value, value interface{}) {
reflect.NewAt(field.Type(), unsafe.Pointer(field.UnsafeAddr())).
Elem().
Set(reflect.ValueOf(value))
}
reflect.NewAt might be confusing to read at first. It returns a reflect.Value representing a pointer to a value of the specified field.Type(), using unsafe.Pointer(field.UnsafeAddr()) as that pointer. In this context reflect.NewAt is different than reflect.New, which would return a pointer to a freshly initialized value.
Example:
type Foo struct {
unexportedField string
}
GetUnexportedField(reflect.ValueOf(&Foo{}).Elem().FieldByName("unexportedField"))
https://play.golang.org/p/IgjlQPYdKFR
reflect.DeepEqual() can do it because it has access to unexported features of the reflect package, in this case namely for the valueInterface() function, which takes a safe argument, which denies access to unexported field values via the Value.Interface() method if safe=true. reflect.DeepEqual() will (might) call that passing safe=false.
You can still do it, but you cannot use Value.Interface() for unexported fields. Instead you have to use type-specific methods, such as Value.String() for string, Value.Float() for floats, Value.Int() for ints etc. These will return you a copy of the value (which is enough to inspect it), but will not allow you to modify the field's value (which might be "partly" possible if Value.Interface() would work and the field type would be a pointer type).
If a field happens to be an interface type, you may use Value.Elem() to get to the value contained / wrapped by the interface value.
To demonstrate:
type Foo struct {
s string
i int
j interface{}
}
func main() {
x := Foo{"hello", 2, 3.0}
v := reflect.ValueOf(x)
s := v.FieldByName("s")
fmt.Printf("%T %v\n", s.String(), s.String())
i := v.FieldByName("i")
fmt.Printf("%T %v\n", i.Int(), i.Int())
j := v.FieldByName("j").Elem()
fmt.Printf("%T %v\n", j.Float(), j.Float())
}
Output (try it on the Go Playground):
string hello
int64 2
float64 3
package main
import (
"fmt"
"reflect"
"strings"
"unsafe"
)
type Person1 struct {
W3ID string
Name string
}
type Address1 struct {
city string
country string
}
type User1 struct {
name string
age int
address Address1
manager Person1
developer Person1
tech Person1
}
func showDetails(load, email interface{}) {
if reflect.ValueOf(load).Kind() == reflect.Struct {
typ := reflect.TypeOf(load)
value := reflect.ValueOf(load)
//#1 For struct, not addressable create a copy With Element.
value2 := reflect.New(value.Type()).Elem()
//#2 Value2 is addressable and can be set
value2.Set(value)
for i := 0; i < typ.NumField(); i++ {
if value.Field(i).Kind() == reflect.Struct {
rf := value2.Field(i)
/* #nosec G103 */
rf = reflect.NewAt(rf.Type(), unsafe.Pointer(rf.UnsafeAddr())).Elem()
irf := rf.Interface()
typrf := reflect.TypeOf(irf)
nameP := typrf.String()
if strings.Contains(nameP, "Person") {
//fmt.Println(nameP, "FOUND !!!!!!! ")
for j := 0; j < typrf.NumField(); j++ {
re := rf.Field(j)
nameW := typrf.Field(j).Name
if strings.Contains(nameW, "W3ID") {
valueW := re.Interface()
fetchEmail := valueW.(string)
if fetchEmail == email {
fmt.Println(fetchEmail, " MATCH!!!!")
}
}
}
}
showDetails(irf, email)
} else {
// fmt.Printf("%d.Type:%T || Value:%#v\n",
// (i + 1), value.Field(i), value.Field(i))
}
}
}
}
func main() {
iD := "tsumi#in.org.com"
load := User1{
name: "John Doe",
age: 34,
address: Address1{
city: "New York",
country: "USA",
},
manager: Person1{
W3ID: "jBult#in.org.com",
Name: "Bualt",
},
developer: Person1{
W3ID: "tsumi#in.org.com",
Name: "Sumi",
},
tech: Person1{
W3ID: "lPaul#in.org.com",
Name: "Paul",
},
}
showDetails(load, iD)
}

golang, type int does not support indexing

I have a trouble in a part of code. I'm writing on revel framework(to be clear). This is a Worker go routine, and I want it to do several things:
switch the struct type of the stat variable, according to the
source, that would come. I made a switch, but before the all other
code would be correct, I don't really know if switch is written
properly.
I get cache for the date, and put it in new Work item.
I send Work to channel
here is what I got by now:
func worker(in <-chan Task, out chan <- Work, wg *sync.WaitGroup) {
for t := range in {
for sourceName, charts := range t.Request.Charts {
var stat interface{}
switch sourceName {
case "noagg":
stat = stat.([]NoaggModel)
case "oracle":
stat = stat.([]OracleModel)
default:
panic("Invalid type for Work model!")
}
w := Work{Name:"", Data:""}
err := cache.Get(string(sourceName)+"_"+string(t.Date), &stat);
for chart := range charts{
w.Name = chart["name"]
if err == nil{
w.Data = countDataByName( stat, t.Request.Filters, string(chart["name"]))
}
out <- w
}
}
}
wg.Done() // this worker is now done; let the WaitGroup know.
}
But now I got error that invalid operation: chart["name"] (type int does not support indexing)
But I have structs :
type Chart struct {
Name string `json:"name"`
Type string `json:"type"`
}
type Filter struct {
DayStart string `json:"dayStart"`
DayEnd string `json:"dayEnd"`
TimePeriods interface{} `json:"timePeriods"`
Lines []string `json:"lines"`
}
type Task struct {
Date string
Request ChartOptins
}
type Work struct {
Name string
Data interface{}
}
How should I write in a better way the correct switch, if the type of struct for cache can be different, and why is my name adding is bad and call error?
The for in the slice is missing a variable
for chart := range charts{
when iterating on a slice the first variable is the key and the second is the real value you want. In this case you can omit the key (an int) so the proper instruction should be
for _, chart := range charts{

Resources