Golang - Sorting on fan in - sorting

I am randomly generating a bunch of log messages, and after they have been generated, I need to sort them by timestamp before writing them to the logs. I'm utilising the sort.Interface aspect of the sort library so I can sort based on my timestamp. I'm using a fan-in concurrency design, so my sorting function aggregates all the log messages from the goroutines, then sorts them.
Here is my code:
type CommonLogFormat struct {
HostIP string
UserIdent string
User string
Timestamp string
Request string
HttpStatusCode int
Size int
}
type Logs struct {
Messages []*CommonLogFormat
}
func sortByTimestamp(ch chan <- *CommonLogFormat) *Logs {
logs := &Logs{Messages: make([]*CommonLogFormat, 1)}
for i := range ch {
logs.Messages = append(logs.Messages, <- i)
}
sort.Sort(logs)
return logs
}
func (l Logs) Len() int {
return len(l.Messages)
}
func (l Logs) Less(i,j int) bool {
return l.Messages[i].Timestamp < l.Messages[j].Timestamp
}
func (l *Logs) Swap(i,j int) {
l.Messages[i], l.Messages[j] = l.Messages[j], l.Messages[i]
}
However, when I go to receive a log message from the channel, I get this error:
invalid operation: <-i (receive from non-chan type *CommonLogFormat)
Why can't I receive a value from the channel?

I think the error message is pretty self-explanatory. Look at this:
for i := range ch {
logs.Messages = append(logs.Messages, <- i)
}
ch is of type chan <- *CommonLogFormat. ch is a channel. The for range loop over the channel yields the values sent on the channel, which will be stored in the loop variable i. i is not a channel, but the values sent on the channel, so it will be of type *CommonLogFormat.
So no need, and you actually can't receive from it, it is already what you would want to receive from it. Simply append i:
for i := range ch {
logs.Messages = append(logs.Messages, i)
}
The Spec: For statements details what the loop variables are in case of for range:
Range expression 1st value 2nd value
array or slice a [n]E, *[n]E, or []E index i int a[i] E
string s string type index i int see below rune
map m map[K]V key k K m[k] V
channel c chan E, <-chan E element e E
The last line applies in case of ranging over a channel, and the first iteration value is the element.
For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever.

Related

Generic function accepting both channels and slices

I'm trying to write generic function in Golang that would search for a value in slices and in channels in the similar way. Here is an example:
// MinOf returns the smallest number found among the channel / slice contents
func MinOf[T chan int | []int](input T) (result int) {
for _, value := range input {
if result > value {
result = value
}
}
return
}
But I'm getting following compilation error: cannot range over input (variable of type T constrained by chan int|[]int) (T has no core type).
I have tried to create common interface, like so:
type Rangable interface {
chan int | []int
}
// MinOf returns the smallest number found among the channel / slice contents
func MinOf[T Rangable](input T) (result int) {
for _, value := range input {
if result > value {
result = value
}
}
return
}
Though, error has changed to cannot range over input (variable of type T constrained by Rangable) (T has no core type) it remains basically the same...
Is there any way how to solve this task using generics or channels and slices could not be "casted" to same core type?
Thank you for any suggestions and ideas!
You can't do this.
The range expression must have a core type to begin with. Unions with diverse type terms, do not have a core type because there isn't one single underlying type in common.
You can also intuitively see why range requires a core type: the semantics of ranging over slices and channels are different.
Ranging over a channel is potentially a blocking operation, ranging over a slice isn't
The iteration variables are different
for i, item := range someSlice {}
With slices i is the index of type int and item is the type of the slice elements.
for item := range someChan {}
With channels, item is the type of the chan elements and that's the only possible range variable.
The best you can have is a type switch:
func MinOf[T any, U chan T | []T](input U) (result int) {
switch t := any(input).(type) {
case chan T:
// range over chan
case []T:
// range over slice
}
return
}
But again, the behavior of this function (blocking vs. non-blocking) is type dependant, and it's unclear what advantages you get by using generics here.

Prometheus counters: How to get current value with golang client?

I am using counters to count the number of requests. Is there any way to get current value of a prometheus counter?
My aim is to reuse existing counter without allocating another variable.
Golang prometheus client version is 1.1.0.
It's easy, have a function to fetch Prometheus counter value
import (
"github.com/prometheus/client_golang/prometheus"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/log"
)
func GetCounterValue(metric *prometheus.CounterVec) float64 {
var m = &dto.Metric{}
if err := metric.WithLabelValues("label1", "label2").Write(m); err != nil {
log.Error(err)
return 0
}
return m.Counter.GetValue()
}
Currently there is no way to get the value of a counter in the official Golang implementation.
You can also avoid double counting by incrementing your own counter and use an CounterFunc to collect it.
Note: use integral type and atomic to avoid concurrent access issues
// declare the counter as unsigned int
var requestsCounter uint64 = 0
// register counter in Prometheus collector
prometheus.MustRegister(prometheus.NewCounterFunc(
prometheus.CounterOpts{
Name: "requests_total",
Help: "Counts number of requests",
},
func() float64 {
return float64(atomic.LoadUint64(&requestsCounter))
}))
// somewhere in your code
atomic.AddUint64(&requestsCounter, 1)
It is possible to read the value of a counter (or any metric) in the official Golang implementation. I'm not sure when it was added.
This works for me for a simple metric with no vector:
func getMetricValue(col prometheus.Collector) float64 {
c := make(chan prometheus.Metric, 1) // 1 for metric with no vector
col.Collect(c) // collect current metric value into the channel
m := dto.Metric{}
_ = (<-c).Write(&m) // read metric value from the channel
return *m.Counter.Value
}
Update: here's a more general version that works with vectors and on histograms...
// GetMetricValue returns the sum of the Counter metrics associated with the Collector
// e.g. the metric for a non-vector, or the sum of the metrics for vector labels.
// If the metric is a Histogram then number of samples is used.
func GetMetricValue(col prometheus.Collector) float64 {
var total float64
collect(col, func(m dto.Metric) {
if h := m.GetHistogram(); h != nil {
total += float64(h.GetSampleCount())
} else {
total += m.GetCounter().GetValue()
}
})
return total
}
// collect calls the function for each metric associated with the Collector
func collect(col prometheus.Collector, do func(dto.Metric)) {
c := make(chan prometheus.Metric)
go func(c chan prometheus.Metric) {
col.Collect(c)
close(c)
}(c)
for x := range c { // eg range across distinct label vector values
m := dto.Metric{}
_ = x.Write(&m)
do(m)
}
}
While it is possible to obtain counter values in github.com/prometheus/client_golang as pointed at this answer, this looks too complicated. This can be greatly simplified by using an alternative library for exporing Prometheus metrics - github.com/VictoriaMetrics/metrics:
import (
"github.com/VictoriaMetrics/metrics"
)
var requestsTotal = metrics.NewCounter(`http_requests_total`)
//...
func getRequestsTotal() uint64 {
return requestsTotal.Get()
}
E.g. just call Get() function on the needed counter.

Reading from golang channels in order

I'm trying to achieve parallel processing and communication over the channels in go.
What I basically try to solve is process a specifc data in parallel, and get results in order => introduced type Chunk for the purpose (see bellow).
I just make new channel for each chunk processing and keep them in slice => expect to be ordered once I iterate over them afterwards.
Simplified version of my program is (https://play.golang.org/p/RVtDGgUVCV):
package main
import (
"fmt"
)
type Chunk struct {
from int
to int
}
func main() {
chunks := []Chunk{
Chunk{
from: 0,
to: 2,
},
Chunk{
from: 2,
to: 4,
},
}
outChannels := [](<-chan struct {
string
error
}){}
for _, chunk := range chunks {
outChannels = append(outChannels, processChunk(&chunk))
}
for _, outChannel := range outChannels {
for out := range outChannel {
if out.error != nil {
fmt.Printf("[ERROR] %s", out.error)
return
}
fmt.Printf("[STDOUT] %s", out.string)
}
}
}
func processChunk(c *Chunk) <-chan struct {
string
error
} {
outChannel := make(chan struct {
string
error
})
go func() {
outChannel <- struct {
string
error
}{fmt.Sprintf("from: %d to: %d\n", c.from, c.to), nil}
close(outChannel)
}()
return outChannel
}
The output I see is:
[STDOUT] from: 2 to: 4
[STDOUT] from: 2 to: 4
What I'd however expect to see would be:
[STDOUT] from: 0 to: 2
[STDOUT] from: 2 to: 4
What am I doing wrong here? I don't see it.
The trouble is in the very first for loop of main. When you use for range loop, the loop variable (chunk here) gets created once and is assigned a copy of each slice element per iteration.
When you call processChunk(&chunk), you are passing the address of this loop variable, and the value of this variable changes with each iteration. Thus the function processChunk always ends up working on the last item in the chunks loop since that is what *chunk points to after the for loop finishes.
To fix, use slice indexing:
for i := 0; i < len(chunks); i++ {
// pass chunk objects by indexing chunks
outChannels = append(outChannels, processChunk(&chunks[i]))
}
Fixed code: https://play.golang.org/p/A1_DtkncY_
You can read more about range here.

golang, type int does not support indexing

I have a trouble in a part of code. I'm writing on revel framework(to be clear). This is a Worker go routine, and I want it to do several things:
switch the struct type of the stat variable, according to the
source, that would come. I made a switch, but before the all other
code would be correct, I don't really know if switch is written
properly.
I get cache for the date, and put it in new Work item.
I send Work to channel
here is what I got by now:
func worker(in <-chan Task, out chan <- Work, wg *sync.WaitGroup) {
for t := range in {
for sourceName, charts := range t.Request.Charts {
var stat interface{}
switch sourceName {
case "noagg":
stat = stat.([]NoaggModel)
case "oracle":
stat = stat.([]OracleModel)
default:
panic("Invalid type for Work model!")
}
w := Work{Name:"", Data:""}
err := cache.Get(string(sourceName)+"_"+string(t.Date), &stat);
for chart := range charts{
w.Name = chart["name"]
if err == nil{
w.Data = countDataByName( stat, t.Request.Filters, string(chart["name"]))
}
out <- w
}
}
}
wg.Done() // this worker is now done; let the WaitGroup know.
}
But now I got error that invalid operation: chart["name"] (type int does not support indexing)
But I have structs :
type Chart struct {
Name string `json:"name"`
Type string `json:"type"`
}
type Filter struct {
DayStart string `json:"dayStart"`
DayEnd string `json:"dayEnd"`
TimePeriods interface{} `json:"timePeriods"`
Lines []string `json:"lines"`
}
type Task struct {
Date string
Request ChartOptins
}
type Work struct {
Name string
Data interface{}
}
How should I write in a better way the correct switch, if the type of struct for cache can be different, and why is my name adding is bad and call error?
The for in the slice is missing a variable
for chart := range charts{
when iterating on a slice the first variable is the key and the second is the real value you want. In this case you can omit the key (an int) so the proper instruction should be
for _, chart := range charts{

Design patterns for map channel?

I am writing a DNS protocol parser in golang, the idea is to use a map like this
var tidMap map[uint16] (chan []byte)
So for the tidMap map, key is the tid (transaction ID), value is a byte array channel.
The idea is that a goroutine will try get value from the channel, another goroutine will try read bytes by listening every imcoming packet, and once found transaction ID, will set response data to the tidMap, so the former goroutine will continue handle the response.
One problem with the design is that I need the make sure the channel has buffer length of 1, so extra values can be pushed into channel without blocking.
So how can I specify channel buffer length in tidMap declaration?
var tidMap map[int] make(chan int, 1)
You can't use make() there.
The length of the channel buffer doesn't convey type, so you will have to add logic to test if the map entry exists, if it doesn't:
tidMap[0] = make(chan int, 1)
The short answer: you can't. When you make a map, you define the data types of its keys and values, and the capacity of a channel is not part of its type.
The longer answer is: create an abstract data type that hides this implementation detail. Something like this:
type ChannelMap struct {
tidMap map[int](chan []byte)
}
func NewChannelMap() *ChannelMap { ... }
func (c *ChannelMap) Put(tid int) (chan int) {
res := make(chan int, 1)
c.tidMap[tid] = res
return res
}
func (c *ChannelMap) Get(tid int) (chan int) {
return c.tidMap[tid]
}
And just to be sure: giving the channel a capacity of 1 does not ensure that senders will never block; if your channel consumers are too slow, producers can fill the channel up to its capacity and will then block until the channel has room again.

Resources