Problems with map, goroutine and mutex - go

This project is made to receive POST routes that will finally count as access to later write to a database. The intuition is to save interaction with the database of another project in production. I decided to do it in go, but I'm new to the language and I'm struggling to understand. I'm trying to make it so that there is no loss or that there are more accesses.
The project basically consists of a controller, a service and two models, just enough to meet the need for which it was created. In my controller I have the function that will be responsible for receiving the POST.
controllers/views.go:
func StoreViews(c *fiber.Ctx) error {
var songview models.SongView
err := c.BodyParser(&songview)
if err != nil {
return c.Status(403).JSON(fiber.Map{
"errors": fiber.Map{"request": err.Error()},
})
}
songview.Date = time.Now()
errs := utils.ValidateStruct(songview)
if len(errs) > 0 {
return c.Status(403).JSON(map[string]interface{}{"errors": errs})
}
go services.StoreViews(songview)
return c.SendStatus(fiber.StatusOK)
}
To handle the received data I made these three functions in my service:
services/views.go
var (
StoreViewsMap = make(map[string]*models.SongView)
StoreControl sync.RWMutex
)
func StoreViews(sview models.SongView) bool {
nameKey := strconv.Itoa(int(sview.SongId)) + sview.Lang + sview.Date.Format("2006-01-02")
songview := getSongView(nameKey)
initSongView(nameKey, songview, sview)
return true
}
func getSongView(name string) *models.SongView {
StoreControl.RLock()
defer StoreControl.RUnlock()
return StoreViewsMap[name]
}
func initSongView(name string, songview *models.SongView, sview models.SongView) bool {
StoreControl.Lock()
defer StoreControl.Unlock()
if songview == nil {
insert := models.SongView{
SongId: sview.SongId,
Lang: sview.Lang,
Date: sview.Date,
Views: 0,
}
songViewNew := &insert // see if & is needed
StoreViewsMap[name] = songViewNew
} else {
songview.Views = songview.Views + 1
}
return true
}
I tried to implement RWMutex to get it to do everything without overlapping anything, but it's not working as it should, sometimes it disappears with views, other times it rescues "songview" in the getSongView function wrongly, among several other problems that I found modifying and reviewing my code. The current code is not in the version that I managed to get closer to the expected result, but I didn't save this version so I decided to bring the current code to exemplify what I'm facing.
I would like you to help me understand how I can deal with several concurrent requests disputing, how I can interact with the data in the best possible way and if there is an error in the use of a pointer I am open to understand. To simulate a POST "attack" to my code I'm using this code in another main.go I made for this test.
var limit int = 10
func main() {
channel := make(chan string)
for i := 0; i < limit; i++ {
go func(i int) {
post("http://localhost:3000/views/store", "lang=pt&song_id=296", i)
channel <- "ok"
}(i)
go func(i int) {
post("http://localhost:3000/views/store", "lang=en&song_id=3016", i)
channel <- "ok"
}(i)
go func(i int) {
post("http://localhost:3000/views/store", "lang=pt&song_id=3016", i)
channel <- "ok"
}(i)
}
for i := 0; i < limit*3; i++ {
<-channel
}
}
func post(url string, json string, index int) {
payload := strings.NewReader(json)
client := &http.Client{}
req, err := http.NewRequest("POST", url, payload)
if err != nil {
fmt.Println(err)
return
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer res.Body.Close()
_, err = ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
return
}
if res.StatusCode != 200 {
fmt.Println(res.StatusCode)
}
}
My song-view model is this: (I'm just using it to sort the data, although the project is connected to the bank of the project in production, it is read-only)
type SongView struct {
Id int64 `json:"id"`
SongId int64 `json:"song_id" form:"song_id" gorm:"notNull" validate:"required,number"`
ArtistId int64 `json:"artist_id"`
Lang string `json:"lang" validate:"required,oneof=pt en es de fr"`
Date time.Time `json:"date" gorm:"column:created_at" validate:"required"`
Views int64 `json:"views"`
}

I believe that this code can be written a little more easily in Go, but that is not the question. From your description, it appears that the data is lost somewhere. Have you tried the Go data race detector tool? Below is a link
https://go.dev/doc/articles/race_detector
Can you provide examples of input data where errors/missing items appear?

It happens because your code has a race condition in between read and write to map.
Example:
G1 - goroutine 1
G2 - goroutine 2
G1: ReadLock and Read songview named "MySong"
G1: MySong doesn't exists. Nil will be inserted and returned.
G1: Unlock
G2: ReadLock and Read songview named "MySong"
G2: MySong exist Nil. Nil will be returned.
G1: WriteLock.
G1: songView = nil, so create a new one. Set counter to 1.
G1: set counter to 1. Insert to map on key "MySong"
G1: Unlock
G2: WriteLock: songView = nil(because you read it on step 2). Create new SongView. Set counter to 1.
G2: unlock
As a result you have 1 "MySong" with counter 1 because you rewrite a previous value.
The idea of locking - Atomicity. So, all your operation should be atomic.
func initSongView(name string, sview models.SongView) bool {
StoreControl.Lock()
defer StoreControl.Unlock()
songview := StoreViewsMap[name]
if songview == nil {
insert := models.SongView{
SongId: sview.SongId,
Lang: sview.Lang,
Date: sview.Date,
Views: 1, // counter should be 1 because it's a first view
}
StoreViewsMap[name] = &insert
} else {
songview.Views = songview.Views + 1
}
return true
}

Related

Parsing prometheus metrics from file and updating counters

I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.

How to get the result and the status of a transaction

I am trying to work with the Ethereum blockchain, with the Solidity contracts.
I am currently deploying a contract and performing some actions on it, but I would like to know how to get some 'feedback/callback/returns' of a specific transaction.
Is there a way to set the status of a transaction to 0 (error) and still get events, for example ?
if (id.length <= 0) {
emit Result("KO", "1");
revert();
}
This will not work (no event) because I revert everything, but the status will be set to 0
if (id.length <= 0) {
emit Result("KO", "1");
return;
}
I will get some event, but the status will stay 1
if (id.length <= 0) {
revert("KO_1");
}
The status will be 0, but I will not have any event
Here is my go code to perform the action :
func testFunction(id []byte) {
//...
//...
tx, err := instance.Action(opt, id)
if (errors.HasError(err)) {
return
}
callbackValue := subscribeToContract(tx.Hash().Hex())
logs.Pretty(tx, callbackValue)
//...
//...
}
func subscribeToContract(hashToRead string) myStruct {
query := ethereum.FilterQuery{
Addresses: []common.Address{address},
}
soc := make(chan types.Log)
sub, err := WssClient.SubscribeFilterLogs(context.Background(), query, soc)
if err != nil {
logs.Error(err)
}
for {
select {
case err := <-sub.Err():
logs.Info(`SOMETHING ERROR`)
logs.Error(err)
case vLog := <-soc:
logs.Info(`SOMETHING`)
contractAbi, _ := abi.JSON(strings.NewReader(string(SignABI)))
event := myStruct{}
contractAbi.Unpack(&event, "Result", vLog.Data)
logs.Info(`New Event from [` + vLog.TxHash.Hex() + `] : ` + event.Message)
}
}
}
If id.length > 0, all good.
But if id.length <= 0, I have no callback from the subscribeToContract function.
Is there a way to directly have the result status, or should loop with tx, err := client.TransactionReceipt(context.Background(), txHash) until I get a Status?
I didn't found any way to subscribe to a specific status change for a transaction, but a workaround :
The go-ethereum package provides 2 functions SubscribeFilterLogs and SubscribeNewHead. We can use the first one to get the logs (if relevant) and the second one to get a block information :
SubscribeNewHead subscribes to notifications about the current
blockchain head on the given channel.
A transaction can be validated or rejected/reverted when a block is mined, so we can use this 'trick'
func checkTransactionReceipt(_txHash string) int {
client, _ := getClient("https://ropsten.infura.io/v3/XXXXXX")
txHash := common.HexToHash(_txHash)
tx, err := client.TransactionReceipt(context.Background(), txHash)
if (Error.HasError(err)) {
return (-1)
}
return (int(tx.Status))
}
func WaitForBlockCompletation(data EthData, hashToRead string) int {
soc := make(chan *types.Header)
sub, err := data.WssClient.SubscribeNewHead(context.Background(), soc)
if (err != nil) {
return -1
}
for {
select {
case err := <-sub.Err():
_ = err
return -1
case header := <-soc:
logs.Info(header.TxHash.Hex())
transactionStatus := checkTransactionReceipt(hashToRead)
if (transactionStatus == 0) {
//FAILURE
sub.Unsubscribe()
return 0
} else if (transactionStatus == 1) {
//SUCCESS
sub.Unsubscribe()
return 1
}
}
}
}
Basically we are waiting for the block to be mined, then we check the TransactionReceipt which fail with an error (not found) if the transaction is not yet validated/rejected. Then, if the transaction is, we can unsubscribe the subscription and return the transaction status (0 fail, 1 success).
Not sure if it's the worst, best, only method, but it's working ! Be free to improve this solution !
An easier/newer solution:
I think the function waitMined is the function you are looking for.
bind.WaitMined(context.Background(), client, signedTx)
Originally posted in here.

How to make goroutines work with anonymous functions returning value in a loop

I am working on a custom script to fetch data from RackSpace cloudfiles container and make a list of all the files in a given container (container has around 100 million files) and I have been working on parallelizing the code and currently stuck.
// function to read data from channel and display
// currently just displaying, but there will be allot of processing done on this data
func extractObjectItemsFromList(objListChan <-chan []string) {
fmt.Println("ExtractObjectItemsFromList")
for _, c := range <-objListChan {
fmt.Println(urlPrefix, c, "\t", count)
}
}
func main()
// fetching data using flags
ao := gophercloud.AuthOptions{
Username: *userName,
APIKey: *apiKey,
}
provider, err := rackspace.AuthenticatedClient(ao)
client, err := rackspace.NewObjectStorageV1(provider,gophercloud.EndpointOpts{
Region: *region,
})
if err != nil {
logFatal(err)
}
// We have the option of filtering objects by their attributes
opts := &objects.ListOpts{
Full: true,
Prefix: *prefix,
}
var objectListChan = make(chan []string)
go extractObjectItemsFromList(objectListChan)
// Retrieve a pager (i.e. a paginated collection)
pager := objects.List(client, *containerName, opts)
// Not working
// By default EachPage contains 10000 records
// Define an anonymous function to be executed on each page's iteration
lerr := pager.EachPage(func(page pagination.Page) (bool, error) { // Get a slice of objects.Object structs
objectList, err := objects.ExtractNames(page)
if err != nil {
logFatal(err)
}
for _, o := range objectList {
_ = o
}
objectListChan <- objectList
return true, nil
})
if lerr != nil {
logFatal(lerr)
}
//---------------------------------------------------
// below code is working
//---------------------------------------------------
// working, but only works inside the loop, this keeps on fetching new pages and showing new records, 10000 per page
// By default EachPage contains 10000 records
// Define an anonymous function to be executed on each page's iteration
lerr := pager.EachPage(func(page pagination.Page) (bool, error) { // Get a slice of objects.Object structs
objectList, err := objects.ExtractNames(page)
if err != nil {
logFatal(err)
}
for _, o := range objectList {
fmt.Println(o)
}
return true, nil
})
if lerr != nil {
logFatal(lerr)
}
The first 10000 records are displayed but then it stuck and nothing happens. If I do not use channel and just run the plain loop it works perfectly fine, which kills the purpose of parallelizing.
for _, c := range <-objListChan {
fmt.Println(urlPrefix, c, "\t", count)
}
Your async worker pops one list from the channel, iterates it and exits. You need to have two loops: one reading the channel (range objListChan), the other - reading the (just retrieved) object list.

Understand this code (golang), double parantheses ()()

I was wondering if someone can explain this syntax to me. In the google maps go api, they have
type Client struct {
httpClient *http.Client
apiKey string
baseURL string
clientID string
signature []byte
requestsPerSecond int
rateLimiter chan int
}
// NewClient constructs a new Client which can make requests to the Google Maps WebService APIs.
func NewClient(options ...ClientOption) (*Client, error) {
c := &Client{requestsPerSecond: defaultRequestsPerSecond}
WithHTTPClient(&http.Client{})(c) //???????????
for _, option := range options {
err := option(c)
if err != nil {
return nil, err
}
}
if c.apiKey == "" && (c.clientID == "" || len(c.signature) == 0) {
return nil, errors.New("maps: API Key or Maps for Work credentials missing")
}
// Implement a bursty rate limiter.
// Allow up to 1 second worth of requests to be made at once.
c.rateLimiter = make(chan int, c.requestsPerSecond)
// Prefill rateLimiter with 1 seconds worth of requests.
for i := 0; i < c.requestsPerSecond; i++ {
c.rateLimiter <- 1
}
go func() {
// Wait a second for pre-filled quota to drain
time.Sleep(time.Second)
// Then, refill rateLimiter continuously
for _ = range time.Tick(time.Second / time.Duration(c.requestsPerSecond)) {
c.rateLimiter <- 1
}
}()
return c, nil
}
// WithHTTPClient configures a Maps API client with a http.Client to make requests over.
func WithHTTPClient(c *http.Client) ClientOption {
return func(client *Client) error {
if _, ok := c.Transport.(*transport); !ok {
t := c.Transport
if t != nil {
c.Transport = &transport{Base: t}
} else {
c.Transport = &transport{Base: http.DefaultTransport}
}
}
client.httpClient = c
return nil
}
}
And this is the line I don't understand in NewClient
WithHTTPClient(&http.Client{})(c)
Why are there two ()()?
I see that WithHTTPClient takes in a *http.Client which that line does, but then it also passes in a pointer to the client struct declared above it?
WithHTTPClient returns a function, ie:
func WithHTTPClient(c *http.Client) ClientOption {
return func(client *Client) error {
....
return nil
}
}
WithHTTPClient(&http.Client{})(c) is just calling that function with c (a pointer to a Client) as parameter. It could be written as:
f := WithHTTPClient(&http.Client{})
f(c)

Output of GET request different to view source

I'm trying to extract match data from whoscored.com. When I view the source on firefox, I find on line 816 a big json string with the data I want for that matchid. My goal is to eventually get this json.
In doing this, I've tried to download every page of https://www.whoscored.com/Matches/ID/Live where ID is the id of the match. I wrote a little Go program to GET request each ID up to a certain point:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
// http://www.whoscored.com/Matches/614052/Live is the match for
// Eveton vs Manchester
const match_address = "http://www.whoscored.com/Matches/"
// the max id we get
const max_id = 300
const num_workers = 10
// function that get the bytes of the match id from the website
func match_fetch(matchid int) {
url := fmt.Sprintf("%s%d/Live", match_address, matchid)
resp, err := http.Get(url)
if err != nil {
fmt.Println(err)
return
}
// if we sucessfully got a response, store the
// body in memory
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
// write the body to memory
pwd, _ := os.Getwd()
filepath := fmt.Sprintf("%s/match_data/%d", pwd, matchid)
err = ioutil.WriteFile(filepath, body, 0644)
if err != nil {
fmt.Println(err)
return
}
}
// data type to send to the workers,
// last means this job is the last one
// matchid is the match id to be fetched
// a matchid of -1 means don't fetch a match
type job struct {
last bool
matchid int
}
func create_worker(jobs chan job) {
for {
next_job := <-jobs
if next_job.matchid != -1 {
match_fetch(next_job.matchid)
}
if next_job.last {
return
}
}
}
func main() {
// do the eveton match as a reference
match_fetch(614052)
var joblist [num_workers]chan job
var v int
for i := 0; i < num_workers; i++ {
job_chan := make(chan job)
joblist[i] = job_chan
go create_worker(job_chan)
}
for i := 0; i < max_id; i = i + num_workers {
for index, c := range joblist {
if i+index < max_id {
v = i + index
} else {
v = -1
}
c <- job{false, v}
}
}
for _, c := range joblist {
c <- job{true, -1}
}
}
The code seems to work in that it fills a directory called match_data with html. The problem is that this html is completely different to what I get in the browser! Here is the section which I think does this: (from the body of the GET request of http://www.whoscored.com/Matches/614052/Live.
(function() {
var z="";var b="7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E6577204461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B737769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E697469616C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F63657373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D706C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D6528292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E6577204461746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63617073756C615F5265736F757263653F535748414E45444C3D313536343032333530343538313538333938362C31373139363833393832313930303534313833392C31333935303737313737393531363432383234342C3132363636222C66616C7365293B7868722E73656E64286E756C6C297D63617463682863297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')'));})();
The reason I think this is the case is that the javascript in the page fetches and edits the DOM to what I see on view source. How can I get golang to run the javascript? Is there are library to do this? Better still, could I directly grab the JSON from the servers?
This can be done with https://godoc.org/github.com/sourcegraph/webloop#View.EvaluateJavaScript
Read their main example https://github.com/sourcegraph/webloop
What you need is a "headless browser" in general.
In general it is better to use an Web API vs. scraping. For example, whoscored themselves use OPTA which you should be able to access directly.
http://www.jokecamp.com/blog/guide-to-football-and-soccer-data-and-apis/#opta

Resources