How to parse yaml and get value for interface- deep structue - go

I'm trying to parse this yaml and I want to get the values of the run entry (test1 or test2) without success, here is my working example.
im a bit get lost with the map inside map :( ,
this is given yaml which I couldent change ...
any idea how could I got those values
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
version: "3.2"
run-parameters:
before:
run-parameters:
run: test1
after:
run-parameters:
run: test2
`)
type FTD struct {
Version string `yaml:"version,omitempty"`
BuildParams *RunParams `yaml:"run-parameters,omitempty"`
}
type RunParams struct {
BeforeExec map[string]interface{} `yaml:"before,omitempty"`
AfterExec map[string]interface{} `yaml:"after,omitempty"`
}
func main() {
runners := &FTD{}
// parse mta yaml
err := yaml.Unmarshal(runContent, runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
for k, v := range runners.BuildParams.BeforeExec {
fmt.Println(k, v.(interface{}))
}
}
This is working example
https://play.golang.org/p/qTqUJy3md0c
also I've tried with
this is working
run := runners.BuildParams.BeforeExec["run-parameters"].(map[interface{}]interface{})["run"]
fmt.Println("run: ", run)
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?

what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
You can do
runParams, ok := runners.BuildParams.BeforeExec["run-parameters"]
if !ok {
// handle lack of "run-parameters" in BeforeExec
}
runParamsMap, ok := runParams.(map[interface{}]interface{})
if !ok {
// handle "run-parameters" not being a map
}
run, ok := runParamsMap["run"]
if !ok {
// handle lack of "run" inside "run-parameters"
}
runStr, ok := run.(string)
if !ok {
// handle "run" not being a string
}
fmt.Println("run: ", runStr)
This is quite verbose so you could use something like https://github.com/jmoiron/jsonq, where you can specify a "path" to the desired value nested inside several levels of maps. Despite the "json" in the name, this library works with map[string]interface{} and not json files. But note that the library you use for yaml unmarshalling results in map[interface{}]interface{} instead of map[string]interface{} and you will have to use a different one in order for it to work with jsonq.
run, err := jsonq.NewQuery(runners.BuildParams.BeforeExec).String("run-parameters", "run")
if err != nil {
// handle all possible errors in one place
}
fmt.Println("run: ", run)

Related

Go-github not retrieving tag names correctly

I have the following simple golang code which retrieves tags from terraform repository:
import (
"github.com/google/go-github/v48/github"
"context"
)
func main() {
client := github.NewClient(nil)
tags, _, _ := client.Repositories.ListTags(context.Background(), "hashicorp", "terraform", nil)
if len(tags) > 0 {
latestTag := tags[0]
fmt.Println(latestTag.Name)
} else {
fmt.Printf("No tags yet")
}
}
Which returns a strange hexadecimal value:
0x1400035c4a0
And I would want to return:
v1.4.0-alpha20221207
Following the official docs, the function ListTags should return the name encoded into a struct:
https://pkg.go.dev/github.com/google/go-github/github#RepositoriesService.ListTags
Many thanks
I did try to execute a simple GET request https://api.github.com/repos/hashicorp/terraform/tags and I can see that the github api returns the tags correctly
IDK why, but I realize the latestTag.Name is a pointer and what you're printing is the address of the memory: 0x1400035c4a0.
You just need to dereference it:
fmt.Println(*latestTag.Name)
Bonus, check error with if condition that is returned by the function call to avoid having to go something like this:
tags, response, err := client.Repositories.ListTags(context.Background(), "hashicorp", "terraform", nil)
fmt.Println(response)
if err != nil {
fmt.Println(err)
} else {
if len(tags) > 0 {
latestTag := tags[0]
fmt.Println(*latestTag.Name)
} else {
fmt.Printf("No tags yet")
}
}

Is there a better way where I can check if a template property was not resolved?

I am trying to build a string using text/template, where the template string could have arbitrary properties that are resolved via a map.
What I am trying to accomplish is identifying where one/any of the template properties is not resolved and return an error.
At the moment, I am using regexp but reaching out to the community of see if there was a better solution.
package main
import (
"bytes"
"fmt"
"regexp"
"text/template"
)
func main() {
data := "teststring/{{.someData}}/{{.notExist}}/{{.another}}"
// the issue here is that data can be arbitrary so i cannot do
// a lot of unknown if statements
t := template.Must(template.New("").Parse(data))
var b bytes.Buffer
fillers := map[string]interface{}{
"someData": "123",
"another": true,
// in this case, notExist is not defined, so the template will
// note resolve it
}
if err := t.Execute(&b, fillers); err != nil {
panic(err)
}
fmt.Println(b.String())
// teststring/123/<no value>/true
// here i am trying to catch if a the template required a value that was not provided
hasResolved := regexp.MustCompile(`<no value>`)
fmt.Println(hasResolved.MatchString(b.String()))
// add notExist to the fillers map
fillers["notExist"] = "testdata"
b.Reset()
if err := t.Execute(&b, fillers); err != nil {
panic(err)
}
fmt.Println(b.String())
fmt.Println(hasResolved.MatchString(b.String()))
// Output:
// teststring/123/<no value>/true
// true
// teststring/123/testdata/true
// false
}
You can let it fail by settings the options on the template:
func (t *Template) Option(opt ...string) *Template
"missingkey=default" or "missingkey=invalid"
The default behavior: Do nothing and continue execution.
If printed, the result of the index operation is the string
"<no value>".
"missingkey=zero"
The operation returns the zero value for the map type's element.
"missingkey=error"
Execution stops immediately with an error.
If you set it to missingkey=error, you get what what want.
t = t.Options("missingkey=error")

Parsing prometheus metrics from file and updating counters

I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.

How to implement OptionFlag with urfave/cli

Is there a way to define an OptionFlag with urfave/cli?
I'm looking for something that will look like this.
mycli --format json
mycli --format xml
I know I can use the StringFlag, but it would be great if I could have the --help show what are the valid options/values for this flag, so it is transparent for the end-user of mycli.
This way the Flag could also be validated against the options to inform the user he has provided an invalid value for this flag for example, Which ofcourse can also be done with the StringFlag, but would rather have something more sofisticated that does all of this.
I also filed an issue on the Github repository. Maybe it is a missing feature, which I would be happy to contribute with some guidance.
https://github.com/urfave/cli/issues/1154
I think you want the StringSliceFlag It allows you to define valid/default values for a flag.
package main
import (
"fmt"
"log"
"os"
"github.com/urfave/cli/v2"
)
func main() {
validMeetings := []string{"standup", "postmortem", "jourfix"}
meetings := cli.NewStringSlice(validMeetings...)
app := &cli.App{
Flags: []cli.Flag{
&cli.StringSliceFlag{
Value: meetings,
Name: "meeting",
Usage: "use one of the default values"},
},
Action: func(c *cli.Context) error {
m := c.StringSlice("meeting")
ok := false
for _, selected := range m {
for _, valid := range validMeetings {
if selected == valid {
ok = true
}
}
}
if !ok {
return fmt.Errorf("you must use one of %v", validMeetings)
}
fmt.Printf("%s\n", c.String("meeting"))
return nil
},
}
err := app.Run(os.Args)
if err != nil {
log.Fatal(err)
}
}
https://github.com/urfave/cli/blob/4f74020d9f07911f0fdb8facbc1f557a12cd2a93/app_test.go#L760

Golang - create an object of the same type as passed

I'm trying to build a generic function which will parse input (in JSON) into a specified structure. The structure may vary at run-time, based on parameters which are passed to the function. I'm currently trying to achieve this by passing an object of the right type and using reflect.New() to create a new output object of the same type.
I'm then parsing the JSON into this object, and scanning the fields.
If I create the object and specify the type in code, everything works. If I pass an object and try to create a replica, I get an "invalid indirect" error a few steps down (see code).
import (
"fmt"
"reflect"
"encoding/json"
"strings"
)
type Test struct {
FirstName *string `json:"FirstName"`
LastName *string `json:"LastName"`
}
func genericParser(incomingData *strings.Reader, inputStructure interface{}) (interface{}, error) {
//******* Use the line below and things work *******
//parsedInput := new(Test)
//******* Use vvv the line below and things don't work *******
parsedInput := reflect.New(reflect.TypeOf(inputStructure))
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(&parsedInput)
if err != nil {
//parsing error
return nil, err
}
//******* This is the line that generates the error "invalid indirect of parsedInput (type reflect.Value)" *******
contentValues := reflect.ValueOf(*parsedInput)
for i := 0; i < contentValues.NumField(); i++ {
//do stuff with each field
fmt.Printf("Field name was: %s\n", reflect.TypeOf(parsedInput).Elem().Field(i).Name)
}
return parsedInput, nil
}
func main() {
inputData := strings.NewReader("{\"FirstName\":\"John\", \"LastName\":\"Smith\"}")
exampleObject := new(Test)
processedData, err := genericParser(inputData, exampleObject)
if err != nil {
fmt.Println("Parsing error")
} else {
fmt.Printf("Success: %v", processedData)
}
}
If I can't create a replica of the object, then a way of updating / returning the one supplied would be feasible. The key thing is that this function must be completely agnostic to the different structures available.
reflect.New isn't a direct analog to new, as it can't return a specific type, it only can return a reflect.Value. This means that you are attempting to unmarshal into a *reflect.Value, which obviously isn't going to work (even if it did, your code would have passed in **Type, which isn't what you want either).
Use parsedInput.Interface() to get the underlying value after creating the new value to unmarshal into. You then don't need to reflect on the same value a second time, as that would be a reflect.Value of a reflect.Value, which again isn't going to do anything useful.
Finally, you need to use parsedInput.Interface() before you return, otherwise you are returning the reflect.Value rather than the value of the input type.
For example:
func genericParser(incomingData io.Reader, inputStructure interface{}) (interface{}, error) {
parsedInput := reflect.New(reflect.TypeOf(inputStructure).Elem())
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(parsedInput.Interface())
if err != nil {
return nil, err
}
for i := 0; i < parsedInput.Elem().NumField(); i++ {
fmt.Printf("Field name was: %s\n", parsedInput.Type().Elem().Field(i).Name)
}
return parsedInput.Interface(), nil
}
https://play.golang.org/p/CzDrj6sgQNt

Resources