I have a configuration file in YAML format. I am trying to read the configuration in some custom format. I couldn't guess any pattern that I can go for like tree, json etc.
Eg. application.yaml
organization:
products:
product1:
manager: "Rob"
engineer: "John"
product2:
manager: "Henry"
lead: "patrick"
The configuration file can have huge information and that can vary from file to file. I want to construct data in the following format,
organization/products/product1/manager = Rob
organization/products/product1/engineer = John
organization/products/product2/lead = patrick
OR
{"organization/products/product1/manager":"Rob","organization/products/product2/lead":"patrick"}
Any idea how I can achieve this pattern?
This is essentially an exercise in printing trees. The exact implementation will depend on the particular YAML parser you pick, but pretty much all of them will have some kind of "map of anything" type. In the very popular gopkg.in/yaml.v2 this type is named MapSlice (don't let the name confuse you; it leaks its implementation which has to deal with flexible key types).
Just throw it at your favorite tree traversal algorithm to render the text file. Here is a simple example that works with only string keys and only some scalar leaf nodes:
package main
import (
"bytes"
"fmt"
"io"
"log"
"path/filepath"
)
func main() {
var tree yaml.MapSlice
if err := yaml.Unmarshal(input, &tree); err != nil {
log.Fatal(err)
}
var buf bytes.Buffer
if err := render(&buf, tree, ""); err != nil {
log.Fatal(err)
}
}
func render(w io.Writer, tree yaml.MapSlice, prefix string) error {
for _, branch := range tree {
key, ok := branch.Key.(string)
if !ok {
return fmt.Errorf("unsupported key type: %T", branch.Key)
}
prefix := filepath.Join(prefix, key)
switch x := branch.Value.(type) {
default:
return fmt.Errorf("unsupported value type: %T", branch.Value)
case yaml.MapSlice:
// recurse
if err := render(w, x, prefix); err != nil {
return err
}
continue
// scalar values
case string:
case int:
case float64:
// ...
}
// print scalar
if _, err := fmt.Fprintf(w, "%s = %v\n", prefix, branch.Value); err != nil {
return err
}
}
return nil
}
Related
I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.
How do I write the code below to get a string from my nested yaml struct?
Here is my yaml:
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
Here is example code
viper.SetConfigName("main_config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.GetString("element.one.url")
My problem:
I get a blank string when I print this. According to the docs, this is how you get a nested element. I suspect its not working because the elements are lists. Do I need to do a struct? I am not sure how to make one, especially if it needs to be nested.
You can unmarshal a nested configuration file.
main.go
package main
import (
"fmt"
"github.com/spf13/viper"
)
type NestedURL struct {
URL string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
type ZipCode struct {
Zipcode string `mapstructure:"zipcode"`
}
type Config struct {
Element [] map[string]NestedURL `mapstructure:"element"`
Weather [] map[string]ZipCode `mapstructure:"weather"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
var config Config
if err := viper.Unmarshal(&config); err != nil {
fmt.Println(err)
return
}
fmt.Println(config)
}
config.yml
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
There are different Get methods available in viper library and your YML structure is of type []map[string]string, so to parse your YML configuration file you have to use viper.Get method. So you have to parse your file something like this..
Note: You can use struct as well to un-marshal the data. Please refer this post mapping-nested-config-yaml-to-struct
package main
import (
"fmt"
"github.com/spf13/viper"
)
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.Get("element")
fmt.Println(testvar)
elementsMap := testvar.([]interface{})
for k, vmap := range elementsMap {
fmt.Print("Key: ", k)
fmt.Println(" Value: ", vmap)
eachElementsMap := vmap.(map[interface{}]interface{})
for k, vEachValMap := range eachElementsMap {
fmt.Printf("%v: %v \n", k, vEachValMap)
vEachValDataMap := vEachValMap.(map[interface{}]interface{})
for k, v := range vEachValDataMap {
fmt.Printf("%v: %v \n", k, v)
}
}
}
}
// Output:
/*
Key: 0 Value: map[one:map[url:http://test nested:123]]
one: map[url:http://test nested:123]
url: http://test
nested: 123
Key: 1 Value: map[two:map[url:http://test nested:123]]
two: map[url:http://test nested:123]
url: http://test
nested: 123
*/
You can use Unmarshal or UnmarshalKey to parse all or part of your data and fill a struct. It is very similar to unmarshaling a json.
In your case, code will be like this:
package main
import (
"fmt"
"github.com/spf13/viper"
)
// here we define schema of data, just like what we might do when we parse json
type Element struct {
Url string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
// data in `element` key is a map of string to Element. We define a variable to store data into it.
elementParsed := make(map[string]*Element)
// read the key `element` in the yaml file, and parse it's data and put it in `elementParsed` variable
err = viper.UnmarshalKey("element", &elementParsed)
if err != nil {
panic(err)
}
fmt.Println(elementParsed["one"].Url) // will print: http://test
fmt.Println(elementParsed["one"].Nested) // will print: 123
}
I am trying to use WordNik () to get random words for the dictionary for this script: https://github.com/jmagrippis/password
WordNik is outputting:
[{"id":7936915,"word":"Tanganyikan"},{"id":27180,"word":"cartographic"},{"id":48094,"word":"deterministic"},{"id":1485119,"word":"higher-risk"},{"id":120986,"word":"juristic"},{"id":1830806,"word":"magnetorheological"},{"id":320495,"word":"quelled"},{"id":324610,"word":"remoter"},{"id":215158,"word":"telemetric"},{"id":225207,"word":"uninquisitive"}]
Here is my code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/jmagrippis/password"
)
type Words struct {
id []int64
word []string
}
type GetWordsResponse struct {
WordList []Words
}
func getWords(speech string) (*GetWordsResponse, error) {
url := fmt.Sprintf("http://api.wordnik.com/v4/words.json/randomWords?hasDictionaryDef=false&includePartOfSpeech=%s&minCorpusCount=0&maxCorpusCount=-1&minDictionaryCount=1&maxDictionaryCount=-1&minLength=5&maxLength=-1&limit=10&api_key=api_key", speech)
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
var s = new(GetWordsResponse)
var arr []string
_ = json.Unmarshal([]byte(body), &arr)
log.Printf("Unmarshaled: %v", arr)
return s, err
}
func main() {
dictionary := &password.Dictionary{
Adjectives: []string{"beautiful", "homely", "magical", "posh", "excellent"},
Subjects: []string{"mermaids", "unicorns", "lions", "piranhas"},
Verbs: []string{"love", "fancy", "eat", "bring", "fear", "aggravate"},
Adverbs: []string{"cuddling", "slapping", "shouting", "jumping"},
Objects: []string{"teddy-bears", "diamonds", "buckets", "boxes"},
}
generator := password.NewGenerator(dictionary, time.Now().UnixNano())
pass := generator.Generate()
fmt.Printf("%s", pass)
getWords("Verb")
}
As you can see, what I am trying to do is use WordNik API to request Adverbs, nouns, etc and then make a dictionary based off those words to generate a password. I am horrible with arrays and handling data.
As pointed out in the comments you need to export the fields so they can be unmarshaled
The encoding/json package relies on reflection and since it is in another package, it cannot access unexported fields. ( in go, fields, methods or functions that start with a small letter are unexported whereas with capital letters they are exported )
Then your example json does not contain the WordList at all, so what you want is unmarshaling directly into the array of Words. Also a words object only consists of Id and Word and not of arrays themselves.
type Words struct {
Id int64
Word string
}
func main() {
....
var words []Words
// you don't need to read the whole body first, you can decode in the same turn like this
err := json.NewDecoder(req.Body).Decode(&words)
if nil != err {
log.Fatal(err)
}
...
}
Another very important thing is that you should not ignore errors. This would have helped you to debug the issue. ( What I mean is _ = json.Unmarshal)
As for beginning with go, you could implement a simple test to see if your code works as intended.
https://play.golang.org/p/nuz9uXdka5S < check this working example for reference.
How to parse xml in such silly format:
<key>KEY1</key><string>VALUE OF KEY1</string>
<key>KEY2</key><string>VALUE OF KEY2</string>
<key>KEY3</key><integer>42</integer>
<key>KEY3</key><array>
<integer>1</integer>
<integer>2</integer>
</array>
Parsing would be very simple if all values would have same type - for example strings. But in my case each value could be string, data, integer, boolean, array or dict.
This xml looks nearly like json, but unfortunately format is fixed, and I cannot change it. And I would prefer solution without any external packages.
Use a lower-level parsing interface provided by encoding/xml which allows you to iterate over individual tokens in the XML stream (such as "start element", "end element" etc).
See the Token() method of the encoding/xml's Decoder type.
Since the data is not well structured, and you can't modify the format, you can't use xml.Unmarshal, so you can process the XML elements by creating a new Decoder, then iterate over the tokens and use DecodeElement to process them one by one. In my sample code below, it puts everything in a map. The code is also on github here...
package main
import (
"encoding/xml"
"strings"
"fmt"
)
type PlistArray struct {
Integer []int `xml:"integer"`
}
const in = "<key>KEY1</key><string>VALUE OF KEY1</string><key>KEY2</key><string>VALUE OF KEY2</string><key>KEY3</key><integer>42</integer><key>KEY3</key><array><integer>1</integer><integer>2</integer></array>"
func main() {
result := map[string]interface{}{}
dec := xml.NewDecoder(strings.NewReader(in))
dec.Strict = false
var workingKey string
for {
token, _ := dec.Token()
if token == nil {
break
}
switch start := token.(type) {
case xml.StartElement:
fmt.Printf("startElement = %+v\n", start)
switch start.Name.Local {
case "key":
var k string
err := dec.DecodeElement(&k, &start)
if err != nil {
fmt.Println(err.Error())
}
workingKey = k
case "string":
var s string
err := dec.DecodeElement(&s, &start)
if err != nil {
fmt.Println(err.Error())
}
result[workingKey] = s
workingKey = ""
case "integer":
var i int
err := dec.DecodeElement(&i, &start)
if err != nil {
fmt.Println(err.Error())
}
result[workingKey] = i
workingKey = ""
case "array":
var ai PlistArray
err := dec.DecodeElement(&ai, &start)
if err != nil {
fmt.Println(err.Error())
}
result[workingKey] = ai
workingKey = ""
default:
fmt.Errorf("Unrecognized token")
}
}
}
fmt.Printf("%+v", result)
}
package main
import (
"bytes"
"code.google.com/p/go.net/html"
"fmt"
"log"
"strings"
)
func main() {
s := "Blah. <b>Blah.</b> Blah."
n, err := html.Parse(strings.NewReader(s))
if err != nil {
log.Fatalf("Parse error: %s", err)
}
var buf bytes.Buffer
if err := html.Render(&buf, n); err != nil {
log.Fatalf("Render error: %s", err)
}
fmt.Println(buf.String())
}
Output:
<html><head></head><body>Blah. <b>Blah.</b> Blah.</body></html>
Is there a way to stop html.Parse from making a document out of fragments (ie avoid adding <html>, <body> etc.)? I'm aware of html.ParseFragment but it seems to exhibit the same behaviour.
You can get around it by wrapping the text to be parsed with a parent element such as <span> then doing something like the following:
n = n.FirstChild.LastChild.FirstChild
but that seems, well, kludgy to say the least.
Ideally I'd like to: accept input, manipulate or remove nodes found within it, and write the result back to a string, even if the result is an incomplete document.
You need to provide a context to ParseFragment. The following program prints out the original text:
package main
import (
"bytes"
"code.google.com/p/go.net/html"
"code.google.com/p/go.net/html/atom"
"fmt"
"log"
"strings"
)
func main() {
s := "Blah. <b>Blah.</b> Blah."
n, err := html.ParseFragment(strings.NewReader(s), &html.Node{
Type: html.ElementNode,
Data: "body",
DataAtom: atom.Body,
})
if err != nil {
log.Fatalf("Parse error: %s", err)
}
var buf bytes.Buffer
for _, node := range n {
if err := html.Render(&buf, node); err != nil {
log.Fatalf("Render error: %s", err)
}
}
fmt.Println(buf.String())
}
You want http://godoc.org/code.google.com/p/go.net/html#ParseFragment. Pass in a fake Body element as your context and the fragment will be returned as a slice of just the elements in your fragment.
You can see an example in the Partial* functions for go-html-transform's go.net/html wrapper package. https://code.google.com/p/go-html-transform/source/browse/h5/h5.go#32