Im a newbie in golang. I am trying to compare two yaml files and update the 2nd file's value if there is any new value in 1st yaml for that particular key.
So the files are of format: These are sample yaml files. Real yaml files have much more nested complicated maps with different datatypes for each key.
1st yaml:
name: john
city: washington
2nd yaml:
name: peter
city: washington
Final result for 2nd yaml file should be:
name: john
city: washington
Tried creating a map string interface for both yaml files using unmarshal. But having trouble how to compare both maps. Was trying to loop over each key of map and search for that key in 2nd yaml map. If key exists update the value in 2nd yaml map. But i am not able to implement that. Any suggestions/better ideas?
Edit: Updated code
package main
import (
"fmt"
"io/ioutil"
"log"
"github.com/imdario/mergo"
"gopkg.in/yaml.v3"
)
func main() {
yfile, err := ioutil.ReadFile("C:/Users/212764682/lifecycle/userconfig.yaml")
if err != nil {
log.Fatal(err)
}
data := make(map[string]interface{})
err2 := yaml.Unmarshal(yfile, &data)
if err2 != nil {
log.Fatal(err2)
} else {
yfile1, err3 := ioutil.ReadFile("C:/Users/212764682/lifecycle/conf.yaml")
yfile2, err4 := ioutil.ReadFile("C:/Users/212764682/lifecycle/prof.yaml")
if err3 != nil && err4 != nil {
log.Fatal(err3)
log.Fatal(err4)
} else {
dat := make(map[string]interface{})
dat2 := make(map[string]interface{})
err5 := yaml.Unmarshal(yfile1, &dat)
err6 := yaml.Unmarshal(yfile2, &dat2)
_ = err5
_ = err6
for key1, element1 := range data {
for key2, element2 := range dat {
if key1 == key2 {
if element1 == element2 {
} else {
element2 = element1
}
} else {
dat[key1] = data[key1]
}
}
}
}
}
}
So im want to compare each key of data with dat. If that key exists in dat, check for value in data. If value different in dat, update with value of data in dat for that key. Also, if any key of data dosent exist in dat, then append that key in dat. But not able to implement it correctly.
You can try to compare the map and then update it if the key exists. Here's some example using your case.
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
func main() {
// read file
yfile1, err := ioutil.ReadFile("file1.yaml")
if err != nil {
log.Fatal(err)
return
}
yfile2, err := ioutil.ReadFile("file2.yaml")
if err != nil {
log.Fatal(err)
return
}
// unmarshal ymal
data1 := make(map[string]interface{})
if err := yaml.Unmarshal(yfile1, &data1); err != nil {
log.Fatal(err)
return
}
data2 := make(map[string]interface{})
if err := yaml.Unmarshal(yfile2, &data2); err != nil {
log.Fatal(err)
return
}
// from this we can iterate the key in data2 then check whether it exists in data1
// if so then we can update the value in data2
// iterate key in data2
for key2 := range data2 {
// check whether key2 exists in data1
if val1, ok := data1[key2]; ok {
// update the value of key2 in data2
data2[key2] = val1
}
}
fmt.Printf("data2: %v", data2)
// output:
// data2: map[city:washington name:john]
// you can write the data2 into ymal
newYfile2, err := yaml.Marshal(&data2)
if err != nil {
log.Fatal(err)
return
}
// write to file
if err = ioutil.WriteFile("new_file2.yaml", newYfile2, 0644); err != nil {
log.Fatal(err)
return
}
}
Inside new_file2.yaml will be like this:
city: washington
name: john
One thing that you need to take a not is that map in Go doesn't maintain the order (AFAIK Go doesn't have built-in OrderedMap type per 7 May 2022) so the order key in the new file will be random
Additional note: for error handling, you better handle it right away (right after you got the error). Here's a good article about it Error handling and Go
There is a maps package, since 1.18 I believe. If you don't care about new keys being added to the destination map you can use its copy function.
func Copy[M ~map[K]V, K comparable, V any](dst, src M)
Copy copies all key/value pairs in src adding them to dst. When a key in src is already present in dst, the value in dst will be overwritten by the value associated with the key in src.
The source code of that function is very simple:
func Copy[M ~map[K]V, K comparable, V any](dst, src M) {
for k, v := range src {
dst[k] = v
}
}
You could also do it yourself. The below code is from the helm source code. Unlike the copy function above, it works recursivly:
func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
out := make(map[string]interface{}, len(a))
for k, v := range a {
out[k] = v
}
for k, v := range b {
if v, ok := v.(map[string]interface{}); ok {
if bv, ok := out[k]; ok {
if bv, ok := bv.(map[string]interface{}); ok {
out[k] = mergeMaps(bv, v)
continue
}
}
}
out[k] = v
}
return out
}
Related
I use the goldmark-meta package to read a YAML file.
The contents I'm interested in look like this
in the YAML:
burger:
- a: ay
- b: bee
- c: see
I want to get access to both keys and values
of the returned interface, and I'm stuck.
Iterating through the return gives me a
list of the key/value pairs, but I don't
know how to obtain any info when I don't
know the key names in advance.
This program prints out the following:
func (c *config) burger() string {
// c.pageFm is type map[string]interface{}
b, ok := c.pageFm["burger"].([]interface{})
if !ok {
// No burger entry in yaml
return ""
}
debug("Burger list has %v items:\n%v", len(b), b)
debug("Type: %v", reflect.TypeOf(b))
for i := 0; i < len(b); i++ {
debug("%v", b[i])
}
return ""
}
Burger list has 3 items:
[map[a:ay] map[b:bee] map[c:see]]
Type: []interface {}
map[a:ay]
map[b:bee]
map[c:see]
How do I obtain the key and value strings?
In your YAML data, you have a key (burger) the value of which is a list of maps (and each map has a single key). We can iterate over the items in burger like this:
b, ok := c.pageFm["burger"].([]interface{})
if !ok {
return ""
}
for _, item := range burger {
...
}
For each item, we can iterate over available keys and values:
for _, item := range burger {
for k, v := range item.(map[interface{}]interface{}) {
...
}
}
We can convert keys and values from interface{} into string using fmt.Sprintf:
for _, item := range burger {
for k, v := range item.(map[interface{}]interface{}) {
k_str := fmt.Sprintf("%v", k)
v_str := fmt.Sprintf("%v", v)
fmt.Printf("key %s value %s\n", k_str, v_str)
}
}
Starting with the sample code from goldmark-meta, I put together this example:
package main
import (
"bytes"
"fmt"
"github.com/yuin/goldmark"
meta "github.com/yuin/goldmark-meta"
"github.com/yuin/goldmark/parser"
)
func main() {
markdown := goldmark.New(
goldmark.WithExtensions(
meta.Meta,
),
)
source := `---
burger:
- a: ay
- b: bee
- c: see
---
# Hello goldmark-meta
`
var buf bytes.Buffer
context := parser.NewContext()
if err := markdown.Convert([]byte(source), &buf, parser.WithContext(context)); err != nil {
panic(err)
}
metaData := meta.Get(context)
burger := metaData["burger"].([]interface{})
for _, item := range burger {
for k, v := range item.(map[interface{}]interface{}) {
k_str := fmt.Sprintf("%v", k)
v_str := fmt.Sprintf("%v", v)
fmt.Printf("key %s value %s\n", k_str, v_str)
}
}
}
Which outputs:
key a value ay
key b value bee
key c value see
What I am trying to do?
For every change in the database, I am trying to convert the debezium event into a CSV of the database values for loading into Redshift.
For the below 110 change, i am trying to make a csv file: 110,vck,desc,221.1
mysql> select * from products;
+-----+-------------+---------------------------------------------------------+--------+
| id | name | description | weight |
+-----+-------------+---------------------------------------------------------+--------+
| 110 | vck | desc | 221.1 |
+-----+-------------+---------------------------------------------------------+--------+
Here was my attempt to do it using goavro.
Program
https://play.golang.org/p/A8Wd0sZPUEQ
package main
import (
"fmt"
"encoding/json"
)
func main() {
debeziumEvent := `{"before":null,"after":{"datapipe.inventory.products.Value":{"id":110,"name":"vck","description":{"string":"desc"},"weight":{"double":221.10000610351562}}},"source":{"query":null,"snapshot":{"string":"true"},"server_id":0,"gtid":null,"name":"datapipe","thread":null,"ts_ms":0,"file":"mysql-bin.000049","version":"1.2.1.Final","connector":"mysql","pos":154,"table":{"string":"products"},"row":0,"db":"inventory"},"op":"c","ts_ms":{"long":1597649700266},"transaction":null}`
var data map[string]interface{}
err := json.Unmarshal([]byte(debeziumEvent), &data)
if err != nil {
panic(err)
}
after := data["after"].(map[string]interface{})
csv := make([]interface{}, 0)
for _, v := range after {
for _, v2 := range v.(map[string]interface{}) {
switch stype := v2.(type) {
case map[string]interface{}:
for _, v3 := range v2.(map[string]interface{}) {
csv = append(csv, v3)
}
case string:
csv = append(csv, v2)
case int:
csv = append(csv, v2)
case float64:
csv = append(csv, v2)
default:
fmt.Printf("type %s not handled\n", stype)
panic("unhandled type")
}
}
}
fmt.Println(csv)
}
Is there a way to do this any better? For every data type i would need to have a switch statement here....
Linked GoAVRO issue: https://github.com/linkedin/goavro/issues/217
fmt.Sprintf can be used to convert interfaces to string.
str := fmt.Sprintf("%v", v)
Doing this reduces the case statements to 2:
package main
import (
"fmt"
"encoding/json"
)
func main() {
debeziumEvent := `{"before":null,"after":{"datapipe.inventory.products.Value":{"id":110,"name":"vck","description":{"string":"desc"},"weight":{"double":221.10000610351562}}},"source":{"query":null,"snapshot":{"string":"true"},"server_id":0,"gtid":null,"name":"datapipe","thread":null,"ts_ms":0,"file":"mysql-bin.000049","version":"1.2.1.Final","connector":"mysql","pos":154,"table":{"string":"products"},"row":0,"db":"inventory"},"op":"c","ts_ms":{"long":1597649700266},"transaction":null}`
var data map[string]interface{}
err := json.Unmarshal([]byte(debeziumEvent), &data)
if err != nil {
panic(err)
}
//fmt.Printf("data=%v\n", data)
after := data["after"].(map[string]interface{})
csv := []string{}
for _, v := range after {
for _, v2 := range v.(map[string]interface{}) {
switch v2.(type) {
case map[string]interface{}:
for _, v3 := range v2.(map[string]interface{}) {
csv = append(csv, fmt.Sprintf("%v", v3))
}
default:
csv = append(csv, fmt.Sprintf("%v", v2))
}
}
}
fmt.Println(csv)
}
I am rather new to working with yaml and golang. Currently, I am creating a golang program that parses an rpm package to check for subsystem dependencies. It extends the go-rpmutils library.
So far this is the code I have within my main function to handle conditions:
func main() {
// Parse the rpm
rpm, err := rpmutils.ReadRpm("file.rpm")
if err != nil {
panic(err)
}
// Get RPM Deps
dependencies, err := rpm.Header.GetStrings(rpmutils.REQUIRENAME)
if err != nil {
panic(err)
}
// Check for specific dep condition
for _, p := range dependencies {
if strings.HasPrefix(p, "prefix1") && p != "string-including-prefix1" {
fmt.Printf("\t%s\n", p)
defer os.Exit(1)
}
}
}
I am able to output the dependencies but want to set up several if else conditions for when specific subsystem dependencies exist.
In a separate yaml file, I have:
allowed-deps:
-dep1
-dep2
-dep3
third-party-deps:
-dep4
-dep5
-dep6
internal-deps:
-dep7
-dep8
-dep9
I'd like to compare the value of var p from the for loop with the values in the yaml file. So for example:
if p only equals values from allowed-deps, print "successfully built rpm" and do not prompt os.Exit(1)
if p equals any of the third-party-deps, print "err msg for third-party deps" and os.Exit(1)
if p equals any internal-deps, print "another err mssg" and os.Exit(1)
How can I go about doing this?
You can use a YAML package (like https://github.com/go-yaml/yaml), load your file into a variable and check it on every step in the ifs that you propose. I would use maps as it seems that you will be checking very frequently the sets.
Here you have a simple example that I made using that package so you can see how to unmarshal your file, convert into maps, and check the maps: https://play.golang.org/p/t1GhUPvAQNQ
package main
import (
"fmt"
"github.com/go-yaml/yaml"
)
const str = `
allowed-deps:
- dep1
- dep2
- dep3
third-party-deps:
- dep4
- dep5
- dep6
internal-deps:
- dep7
- dep8
- dep9
`
type MyYAML struct {
AllowedDeps []string `yaml:"allowed-deps"`
ThirdPartyDeps []string `yaml:"third-party-deps"`
InternalDeps []string `yaml:"internal-deps"`
}
func main() {
var a MyYAML
err := yaml.Unmarshal([]byte(str), &a)
if err != nil {
panic(err)
}
// Build a map for every section.
allowedDeps := map[string]struct{}{}
thirdPartyDeps := map[string]struct{}{}
internalDeps := map[string]struct{}{}
for _, dep := range a.AllowedDeps {
allowedDeps[dep] = struct{}{}
}
for _, dep := range a.ThirdPartyDeps {
thirdPartyDeps[dep] = struct{}{}
}
for _, dep := range a.InternalDeps {
internalDeps[dep] = struct{}{}
}
// Some checking examples.
if _, ok := allowedDeps["dep1"]; ok {
fmt.Println("dep1 found")
}
if _, ok := thirdPartyDeps["dep1"]; ok {
fmt.Println("dep1 found")
}
if _, ok := internalDeps["dep8"]; ok {
fmt.Println("dep8 found")
}
}
I'm trying to write a func to get all paths to values from a yaml file and I don't know know how that possible, here is my code:
func getpath(fileyaml) string {
if _, err := os.Stat(fileyaml); err == nil {
bfile, err := ioutil.ReadFile(fileyaml)
bjson, err := yaml.YAMLToJSON(bfile)
if err != nil {
fmt.Printf("YAMLToJSON err: %v\n", err)
}
json := string(bjson)
println json
paths := ? // don't know
return path
here my yaml file :
sentinel:
number: 3
server:
number: 7
config:
fere_size: 5
lcmea:
eza_ze: all
my function will convert it to a json: {"config":{"fere_size":5},"lcmea":{"eza_hooks":"all"},"sentinel":{"number":3},"server":{"number":7}}
the output that i want :
sentinel.number=3, server.number=3,config.fere_size=5,lcmea.eza_ze=all
how to parse this json in order to get this desired output?
I'm using "github.com/tidwall/gjson" to read the yaml and convert it to json
This is an example of how you could do it: https://play.golang.org/p/7yLq_PDLdXF
It is pretty naive and definitely could be improved, but it may give you an idea about how to parse the yaml file and then print the output in your desired format:
package main
import (
"fmt"
"log"
"strings"
"github.com/go-yaml/yaml"
)
var data = `
sentinel:
number: 3
server:
number: 7
config:
fere_size: 5
lcmea:
eza_ze: all
`
func main() {
m := make(map[string]map[string]interface{})
err := yaml.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatalf("error: %v", err)
}
out := []string{}
for k, v := range m {
for j, i := range v {
out = append(out, fmt.Sprintf("%s.%v=%v", k, j, i))
}
}
fmt.Println(strings.Join(out, ", "))
}
It will return:
$ go run main.go
sentinel.number=3, server.number=7, config.fere_size=5, lcmea.eza_ze=all
It is using https://github.com/go-yaml/yaml, check more examples on the README.md
I am trying to use maps and slice of those maps to store rows returned from a database query. But what I get in every iteration of the rows.Next() and in final is the slice of the one same row from the query. It seems the problem is related to the memory place being same I store the cols, yet I could not resolve it until now.
What is the thing am I missing here:
The source code is as follows:
package main
import (
"database/sql"
_ "github.com/lib/pq"
"fmt"
"log"
"reflect"
)
var myMap = make(map[string]interface{})
var mySlice = make([]map[string]interface{}, 0)
func main(){
fmt.Println("this is my go playground.")
// DB Connection-
db, err := sql.Open("postgres", "user=postgres dbname=proj2-dbcruddb-dev password=12345 sslmode=disable")
if err != nil {
log.Fatalln(err)
}
rows, err := db.Query("SELECT id, username, password FROM userstable")
defer rows.Close()
if err != nil {
log.Fatal(err)
}
colNames, err := rows.Columns()
if err != nil {
log.Fatal(err)
}
cols := make([]interface{}, len(colNames))
colPtrs := make([]interface{}, len(colNames))
for i := 0; i < len(colNames); i++ {
colPtrs[i] = &cols[i]
}
for rows.Next() {
err = rows.Scan(colPtrs...)
if err != nil {
log.Fatal(err)
}
fmt.Println("cols: ", cols)
for i, col := range cols {
myMap[colNames[i]] = col
}
mySlice = append(mySlice, myMap)
// Do something with the map
for key, val := range myMap {
fmt.Println("Key:", key, "Value:", val, "Value Type:", reflect.TypeOf(val))
}
fmt.Println("myMap: ", myMap)
fmt.Println("mySlice: ", mySlice)
}
fmt.Println(mySlice)
}
This is because what you are storing in the slice is a pointer to a map rather than a copy of the map.
From Go maps in action:
Map types are reference types, like pointers or slices...
Since you create the map outside the loop that updates it, you keep overwriting the data in the map with new data and are appending a pointer to the same map to the slice each time. Thus you get multiple copies of the same thing in your slice (being the last record read from your table).
To handle, move var myMap = make(map[string]interface{}) into the for rows.Next() loop so a new map is create on each iteration and then appended to the slice.