Checking if string from golang file exists in yaml file - go

I am rather new to working with yaml and golang. Currently, I am creating a golang program that parses an rpm package to check for subsystem dependencies. It extends the go-rpmutils library.
So far this is the code I have within my main function to handle conditions:
func main() {
// Parse the rpm
rpm, err := rpmutils.ReadRpm("file.rpm")
if err != nil {
panic(err)
}
// Get RPM Deps
dependencies, err := rpm.Header.GetStrings(rpmutils.REQUIRENAME)
if err != nil {
panic(err)
}
// Check for specific dep condition
for _, p := range dependencies {
if strings.HasPrefix(p, "prefix1") && p != "string-including-prefix1" {
fmt.Printf("\t%s\n", p)
defer os.Exit(1)
}
}
}
I am able to output the dependencies but want to set up several if else conditions for when specific subsystem dependencies exist.
In a separate yaml file, I have:
allowed-deps:
-dep1
-dep2
-dep3
third-party-deps:
-dep4
-dep5
-dep6
internal-deps:
-dep7
-dep8
-dep9
I'd like to compare the value of var p from the for loop with the values in the yaml file. So for example:
if p only equals values from allowed-deps, print "successfully built rpm" and do not prompt os.Exit(1)
if p equals any of the third-party-deps, print "err msg for third-party deps" and os.Exit(1)
if p equals any internal-deps, print "another err mssg" and os.Exit(1)
How can I go about doing this?

You can use a YAML package (like https://github.com/go-yaml/yaml), load your file into a variable and check it on every step in the ifs that you propose. I would use maps as it seems that you will be checking very frequently the sets.
Here you have a simple example that I made using that package so you can see how to unmarshal your file, convert into maps, and check the maps: https://play.golang.org/p/t1GhUPvAQNQ
package main
import (
"fmt"
"github.com/go-yaml/yaml"
)
const str = `
allowed-deps:
- dep1
- dep2
- dep3
third-party-deps:
- dep4
- dep5
- dep6
internal-deps:
- dep7
- dep8
- dep9
`
type MyYAML struct {
AllowedDeps []string `yaml:"allowed-deps"`
ThirdPartyDeps []string `yaml:"third-party-deps"`
InternalDeps []string `yaml:"internal-deps"`
}
func main() {
var a MyYAML
err := yaml.Unmarshal([]byte(str), &a)
if err != nil {
panic(err)
}
// Build a map for every section.
allowedDeps := map[string]struct{}{}
thirdPartyDeps := map[string]struct{}{}
internalDeps := map[string]struct{}{}
for _, dep := range a.AllowedDeps {
allowedDeps[dep] = struct{}{}
}
for _, dep := range a.ThirdPartyDeps {
thirdPartyDeps[dep] = struct{}{}
}
for _, dep := range a.InternalDeps {
internalDeps[dep] = struct{}{}
}
// Some checking examples.
if _, ok := allowedDeps["dep1"]; ok {
fmt.Println("dep1 found")
}
if _, ok := thirdPartyDeps["dep1"]; ok {
fmt.Println("dep1 found")
}
if _, ok := internalDeps["dep8"]; ok {
fmt.Println("dep8 found")
}
}

Related

Golang comparing two yaml files and updating them

Im a newbie in golang. I am trying to compare two yaml files and update the 2nd file's value if there is any new value in 1st yaml for that particular key.
So the files are of format: These are sample yaml files. Real yaml files have much more nested complicated maps with different datatypes for each key.
1st yaml:
name: john
city: washington
2nd yaml:
name: peter
city: washington
Final result for 2nd yaml file should be:
name: john
city: washington
Tried creating a map string interface for both yaml files using unmarshal. But having trouble how to compare both maps. Was trying to loop over each key of map and search for that key in 2nd yaml map. If key exists update the value in 2nd yaml map. But i am not able to implement that. Any suggestions/better ideas?
Edit: Updated code
package main
import (
"fmt"
"io/ioutil"
"log"
"github.com/imdario/mergo"
"gopkg.in/yaml.v3"
)
func main() {
yfile, err := ioutil.ReadFile("C:/Users/212764682/lifecycle/userconfig.yaml")
if err != nil {
log.Fatal(err)
}
data := make(map[string]interface{})
err2 := yaml.Unmarshal(yfile, &data)
if err2 != nil {
log.Fatal(err2)
} else {
yfile1, err3 := ioutil.ReadFile("C:/Users/212764682/lifecycle/conf.yaml")
yfile2, err4 := ioutil.ReadFile("C:/Users/212764682/lifecycle/prof.yaml")
if err3 != nil && err4 != nil {
log.Fatal(err3)
log.Fatal(err4)
} else {
dat := make(map[string]interface{})
dat2 := make(map[string]interface{})
err5 := yaml.Unmarshal(yfile1, &dat)
err6 := yaml.Unmarshal(yfile2, &dat2)
_ = err5
_ = err6
for key1, element1 := range data {
for key2, element2 := range dat {
if key1 == key2 {
if element1 == element2 {
} else {
element2 = element1
}
} else {
dat[key1] = data[key1]
}
}
}
}
}
}
So im want to compare each key of data with dat. If that key exists in dat, check for value in data. If value different in dat, update with value of data in dat for that key. Also, if any key of data dosent exist in dat, then append that key in dat. But not able to implement it correctly.
You can try to compare the map and then update it if the key exists. Here's some example using your case.
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
func main() {
// read file
yfile1, err := ioutil.ReadFile("file1.yaml")
if err != nil {
log.Fatal(err)
return
}
yfile2, err := ioutil.ReadFile("file2.yaml")
if err != nil {
log.Fatal(err)
return
}
// unmarshal ymal
data1 := make(map[string]interface{})
if err := yaml.Unmarshal(yfile1, &data1); err != nil {
log.Fatal(err)
return
}
data2 := make(map[string]interface{})
if err := yaml.Unmarshal(yfile2, &data2); err != nil {
log.Fatal(err)
return
}
// from this we can iterate the key in data2 then check whether it exists in data1
// if so then we can update the value in data2
// iterate key in data2
for key2 := range data2 {
// check whether key2 exists in data1
if val1, ok := data1[key2]; ok {
// update the value of key2 in data2
data2[key2] = val1
}
}
fmt.Printf("data2: %v", data2)
// output:
// data2: map[city:washington name:john]
// you can write the data2 into ymal
newYfile2, err := yaml.Marshal(&data2)
if err != nil {
log.Fatal(err)
return
}
// write to file
if err = ioutil.WriteFile("new_file2.yaml", newYfile2, 0644); err != nil {
log.Fatal(err)
return
}
}
Inside new_file2.yaml will be like this:
city: washington
name: john
One thing that you need to take a not is that map in Go doesn't maintain the order (AFAIK Go doesn't have built-in OrderedMap type per 7 May 2022) so the order key in the new file will be random
Additional note: for error handling, you better handle it right away (right after you got the error). Here's a good article about it Error handling and Go
There is a maps package, since 1.18 I believe. If you don't care about new keys being added to the destination map you can use its copy function.
func Copy[M ~map[K]V, K comparable, V any](dst, src M)
Copy copies all key/value pairs in src adding them to dst. When a key in src is already present in dst, the value in dst will be overwritten by the value associated with the key in src.
The source code of that function is very simple:
func Copy[M ~map[K]V, K comparable, V any](dst, src M) {
for k, v := range src {
dst[k] = v
}
}
You could also do it yourself. The below code is from the helm source code. Unlike the copy function above, it works recursivly:
func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
out := make(map[string]interface{}, len(a))
for k, v := range a {
out[k] = v
}
for k, v := range b {
if v, ok := v.(map[string]interface{}); ok {
if bv, ok := out[k]; ok {
if bv, ok := bv.(map[string]interface{}); ok {
out[k] = mergeMaps(bv, v)
continue
}
}
}
out[k] = v
}
return out
}

Read Exif Metadata with Go

Does anyone have an example on using "github.com/gohugoio/hugo/resources/images/exif" to extract metadata from a local image using Go?
I looked through the docs and since I'm new to Go I'm not 100% sure if I'm doing things write. I do read the image, but I'm not sure what the next step would be.
fname := "image.jpg"
f, err := os.Open(fname)
if err != nil {
log.Fatal("Error: ", err)
}
(Edit 1)
Actually I think I found a solution:
d, err := exif.NewDecoder(exif.IncludeFields("File Type"))
x, err := d.Decode(f)
if err != nil {
log.Fatal("Error: ", err)
}
fmt.Println(x)
however, the question is how do I know what fields are available? File Type for example returns <nil>
Looks like Hugo uses github.com/rwcarlsen/goexif.
The documentation of the package on go.dev shows Exif.Walk can walk the name and tag for every non-nil EXIF field.
Eg, a small program:
package main
import (
"fmt"
"log"
"os"
"github.com/rwcarlsen/goexif/exif"
"github.com/rwcarlsen/goexif/tiff"
)
type Printer struct{}
func (p Printer) Walk(name exif.FieldName, tag *tiff.Tag) error {
fmt.Printf("%40s: %s\n", name, tag)
return nil
}
func main() {
if len(os.Args) < 2 {
log.Fatal("please give filename as argument")
}
fname := os.Args[1]
f, err := os.Open(fname)
if err != nil {
log.Fatal(err)
}
x, err := exif.Decode(f)
if err != nil {
log.Fatal(err)
}
var p Printer
x.Walk(p)
}
Example:
$ go run main.go IMG_123.JPG
ResolutionUnit: 2
YCbCrPositioning: 2
Make: "Canon"
Model: "Canon IXUS 255 HS"
ThumbJPEGInterchangeFormat: 5620
PixelYDimension: 3000
FocalPlaneResolutionUnit: 2
GPSVersionID: [2,3,0,0]
ExifVersion: "0230"
WhiteBalance: 1
DateTime: "2016:10:04 17:27:56"
CompressedBitsPerPixel: "5/1"
... etc ...
Orientation: 1
MeteringMode: 5
FocalLength: "4300/1000"
PixelXDimension: 4000
InteroperabilityIFDPointer: 4982
FocalPlaneXResolution: "4000000/244"
XResolution: "180/1"
ComponentsConfiguration: ""
ShutterSpeedValue: "96/32"
ApertureValue: "101/32"
ExposureBiasValue: "-1/3"
FocalPlaneYResolution: "3000000/183"
SceneCaptureType: 0

Get JSON paths to value from JSON

I'm trying to write a func to get all paths to values from a yaml file and I don't know know how that possible, here is my code:
func getpath(fileyaml) string {
if _, err := os.Stat(fileyaml); err == nil {
bfile, err := ioutil.ReadFile(fileyaml)
bjson, err := yaml.YAMLToJSON(bfile)
if err != nil {
fmt.Printf("YAMLToJSON err: %v\n", err)
}
json := string(bjson)
println json
paths := ? // don't know
return path
here my yaml file :
sentinel:
number: 3
server:
number: 7
config:
fere_size: 5
lcmea:
eza_ze: all
my function will convert it to a json: {"config":{"fere_size":5},"lcmea":{"eza_hooks":"all"},"sentinel":{"number":3},"server":{"number":7}}
the output that i want :
sentinel.number=3, server.number=3,config.fere_size=5,lcmea.eza_ze=all
how to parse this json in order to get this desired output?
I'm using "github.com/tidwall/gjson" to read the yaml and convert it to json
This is an example of how you could do it: https://play.golang.org/p/7yLq_PDLdXF
It is pretty naive and definitely could be improved, but it may give you an idea about how to parse the yaml file and then print the output in your desired format:
package main
import (
"fmt"
"log"
"strings"
"github.com/go-yaml/yaml"
)
var data = `
sentinel:
number: 3
server:
number: 7
config:
fere_size: 5
lcmea:
eza_ze: all
`
func main() {
m := make(map[string]map[string]interface{})
err := yaml.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatalf("error: %v", err)
}
out := []string{}
for k, v := range m {
for j, i := range v {
out = append(out, fmt.Sprintf("%s.%v=%v", k, j, i))
}
}
fmt.Println(strings.Join(out, ", "))
}
It will return:
$ go run main.go
sentinel.number=3, server.number=7, config.fere_size=5, lcmea.eza_ze=all
It is using https://github.com/go-yaml/yaml, check more examples on the README.md

Replacing a line within a file with Golang

I'm new to Golang, starting out with some examples. Currently, what I'm trying to do is reading a file line by line and replace it with another string in case it meets a certain condition.
The file is use for testing purposes contains four lines:
one
two
three
four
The code working on that file looks like this:
func main() {
file, err := os.OpenFile("test.txt", os.O_RDWR, 0666)
if err != nil {
panic(err)
}
reader := bufio.NewReader(file)
for {
fmt.Print("Try to read ...\n")
pos,_ := file.Seek(0, 1)
log.Printf("Position in file is: %d", pos)
bytes, _, _ := reader.ReadLine()
if (len(bytes) == 0) {
break
}
lineString := string(bytes)
if(lineString == "two") {
file.Seek(int64(-(len(lineString))), 1)
file.WriteString("This is a test.")
}
fmt.Printf(lineString + "\n")
}
file.Close()
}
As you can see in the code snippet, I want to replace the string "two" with "This is a test" as soon as this string is read from the file.
In order to get the current position within the file I use Go's Seek method.
However, what happens is that always the last line gets replaced by This is a test, making the file looking like this:
one
two
three
This is a test
Examining the output of the print statement which writes the current file position to the terminal, I get that kind of output after the first line has been read:
2016/12/28 21:10:31 Try to read ...
2016/12/28 21:10:31 Position in file is: 19
So after the first read, the position cursor already points to the end of my file, which explains why the new string gets appended to the end. Does anyone understand what is happening here or rather what is causing that behavior?
The Reader is not controller by the file.Seek. You have declared the reader as: reader := bufio.NewReader(file) and then you read one line at a time bytes, _, _ := reader.ReadLine() however the file.Seek does not change the position that the reader is reading.
Suggest you read about the ReadSeeker in the docs and switch over to using that. Also there is an example using the SectionReader.
Aside from the incorrect seek usage, the difficulty is that the line you're replacing isn't the same length as the replacement. The standard approach is to create a new (temporary) file with the modifications. Assuming that is successful, replace the original file with the new one.
package main
import (
"bufio"
"io"
"io/ioutil"
"log"
"os"
)
func main() {
// file we're modifying
name := "text.txt"
// open original file
f, err := os.Open(name)
if err != nil {
log.Fatal(err)
}
defer f.Close()
// create temp file
tmp, err := ioutil.TempFile("", "replace-*")
if err != nil {
log.Fatal(err)
}
defer tmp.Close()
// replace while copying from f to tmp
if err := replace(f, tmp); err != nil {
log.Fatal(err)
}
// make sure the tmp file was successfully written to
if err := tmp.Close(); err != nil {
log.Fatal(err)
}
// close the file we're reading from
if err := f.Close(); err != nil {
log.Fatal(err)
}
// overwrite the original file with the temp file
if err := os.Rename(tmp.Name(), name); err != nil {
log.Fatal(err)
}
}
func replace(r io.Reader, w io.Writer) error {
// use scanner to read line by line
sc := bufio.NewScanner(r)
for sc.Scan() {
line := sc.Text()
if line == "two" {
line = "This is a test."
}
if _, err := io.WriteString(w, line+"\n"); err != nil {
return err
}
}
return sc.Err()
}
For more complex replacements, I've implemented a package which can replace regular expression matches. https://github.com/icholy/replace
import (
"io"
"regexp"
"github.com/icholy/replace"
"golang.org/x/text/transform"
)
func replace2(r io.Reader, w io.Writer) error {
// compile multi-line regular expression
re := regexp.MustCompile(`(?m)^two$`)
// create replace transformer
tr := replace.RegexpString(re, "This is a test.")
// copy while transforming
_, err := io.Copy(w, transform.NewReader(r, tr))
return err
}
OS package has Expand function which I believe can be used to solve similar problem.
Explanation:
file.txt
one
two
${num}
four
main.go
package main
import (
"fmt"
"os"
)
var FILENAME = "file.txt"
func main() {
file, err := os.ReadFile(FILENAME)
if err != nil {
panic(err)
}
mapper := func(placeholderName string) string {
switch placeholderName {
case "num":
return "three"
}
return ""
}
fmt.Println(os.Expand(string(file), mapper))
}
output
one
two
three
four
Additionally, you may create a config (yml or json) and
populate that data in the map that can be used as a lookup table to store placeholders as well as their replacement strings and modify mapper part to use this table to lookup placeholders from input file.
e.g map will look like this,
table := map[string]string {
"num": "three"
}
mapper := func(placeholderName string) string {
if val, ok := table[placeholderName]; ok {
return val
}
return ""
}
References:
os.Expand documentation: https://pkg.go.dev/os#Expand
Playground

Defining Independent FlagSets in GoLang

The Go documentation (http://golang.org/pkg/flag/) says:
The FlagSet type allows one to define independent sets of flags, such as to implement subcommands in a command-line interface.
I need this functionality but I can't figure out how to persuade the flag pkg to do it. When I define two FlagSets, parsing one of them will give me errors and warnings if the commandline has flags that are meant for the second one. Example:
f1 := flag.NewFlagSet("f1", flag.ContinueOnError)
apply := f1.Bool("apply", false, "")
silent := f1.Bool("silent", false, "")
if err := f1.Parse(os.Args[1:]); err == nil {
fmt.Println(*apply, *silent)
}
f2 := flag.NewFlagSet("f2", flag.ContinueOnError)
reset := f2.Bool("reset", false, "")
if err := f2.Parse(os.Args[1:]); err == nil {
fmt.Println(*reset)
}
I get all sorts of warnings if I try to do cmd -apply OR cmd -reset. I want to keep these FlagSets separate because I want to only have -silent work for -apply.
What am I missing?
You are meant to distinguish between subcommands first, and then call Parse on the right FlagSet.
f1 := flag.NewFlagSet("f1", flag.ContinueOnError)
silent := f1.Bool("silent", false, "")
f2 := flag.NewFlagSet("f2", flag.ContinueOnError)
loud := f2.Bool("loud", false, "")
switch os.Args[1] {
case "apply":
if err := f1.Parse(os.Args[2:]); err == nil {
fmt.Println("apply", *silent)
}
case "reset":
if err := f2.Parse(os.Args[2:]); err == nil {
fmt.Println("reset", *loud)
}
}
http://play.golang.org/p/eaEEx_EReX
Turns out it is possible to consume one set of flags while capturing the flags not recognized by the first set and while discarding the error messages that the flag package is prepared to emit whenever it runs into an option it doesn't recognize.
Here's one way to do it.
import (
"flag"
"fmt"
"io/ioutil"
"os"
"strings"
)
func gentleParse(flagset *flag.FlagSet, args []string) []string {
if len(args) == 0 {
return nil
}
r := make([]string, 0, len(args))
flagset.Init(flagset.Name(), flag.ContinueOnError)
w := flagset.Output()
flagset.SetOutput(ioutil.Discard)
defer flagset.SetOutput(w)
next := args
for len(next) > 0 {
if next[0] == "--" {
r = append(r, next...)
break
}
if !strings.HasPrefix(next[0], "-") {
r, next = append(r, next[0]), next[1:]
continue
}
if err := flagset.Parse(next); err != nil {
const prefix = "flag provided but not defined: "
if strings.HasPrefix(err.Error(), prefix) {
pull := strings.TrimPrefix(err.Error(), prefix)
for next[0] != pull {
next = next[1:]
}
r, next = append(r, next[0]), next[1:]
continue
}
fmt.Fprintf(w, "%s\n", err)
flagset.SetOutput(w)
flag.Usage()
os.Exit(1)
}
next = flag.Args()
}
return r
}
Just change these code
if err := f2.Parse(os.Args[1:]); err == nil {
fmt.Println(*reset)
}
to
f2.Parse(os.Args[1:])
fmt.Println(*reset)
but the warning is just left on the console.if u wanna remove it ,modify /usr/local/go/src/flag/flag.go and recompile the golang ..
or do a copy of flag package.
  →_→ 怀疑的眼神~~

Resources