Read multiple yamls in a file - go

How does one parse multiple yamls in a file similar to how kubectl does it?
example.yaml
---
a: Easy!
b:
c: 0
d: [1, 2]
---
a: Peasy!
b:
c: 1000
d: [3, 4]

There's a difference in behavior between gopkg.in/yaml.v2 and gopkg.in/yaml.v3:
V2: https://play.golang.org/p/XScWhdPHukO
V3: https://play.golang.org/p/OfFY4qH5wW2
Both implementations produce an incorrect result IMHO but V3 is apparently slightly worse.
There's a workaround. If you slightly change code in the accepted answer, it works correctly and in the same fashion with both versions of yaml package: https://play.golang.org/p/r4ogBVcRLCb

Current gopkg.in/yaml.v3 Deocder produces quite correct result, you just need to pay attention on creating new structure for each document and check it was parsed correctly (with nil check), and handle EOF error correctly:
package main
import "fmt"
import "gopkg.in/yaml.v3"
import "os"
import "errors"
import "io"
type Spec struct {
Name string `yaml:"name"`
}
func main() {
f, err := os.Open("spec.yaml")
if err != nil {
panic(err)
}
d := yaml.NewDecoder(f)
for {
// create new spec here
spec := new(Spec)
// pass a reference to spec reference
err := d.Decode(&spec)
// check it was parsed
if spec == nil {
continue
}
// break the loop in case of EOF
if errors.Is(err, io.EOF) {
break
}
if err != nil {
panic(err)
}
fmt.Printf("name is '%s'\n", spec.Name)
}
}
Test file spec.yaml:
---
name: "doc first"
---
name: "second"
---
---
name: "skip 3, now 4"
---

Solution I found using gopkg.in/yaml.v2:
package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"path/filepath"
"gopkg.in/yaml.v2"
)
type T struct {
A string
B struct {
RenamedC int `yaml:"c"`
D []int `yaml:",flow"`
}
}
func main() {
filename, _ := filepath.Abs("./example.yaml")
yamlFile, err := ioutil.ReadFile(filename)
if err != nil {
panic(err)
}
r := bytes.NewReader(yamlFile)
dec := yaml.NewDecoder(r)
var t T
for dec.Decode(&t) == nil {
fmt.Printf("a :%v\nb :%v\n", t.A, t.B)
}
}

Related

How to find full package import from CallExpr

The following method extracts all public method calls from the AST of a file. I need to find out the full package from the CallExpr, for example: ast.Inspect() is imported from "go/ast". I want to match the list of pkgsInclude strings with the imported package name:
func functionCalls(path string, node *ast.File, pkgsInclude []string) int {
fCalls := 0
ast.Inspect(node, func(n ast.Node) bool {
switch fCall := n.(type) {
case *ast.CallExpr:
if fun, ok := fCall.Fun.(*ast.SelectorExpr); ok {
if fun.Sel.IsExported() {
fCalls += 1
}
}
}
return true
})
return fCalls
}
To get fully qualified names, the code has to be type checked with the go/types package.
The go/types article by Alan Donovan goes into great detail on how to use the type checker properly, but here is the gist of it. I left a few type assertions in the Visit method for brevity. In production code you shouldn't assume specific node types.
package main
import (
"fmt"
"go/ast"
"go/importer"
"go/parser"
"go/token"
"go/types"
"log"
)
// code to parse. It includes two variants of calling a package function.
var code = `package main
import (
foo "io/ioutil"
. "io/ioutil"
)
func main() {
foo.ReadFile("")
ReadFile("")
}
`
func main() {
fset := &token.FileSet{}
f, err := parser.ParseFile(fset, "", code, 0)
if err != nil {
log.Fatal(err)
}
// info.Uses allows to lookup import paths for identifiers.
info := &types.Info{
Uses: make(map[*ast.Ident]types.Object),
}
// Type check the parsed code using the default importer.
// Use golang.org/x/tools/go/loader to check a program
// consisting of multiple packages.
conf := types.Config{Importer: importer.Default()}
pkg, err := conf.Check("main", fset, []*ast.File{f}, info)
if err != nil {
log.Fatal(err)
}
// Do something with ast, info, and possibly pkg
var _ = pkg
ast.Walk(v{info}, f)
}
type v struct {
info *types.Info
}
func (v v) Visit(node ast.Node) (w ast.Visitor) {
switch node := node.(type) {
case *ast.CallExpr:
// Get some kind of *ast.Ident for the CallExpr that represents the
// package. Then we can look it up in v.info. Where exactly it sits in
// the ast depends on the form of the function call.
switch node := node.Fun.(type) {
case *ast.SelectorExpr: // foo.ReadFile
pkgID := node.X.(*ast.Ident)
fmt.Println(v.info.Uses[pkgID].(*types.PkgName).Imported().Path())
case *ast.Ident: // ReadFile
pkgID := node
fmt.Println(v.info.Uses[pkgID].Pkg().Path())
}
}
return v
}
// Output:
// io/ioutil
// io/ioutil

Golang yaml.v2 marshals an array as a sequence

Given the following YAML:
array.test: ["val1", "val2", "val3"]
I Unmarshal it using gopkg.in/yaml.v2 into a map[string]interface{}. Then I get a single key whose value is an array of 3 values.
When I then Marshal it again to YAML, the resulting YAML looks like this:
array.test:
- val1
- val2
- val3
The array was actually marshaled as a sequence instead of an array.
This is the entire GoLang code:
func main(){
data := `array.test: ["val1", "val2", "val3"]`
conf := make(map[string]interface{})
yaml.Unmarshal([]byte(data), conf)
data2, _ := yaml.Marshal(conf)
fmt.Printf("%s\n", string(data2))
}
How can I overcome this issue?
This one helped me in the same case as you.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var data = `
a: Easy!
b:
c: 2
d.test: ["d1", "d2"]
`
// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
A string
B struct {
RenamedC int `yaml:"c"`
DTest []string `yaml:"d.test,flow"`
}
}
func main() {
// if we use struct containing yaml encoding for yaml formated string
t := T{}
err := yaml.Unmarshal([]byte(data), &t)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t after unmarshal:\n%v\n\n", t)
d, err := yaml.Marshal(&t)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t after marshal:\n%s\n\n", string(d))
}
Ref: https://github.com/go-yaml/yaml
Use flow in struct field tag format, to indicate you desire this behavior. But, of course, this requires unmarshaling to a struct, not to a map.
Flow tag allows you to choose the representation of an array in yaml
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type Conf struct {
Test []string `yaml:"array.test,flow"`
}
func main(){
data := `array.test: ["val1", "val2", "val3"]`
var conf Conf
yaml.Unmarshal([]byte(data), &conf)
data2, _ := yaml.Marshal(conf)
fmt.Printf("%s\n", string(data2))
}

golang comments and docs fields when doing ast.Inspect — why are they blank?

I'm trying to get at the Docs and Comments of structs and struct fields, but I can't seem to be able to do so, they just turn up empty:
package main
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
)
func main() {
src := `package test
// Hello
type A struct {
// Where
B int // Are you
}
`
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "", src, 0)
if err != nil {
panic(err)
}
ast.Inspect(f, func(n ast.Node) bool {
switch t := n.(type) {
case *ast.TypeSpec:
fmt.Println(t.Doc.Text())
case *ast.StructType:
for _, field := range t.Fields.List {
fmt.Println(field.Doc.Text())
fmt.Println(field.Comment.Text())
}
}
return true
})
}
yields three blank lines: https://play.golang.org/p/4Eh9gS-PUg
Saw the similar question Go parser not detecting Doc comments on struct type but when trying to run the accepted example it turns up all empty — so I'm wondering if something has changed since that version.
In order to get comments, you have to pass the parser.ParseComments flag in argument to parser.ParseFile():
parser.ParseFile(fset, "", src, parser.ParseComments)
All possible mode flags are documented here:
https://golang.org/pkg/go/parser/#Mode

How to run Binary Files inside GoLang Program?

I want to execute Binary Files inside GoLang Program.
Here is my code:
package main
import (
"fmt"
"os/exec"
)
func main() {
output, _ := exec.Command("/home/user/Golang/bin/hello").Output()
fmt.Println(output)
}
But I get the output as: []
Thanks in advance.
I can get the output.
package main
import (
"fmt"
"os/exec"
)
func main() {
output, err := exec.Command("/Users/duguying/gopath/bin/test").Output()
if err!=nil {
fmt.Println(err.Error())
}
fmt.Println(string(output))
}
check you binary file first or binary filepath is correcting. try to print out your error message.
When I'm looking at the source of the exec.Command() it doesnt return an error but only returns Cmd which is struct in the package exe :
source
....
func Command(name string, arg ...string) *Cmd {
cmd := &Cmd{
Path: name,
Args: append([]string{name}, arg...),
}
if filepath.Base(name) == name {
if lp, err := LookPath(name); err != nil {
cmd.lookPathErr = err
} else {
cmd.Path = lp
}
}
return cmd
}
....
I have succesfully got the binary file running using this code :
package main
import (
"fmt"
"os/exec"
)
func main() {
command:= exec.Command("Your binary file path")
// set var to get the output
var out bytes.Buffer
// set the output to our variable
command.Stdout = &out
err = command.Run()
if err != nil {
log.Println(err)
}
fmt.Println(out.String())
}
This one works for me for running a binary file that will print some random string.

Any way to use html.Parse without it adding nodes to make a 'well-formed tree'?

package main
import (
"bytes"
"code.google.com/p/go.net/html"
"fmt"
"log"
"strings"
)
func main() {
s := "Blah. <b>Blah.</b> Blah."
n, err := html.Parse(strings.NewReader(s))
if err != nil {
log.Fatalf("Parse error: %s", err)
}
var buf bytes.Buffer
if err := html.Render(&buf, n); err != nil {
log.Fatalf("Render error: %s", err)
}
fmt.Println(buf.String())
}
Output:
<html><head></head><body>Blah. <b>Blah.</b> Blah.</body></html>
Is there a way to stop html.Parse from making a document out of fragments (ie avoid adding <html>, <body> etc.)? I'm aware of html.ParseFragment but it seems to exhibit the same behaviour.
You can get around it by wrapping the text to be parsed with a parent element such as <span> then doing something like the following:
n = n.FirstChild.LastChild.FirstChild
but that seems, well, kludgy to say the least.
Ideally I'd like to: accept input, manipulate or remove nodes found within it, and write the result back to a string, even if the result is an incomplete document.
You need to provide a context to ParseFragment. The following program prints out the original text:
package main
import (
"bytes"
"code.google.com/p/go.net/html"
"code.google.com/p/go.net/html/atom"
"fmt"
"log"
"strings"
)
func main() {
s := "Blah. <b>Blah.</b> Blah."
n, err := html.ParseFragment(strings.NewReader(s), &html.Node{
Type: html.ElementNode,
Data: "body",
DataAtom: atom.Body,
})
if err != nil {
log.Fatalf("Parse error: %s", err)
}
var buf bytes.Buffer
for _, node := range n {
if err := html.Render(&buf, node); err != nil {
log.Fatalf("Render error: %s", err)
}
}
fmt.Println(buf.String())
}
You want http://godoc.org/code.google.com/p/go.net/html#ParseFragment. Pass in a fake Body element as your context and the fragment will be returned as a slice of just the elements in your fragment.
You can see an example in the Partial* functions for go-html-transform's go.net/html wrapper package. https://code.google.com/p/go-html-transform/source/browse/h5/h5.go#32

Resources