I have a yaml file i am trying to read in and loop over to extract some of the data.
example of my yaml
---
THE/NAME/DOESNT/MATTER:
TEST_NAME: THIS/IS/WHAT/IS/DISPLAYS/ON/THE/HTML
RUN_PATH: example/testfiles/release
SOMETHING_ELSE: [1,2,3]
ANOTHER/PATH/LIKE/STRING:
TEST_NAME: USED/FOR/THE/HTML
RUN_PATH: example/testfiles/foo
my code
package main
import (
"fmt"
"os"
"io/ioutil"
"gopkg.in/yaml.v3"
)
func main() {
reportYAML := os.Args[1]
// userDictionary := os.Args[2]
yfile, err := ioutil.ReadFile(reportYAML)
if err != nil {
fmt.Printf("ERROR: Unable to open yaml file : %s\n", err)
}
data := make(map[interface{}]interface{})
error := yaml.Unmarshal([]byte(yfile), &data)
if error != nil {
fmt.Printf("ERROR: Unable to read yaml file : %s\n", err)
}
for _, value := range data {
fmt.Printf("%T\n", value)
}
}
the yaml will repeat with a similar pattern and i am trying to extract both TEST_NAME and RUN_PATH. Printf with %T gives me map[string]interface {} and %s map[RUN_PATH:example/testfiles/release SOMETHING_ELSE:[%!s(int=1) %!s(int=2) %!s(int=3)] TEST_NAME:THIS/IS/WHAT/IS/DISPLAYS/ON/THE/HTML]
I've been trying value["TESTNAME"], value.(string)["TESTNAME"], value["TESTNAME"].(string) and a few other variations but they all give me errors.
I know this is a simple problem but i have very little experience with GO and can't work it out from previous, similar stackoverflow posts
additional context: each yaml top-level-key will contain multiple key:value pairs but i am only interested in the TEST_NAME and RUN_PATH
If you only care about a few specific keys, just create a data structure to expose those values:
package main
import (
"fmt"
"io/ioutil"
"os"
"gopkg.in/yaml.v3"
)
type (
Entry struct {
TestName string `yaml:"TEST_NAME"`
RunPath string `yaml:"RUN_PATH"`
}
)
func main() {
reportYAML := os.Args[1]
// userDictionary := os.Args[2]
yfile, err := ioutil.ReadFile(reportYAML)
if err != nil {
fmt.Printf("ERROR: Unable to open yaml file : %s\n", err)
}
data := make(map[string]Entry)
error := yaml.Unmarshal([]byte(yfile), &data)
if error != nil {
fmt.Printf("ERROR: Unable to read yaml file : %s\n", err)
}
for _, value := range data {
fmt.Printf("test_name: %s\n", value.TestName)
fmt.Printf("run_path: %s\n", value.RunPath)
}
}
Running the above code against your example data produces:
test_name: THIS/IS/WHAT/IS/DISPLAYS/ON/THE/HTML
run_path: example/testfiles/release
test_name: USED/FOR/THE/HTML
run_path: example/testfiles/foo
Related
Language: Go
I am practicing how to read and print the yaml file data but unable to do so and the code is getting through. Can someone help here?
Yaml file:
ENV:
foo: test
boo: test-123-tet222
code:
package test
import (
"testing"
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
type config struct {
foo string
boo string
}
func TestTerraformAzureCosmosDBExample(t *testing.T) {
yFile, err := ioutil.ReadFile("config.yaml")
if err != nil {
log.Fatal(err)
}
data := make(map[string]config)
err2 := yaml.Unmarshal(yFile, &data)
if err2 != nil {
log.Fatal(err2)
}
for k, v := range data {
fmt.Printf(k, v)
}
}
Expected Output:
foo: test
boo: test-123-tet222
Actual Output:
C:\foo\boo>go test -v
=== RUN TestTerraformAzureCosmosDBExample
ENV%!(EXTRA test.config={ })--- PASS: TestTerraformAzureCosmosDBExample (0.00s)
PASS
ok foobo_test 0.179s
Your config struct is missing yaml tags. Edit it as follows. Also the Printf method expects a formatter string, edit that too as follows.
import (
"fmt"
"gopkg.in/yaml.v3"
"io/ioutil"
"log"
"testing"
)
type config struct {
Foo string `yaml:"foo"`
Boo string `yaml:"boo"`
}
func TestTerraformAzureCosmosDBExample(t *testing.T) {
yFile, err := ioutil.ReadFile("config.yaml")
if err != nil {
log.Fatal(err)
}
data := make(map[string]config)
err = yaml.Unmarshal(yFile, &data)
if err != nil {
log.Fatal(err)
}
for k, v := range data {
fmt.Printf(`key: %v, value: %v`, k,v)
}
}
gives the output:
key: ENV, value: {test test-123-tet222}--- PASS: TestTerraformAzureCosmosDBExample (0.00s)
You are iterating over data that is a map[string]config.
That object has one key, ENV and the value for that key is the config object you are looking for.
Try with:
fmt.Printf("Foo: %S\n", data["ENV"].Foo)
fmt.Printf("Boo: %S\n", data["ENV"].Boo)
I'm trying to use gopkg.in/yaml.v3 to read a file containing
multiple YAML documents, and then marshal each individual document
back into YAML. My sample input looks like:
name: doc1
---
name: doc2
---
name: doc3
I'm able to unmarshal the file just fine, but I run into unexpected
errors when trying to marshal the individual documents. My code looks
like:
package main
import (
"errors"
"fmt"
"io"
"os"
"gopkg.in/yaml.v3"
)
func main() {
reader := io.Reader(os.Stdin)
dec := yaml.NewDecoder(reader)
for {
var node yaml.Node
err := dec.Decode(&node)
if errors.Is(err, io.EOF) {
break
}
if err != nil {
panic(err)
}
content, err := yaml.Marshal(node)
if err != nil {
panic(err)
}
fmt.Printf("Found a doc\n\n%s\n", content)
}
}
Running this code with the sample input results in:
panic: yaml: expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS, but got document start
goroutine 1 [running]:
main.main()
/home/lars/projects/operate-first/halberd/main.go:31 +0x451
The code seems to be parsing the document correctly; if I replace it
with:
package main
import (
"errors"
"fmt"
"io"
"os"
"gopkg.in/yaml.v3"
)
func main() {
reader := io.Reader(os.Stdin)
dec := yaml.NewDecoder(reader)
for {
var node yaml.Node
err := dec.Decode(&node)
if errors.Is(err, io.EOF) {
break
}
if err != nil {
panic(err)
}
fmt.Printf("Found a doc\n")
}
}
I end up with what I expect:
Found a doc
Found a doc
Found a doc
The problem only crops up when marshaling. Am I correctly using the v3 api?
There was a small mistake in your code.
It should be content, err := yaml.Marshal(&node). You missed the &.
This works,
package main
import (
"errors"
"fmt"
"io"
"os"
"gopkg.in/yaml.v3"
)
func main() {
reader := io.Reader(os.Stdin)
dec := yaml.NewDecoder(reader)
for {
var node yaml.Node
err := dec.Decode(&node)
if errors.Is(err, io.EOF) {
break
}
if err != nil {
panic(err)
}
content, err := yaml.Marshal(&node)
if err != nil {
panic(err)
}
fmt.Printf("Found a doc\n\n%s\n", content)
}
}
The docs for yaml.Marshal says,
Marshal serializes the value provided into a YAML document. The structure of the generated document will reflect the structure of the value itself. Maps and pointers (to struct, string, int, etc) are accepted as the in value.
Source: https://pkg.go.dev/gopkg.in/yaml.v2#Marshal
Since you are trying to marshal yaml.Node which is a struct, you must pass the pointer to the struct and not the value itself, as the docs say.
I've just recently started trying to learn Go and I'm trying to write a small server/client application for sending a csv file, from a server to a client. I'm running into an invalid type error when trying to encode a struct into BigEndian binary. My struct seems to already be in a binary format, I'm unsure why I get the following error:
Error writing binary buffer to big endian binary.Write: invalid type *main.DataPack
Also, I'd like to keep the TCP connection open after sending the file, that's why I'm not using io.Copy.
Currently I'm triggering the handling of the file upload through by sending a '\x00' byte:
// Server
package main
import (
"path/filepath"
"fmt"
"io/ioutil"
"net"
"os"
"encoding/binary"
"bytes"
)
type DataPack struct {
Length int64
Contents []byte
}
func main() {
absPath, _ := filepath.Abs("./progs.csv")
data, _ := ioutil.ReadFile(absPath)
fmt.Println("%s",data)
tel, err := net.Listen("tcp", "0.0.0.0:23")
if err != nil {
fmt.Println(err)
return
}
for {
conn, err := tel.Accept()
if err != nil {
break
}
fmt.Println("above filehandler")
go fileHandler(conn)
}
}
func fileHandler(conn net.Conn) {
buf := make([]byte, 0)
buf = append(buf, '\x00')
conn.Write(buf)
absPath, _ := filepath.Abs("./progs.csv")
file, err := os.Open(absPath)
defer file.Close()
fi, err := file.Stat()
if err != nil {
fmt.Println("Fatal error reading file: ", err)
}
fmt.Println("This is the length of the file: ", fi.Size())
data := &DataPack{Length: fi.Size()} // , Contents: }
data.Contents = make([]byte, data.Length)
n, err := file.Read(data.Contents)
if err != nil {
fmt.Println("error reading contents into struct: ", err)
}
fmt.Println("tried to read file contents: ", n)
fmt.Println("DataPack: %+v", data)
buf1 := new(bytes.Buffer)
err = binary.Write(buf1, binary.BigEndian, &data)
if err != nil {
fmt.Println("Error writing binary buffer to big endian ", err)
}
conn.Write(buf1.Bytes())
}
Here is the client
package main
import (
"log"
"fmt"
"net"
"strings"
"strconv"
"bufio"
)
const (
host = "127.0.0.1"
port = 23
)
func main() {
addr := strings.Join([]string{host, strconv.Itoa(port)}, ":")
client := NewClient()
var err error
client.socket, err = net.Dial("tcp", addr)
if err != nil {
log.Println("error setting up socket: %s", err)
}
for {
m := bufio.NewReader(client.socket)
b, err := m.ReadByte()
if err != nil {
fmt.Println("here is the error: ", err)
}
if b == '\x00'{
fmt.Println("about to receive a file!!!")
b, _ := m.ReadByte()
fmt.Println("just got another byte ", b )
}
}
log.Printf("Over")
}
why you get the error
Visit https://golang.org/pkg/encoding/binary/#Write
Write writes the binary representation of data into w. Data must be a fixed-size value or a slice of fixed-size values, or a pointer to such data.
type DataPack struct {
Length int64
Contents []byte
}
Contents isn't a fixed-size value, so you got the invalid type error.
how to solve it
go binary
json
others
I have a file, 'test.txt', containing the following data. This file was created from the same structures from the code below using marshaling.
{"VLETXGJM":{"attrib1":"test1","attrib2":"test2"}}
I am trying to read it back from the file and unmarshal it into a map using the same structures. I can successfully read the data from the file. I receive no errors when I try to unmarshal it into the map. However, my map is empty.
The mutex is used to protect the map since my real implementation (this is an extracted test) needs to use a protected map for concurrency. I have tried this same code removing the sync library and received the same negative result.
The test code:
package main
import (
"encoding/json"
"fmt"
"sync"
"os"
)
type TestObject struct {
Attrib1 string `json:"attrib1"`
Attrib2 string `json:"attrib2"`
}
type TestMap map[string]TestObject
type TestList struct {
sync.RWMutex
list TestMap
}
func main() {
tl := TestList{ list: make(TestMap) }
// Read the list back out of the file
fi, err := os.Open("test.txt")
if os.IsNotExist(err) {
fmt.Println("data file does not exist")
panic(nil)
}
if err != nil {
panic(err)
}
defer func() {
if err := fi.Close(); err != nil {
panic(err)
}
}()
data := make([]byte, 1024 * 1024)
count, err := fi.Read(data)
if err != nil {
panic(err)
}
fmt.Printf("read from file: \"%s\"\n",data[:count])
tl.Lock()
err = json.Unmarshal(data[:count], &tl)
if err != nil {
panic(err)
}
tl.Unlock()
// List it out
tl.Lock()
if len(tl.list) == 0 {
fmt.Println("Empty list")
} else {
for key, _ := range tl.list {
fmt.Printf("%s: %s\n", tl.list[key].Attrib1, tl.list[key].Attrib2)
}
}
tl.Unlock()
}
The output of the run is:
read from file: "{"VLETXGJM":{"attrib1":"test1","attrib2":"test2"}}"
Empty list
Thank you for your help. I have searched for similar issues and not yet found an exact duplicate of this scenario.
I think you want to unmarshal into tl.list instead of tl:
err = json.Unmarshal(data[:count], &tl.list)
tl has no exported fields, so Unmarshal into tl won't do anything. tl.list (i.e., type TestMap) matches your data.
I have an issue with reading a YAML file. I think it's something in the file structure but I can't figure out what.
YAML file:
conf:
hits:5
time:5000000
code:
type conf struct {
hits int64 `yaml:"hits"`
time int64 `yaml:"time"`
}
func (c *conf) getConf() *conf {
yamlFile, err := ioutil.ReadFile("conf.yaml")
if err != nil {
log.Printf("yamlFile.Get err #%v ", err)
}
err = yaml.Unmarshal(yamlFile, c)
if err != nil {
log.Fatalf("Unmarshal: %v", err)
}
return c
}
your yaml file must be
hits: 5
time: 5000000
your code should look like this:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"io/ioutil"
"log"
)
type conf struct {
Hits int64 `yaml:"hits"`
Time int64 `yaml:"time"`
}
func (c *conf) getConf() *conf {
yamlFile, err := ioutil.ReadFile("conf.yaml")
if err != nil {
log.Printf("yamlFile.Get err #%v ", err)
}
err = yaml.Unmarshal(yamlFile, c)
if err != nil {
log.Fatalf("Unmarshal: %v", err)
}
return c
}
func main() {
var c conf
c.getConf()
fmt.Println(c)
}
the main error was capital letter for your struct.
Example
Using an upgraded version 3 of yaml package.
An example conf.yaml file:
conf:
hits: 5
time: 5000000
camelCase: sometext
The main.go file:
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
type myData struct {
Conf struct {
Hits int64
Time int64
CamelCase string `yaml:"camelCase"`
}
}
func readConf(filename string) (*myData, error) {
buf, err := ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
c := &myData{}
err = yaml.Unmarshal(buf, c)
if err != nil {
return nil, fmt.Errorf("in file %q: %w", filename, err)
}
return c, err
}
func main() {
c, err := readConf("conf.yaml")
if err != nil {
log.Fatal(err)
}
fmt.Printf("%#v", c)
}
Running instructions (in case it's the first time you step out of stdlib):
cat conf.yaml
go mod init example.com/whatever
go get gopkg.in/yaml.v3
cat go.sum
go run .
About The yaml:"field"
The tags (like yaml:"field") are optional for all-lowercase yaml key identifiers. For demonstration the above example parses an extra camel case identifier which does require such a tag.
Corner Case: JSON+YAML
Confusingly, the useful lowercasing behavior of yaml package is not seen in the standard json package. Do you have a data structure which is sometimes encoded to JSON and sometimes to YAML? If so, consider specifying both JSON tags and YAML tags on literally every field. Verbose, but reduces mistakes. Example below.
type myData struct {
Conf conf `yaml:"conf" json:"conf"`
}
type conf struct {
Hits int64 `yaml:"hits" json:"hits"`
Time int64 `yaml:"time" json:"time"`
CamelCase string `yaml:"camelCase" json:"camelCase"`
}
package main
import (
"gopkg.in/yaml.v2"
"io/ioutil"
"log"
)
type someConf struct {
AccessKeyID string `yaml:"access_key_id"`
//...
}
func getConf(file string, cnf interface{}) error {
yamlFile, err := ioutil.ReadFile(file)
if err == nil {
err = yaml.Unmarshal(yamlFile, cnf)
}
return err
}
func main() {
cfg := &awsConf{}
if err := getConf("conf.yml", cfg); err != nil {
log.Panicln(err)
}
}
shortest getConf ever