How do I reuse the original strings.Reader in Go? - go

I have an input of type strings.Reader. Given the input, I am extracting the id from it and printing it out. I then pass the original input to a generic function that perform other tasks on it. The only way I can think of reusing the original is to read the content and pass it to a bytes.Reader twice.
Is the following the only way to achieve that in Go?
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"strings"
)
type Food struct {
Id int `json:"id"`
Name string `json:"name"`
}
func genericFunction(body io.Reader) {
content, err := ioutil.ReadAll(body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(content))
}
func main() {
// Original input
reader := strings.NewReader("{\"id\": 10, \"name\": \"Pie\"}")
original, err := ioutil.ReadAll(reader)
if err != nil {
log.Fatal(err)
}
foodReader := bytes.NewReader(original)
decoder := json.NewDecoder(foodReader)
var food Food
decoder.Decode(&food)
fmt.Println("About to eat food", food.Id)
foodReader = bytes.NewReader(original)
genericFunction(foodReader)
}

You can seek back to the start of the string with either the strings.Reader or bytes.Reader
reader := bytes.NewReader([]byte("{\"id\": 10, \"name\": \"Pie\"}"))
decoder := json.NewDecoder(reader)
var food Food
decoder.Decode(&food)
fmt.Println("About to eat food", food.Id)
reader.Seek(0, 0)
genericFunction(reader)

Related

Go elasticsearch bulk insert without using the library

HI i wanted to ask the community since I am not an expert in GO how do I add \n at the end of the request so as not to get an error when inserting a large amount of data into elasticsearch bulk
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}
apply the code what I wrote - just say this is a test
package main
import (
"bytes"
json "encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type ElkInsertIndex struct {
Index string `json:"_index"`
ID string `json:"_id"`
}
type ElkBulkInsert struct {
Index []ElkInsertIndex `json:"index"`
}
type ElkUrl struct {
Url string `json:"url"`
}
func main() {
data := ElkBulkInsert{
Index: []ElkInsertIndex{
{
Index: "shut_url",
ID: "FFFFFFFFFFFFFFFF",
},
},
}
js, err := json.Marshal(data)
testBulk := bytes.NewBuffer(js)
resp1, err := http.Post("http://127.0.0:9200/_bulk", "application/json", testBulk)
if err != nil {
log.Println(err)
}
body, err := ioutil.ReadAll(resp1.Body)
if err != nil {
log.Println(err)
}
fmt.Println(string(body))
}

How to extract data from map[string]interface{}?

I am sending the data to the API like following:
{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}
and i am getting following while printing :
map[after:map[amount:811 id:123 status:Hi ] key:[70]]
Is there any way to print individual field like this??
amount::800
id:123
status:Hi
The code:
package main
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"strings"
)
var (
PORT = ":8080"
)
func main() {
fmt.Println("In Main")
http.HandleFunc("/", changedData)
http.ListenAndServe(PORT, nil)
}
type Data struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
type mark map[string]interface{}
func changedData(w http.ResponseWriter, r *http.Request) {
fmt.Println("Coming From API")
reqBody, _ := ioutil.ReadAll(r.Body)
fmt.Println("Data coming from API ", string(reqBody))
digit := json.NewDecoder(strings.NewReader(string(reqBody)))
for digit.More() {
var result mark
err := digit.Decode(&result)
if err != nil {
if err != io.EOF {
log.Fatal(err)
}
break
}
fmt.Println("final_data ", result)
}
}
Decode to a Go type that matches the structure of the JSON document. You declared a type for the "after" field. Wrap that type with a struct to match the document.
func changedData(w http.ResponseWriter, r *http.Request) {
var v struct{ After Data }
err := json.NewDecoder(r.Body).Decode(&v)
if err != nil {
http.Error(w, "bad request", 400)
return
}
fmt.Printf("final_data: %#v", v.After)
}
Playground example.
I think you can define a struct type if you know the JSON file format or if the JSON format is predefined. As far as I know that mostly using interface{} is a way when you don't know the JSON format or there is no predefined format of the JSON. If you define a struct type  and use it while unmarshaling the JSON to struct, you can access the variables by typing like data.Id or data.Status.
Here's an example code:
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
AfterData After `json:"after"`
Key []int `json:"key"`
}
type After struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
func main() {
j := []byte(`{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}`)
var data *Data
err := json.Unmarshal(j, &data)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(data.AfterData)
fmt.Println(data.AfterData.Id)
fmt.Println(data.AfterData.Amount)
fmt.Println(data.AfterData.Status)
}
Output will be
{123 811 Hi}
123
811
Hi
Go Playground

Reading and Unmarshalling API results in Golang

In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}

Check if json has fields other than valid keys in struct

I'm new to Golang and this could be so basic. Below is an example demonstrating my need. Junk should accept only id and name. Even though unmarshal eliminates extras, i need a way to hard stop execution stating json contain invalid key value pair. Could you please let me know if there's any.
package main
import (
"fmt"
"encoding/json"
)
type Junk struct{
ID int `json:"id"`
Name string `json:"name"`
Area string `json:"area"`
}
func main() {
a:=Junk{}
data:= `{"id":1,"name":"gg","junk":"Junk value"}`
err := json.Unmarshal([]byte(data),&a)
if err!=nil{
fmt.Println(err)
}
fmt.Println(a)
}
Use func (*Decoder) DisallowUnknownFields():
DisallowUnknownFields causes the Decoder to return an error when the
destination is a struct and the input contains object keys which do
not match any non-ignored, exported fields in the destination.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type Junk struct {
ID int `json:"id"`
Name string `json:"name"`
Area string `json:"area"`
}
func main() {
a := Junk{}
data := `{"id":1,"name":"gg","junk":"Junk value"}`
d := json.NewDecoder(strings.NewReader(data))
d.DisallowUnknownFields()
if err := d.Decode(&a); err != nil {
fmt.Println(err)
}
fmt.Println(a)
}
Try it out: https://play.golang.org/p/aTj2C-AAuZ7

Golang JSON Array

I am trying to use WordNik () to get random words for the dictionary for this script: https://github.com/jmagrippis/password
WordNik is outputting:
[{"id":7936915,"word":"Tanganyikan"},{"id":27180,"word":"cartographic"},{"id":48094,"word":"deterministic"},{"id":1485119,"word":"higher-risk"},{"id":120986,"word":"juristic"},{"id":1830806,"word":"magnetorheological"},{"id":320495,"word":"quelled"},{"id":324610,"word":"remoter"},{"id":215158,"word":"telemetric"},{"id":225207,"word":"uninquisitive"}]
Here is my code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/jmagrippis/password"
)
type Words struct {
id []int64
word []string
}
type GetWordsResponse struct {
WordList []Words
}
func getWords(speech string) (*GetWordsResponse, error) {
url := fmt.Sprintf("http://api.wordnik.com/v4/words.json/randomWords?hasDictionaryDef=false&includePartOfSpeech=%s&minCorpusCount=0&maxCorpusCount=-1&minDictionaryCount=1&maxDictionaryCount=-1&minLength=5&maxLength=-1&limit=10&api_key=api_key", speech)
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
var s = new(GetWordsResponse)
var arr []string
_ = json.Unmarshal([]byte(body), &arr)
log.Printf("Unmarshaled: %v", arr)
return s, err
}
func main() {
dictionary := &password.Dictionary{
Adjectives: []string{"beautiful", "homely", "magical", "posh", "excellent"},
Subjects: []string{"mermaids", "unicorns", "lions", "piranhas"},
Verbs: []string{"love", "fancy", "eat", "bring", "fear", "aggravate"},
Adverbs: []string{"cuddling", "slapping", "shouting", "jumping"},
Objects: []string{"teddy-bears", "diamonds", "buckets", "boxes"},
}
generator := password.NewGenerator(dictionary, time.Now().UnixNano())
pass := generator.Generate()
fmt.Printf("%s", pass)
getWords("Verb")
}
As you can see, what I am trying to do is use WordNik API to request Adverbs, nouns, etc and then make a dictionary based off those words to generate a password. I am horrible with arrays and handling data.
As pointed out in the comments you need to export the fields so they can be unmarshaled
The encoding/json package relies on reflection and since it is in another package, it cannot access unexported fields. ( in go, fields, methods or functions that start with a small letter are unexported whereas with capital letters they are exported )
Then your example json does not contain the WordList at all, so what you want is unmarshaling directly into the array of Words. Also a words object only consists of Id and Word and not of arrays themselves.
type Words struct {
Id int64
Word string
}
func main() {
....
var words []Words
// you don't need to read the whole body first, you can decode in the same turn like this
err := json.NewDecoder(req.Body).Decode(&words)
if nil != err {
log.Fatal(err)
}
...
}
Another very important thing is that you should not ignore errors. This would have helped you to debug the issue. ( What I mean is _ = json.Unmarshal)
As for beginning with go, you could implement a simple test to see if your code works as intended.
https://play.golang.org/p/nuz9uXdka5S < check this working example for reference.

Resources