HI i wanted to ask the community since I am not an expert in GO how do I add \n at the end of the request so as not to get an error when inserting a large amount of data into elasticsearch bulk
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}
apply the code what I wrote - just say this is a test
package main
import (
"bytes"
json "encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type ElkInsertIndex struct {
Index string `json:"_index"`
ID string `json:"_id"`
}
type ElkBulkInsert struct {
Index []ElkInsertIndex `json:"index"`
}
type ElkUrl struct {
Url string `json:"url"`
}
func main() {
data := ElkBulkInsert{
Index: []ElkInsertIndex{
{
Index: "shut_url",
ID: "FFFFFFFFFFFFFFFF",
},
},
}
js, err := json.Marshal(data)
testBulk := bytes.NewBuffer(js)
resp1, err := http.Post("http://127.0.0:9200/_bulk", "application/json", testBulk)
if err != nil {
log.Println(err)
}
body, err := ioutil.ReadAll(resp1.Body)
if err != nil {
log.Println(err)
}
fmt.Println(string(body))
}
Related
I am sending the data to the API like following:
{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}
and i am getting following while printing :
map[after:map[amount:811 id:123 status:Hi ] key:[70]]
Is there any way to print individual field like this??
amount::800
id:123
status:Hi
The code:
package main
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"strings"
)
var (
PORT = ":8080"
)
func main() {
fmt.Println("In Main")
http.HandleFunc("/", changedData)
http.ListenAndServe(PORT, nil)
}
type Data struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
type mark map[string]interface{}
func changedData(w http.ResponseWriter, r *http.Request) {
fmt.Println("Coming From API")
reqBody, _ := ioutil.ReadAll(r.Body)
fmt.Println("Data coming from API ", string(reqBody))
digit := json.NewDecoder(strings.NewReader(string(reqBody)))
for digit.More() {
var result mark
err := digit.Decode(&result)
if err != nil {
if err != io.EOF {
log.Fatal(err)
}
break
}
fmt.Println("final_data ", result)
}
}
Decode to a Go type that matches the structure of the JSON document. You declared a type for the "after" field. Wrap that type with a struct to match the document.
func changedData(w http.ResponseWriter, r *http.Request) {
var v struct{ After Data }
err := json.NewDecoder(r.Body).Decode(&v)
if err != nil {
http.Error(w, "bad request", 400)
return
}
fmt.Printf("final_data: %#v", v.After)
}
Playground example.
I think you can define a struct type if you know the JSON file format or if the JSON format is predefined. As far as I know that mostly using interface{} is a way when you don't know the JSON format or there is no predefined format of the JSON. If you define a struct type and use it while unmarshaling the JSON to struct, you can access the variables by typing like data.Id or data.Status.
Here's an example code:
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
AfterData After `json:"after"`
Key []int `json:"key"`
}
type After struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
func main() {
j := []byte(`{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}`)
var data *Data
err := json.Unmarshal(j, &data)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(data.AfterData)
fmt.Println(data.AfterData.Id)
fmt.Println(data.AfterData.Amount)
fmt.Println(data.AfterData.Status)
}
Output will be
{123 811 Hi}
123
811
Hi
Go Playground
In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}
Can you see why json values dont get saved:
Update: And if you would like to explain why this is downgraded as "Off topic"?
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type UsedInContext struct {
UsedIn string `json:"usedin"`
ContextName string `json:"contextname"`
}
type ContextData struct {
usedInContext []UsedInContext `json:"usedincontext"`
}
func saveData() {
var jsonBlob = []byte(`{
"usedincontext":
[
{"usedin":"local", "contextname":"desktop"}
]
}`)
usedInContext := UsedInContext{}
err := json.Unmarshal(jsonBlob, &usedInContext)
if err != nil {
}
usedInContextJson, _ := json.Marshal(usedInContext)
err = ioutil.WriteFile("data.json", usedInContextJson, 0644)
fmt.Printf("%+v", usedInContext)
}
I get the following:
{"usedin":"","contextname":""}
You unmarshal your document to the type UsedInContext, while it matches the schema for ContextData:
type ContextData struct {
UsedInContext []UsedInContext `json:"usedincontext"` // exported
}
var data ContextData
json.Unmarshal(jsonBlob, &data)
fmt.Printf("%+v", data)
I'm already posting a file and some params, but now I need to send nested params with a struct and I don't know where to use that (I'm new in Go).
This is what I have: https://play.golang.org/p/L4qx6AZhUO
Now, I'm creating this structs:
type S3 struct {
Accesskeyid string
Secretaccesskey string
Bucket string
}
type Test struct {
Outputformat string
Wait string
Output S3
File []byte
}
And I would like to send the Test struct WITH the file. Any ideas?
Thanks!
Okay, So given what you've told me and what little information I have with your example, I might do something like this.
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)
type S3 struct {
Accesskeyid string
Secretaccesskey string
Bucket string
}
type Test struct {
Outputformat string
Wait string
Output S3
File []byte
}
func main() {
myStrut := Test{
Outputformat: "Json",
Wait: "Some Time",
Output: S3{
Accesskeyid: "my id",
Secretaccesskey: "secret id",
Bucket: "east",
},
File: []byte(`some bytes`),
}
jsonValue, err := json.Marshal(myStrut)
if err != nil {
panic(err)
}
fmt.Printf("Test showing that data was marshalled\n %q\n", jsonValue)
resp, err := http.Post("some url", "application/json", bytes.NewBuffer(jsonValue))
if err != nil {
panic(err)
}
fmt.Println(resp.Status)
}
Now from what i gleamed in the comments you might be also having trouble opening the file as a byte array to assign to your struct. Here's an example you can use you help you open the file as a byte array so that you can assign those bytes to your struct.
package main
import (
"fmt"
"io/ioutil"
)
func main() {
//An example of how to open a file and turn it into bytes for your struct
byteArray, err := ioutil.ReadFile("input.txt")
if err != nil {
panic(err)
}
fmt.Println(byteArray)
}
I'm trying to decode a json I get. Here's an example json I get:
{"response":"1","number":"1234","id":nil}
Here's my struct:
type AutoGenerated struct {
Response string `json:"response"`
Number string `json:"number"`
ID interface{} `json:"id"`
}
I use the decode function in encode/json. What Am I getting wrong? ID has the chance to be both a string or a nil value.
Here's me exact error incase it helps.
panic: EOF
Without you showing how you're doing it, I think the best answer is to show you how to do it.
package main
import (
"fmt"
"log"
"encoding/json"
)
func main() {
j := []byte(`{"response":"1","number":"1234","id":null}`)
data := AutoGenerated{}
err := json.Unmarshal(j, &data)
if err != nil {
log.Println(err.Error())
return
}
fmt.Println(data)
}
type AutoGenerated struct {
Response string `json:"response"`
Number string `json:"number"`
ID interface{} `json:"id"`
}
The JSON string you put here is invalid. You can find this code sample for reference.
If you're going to set the id field to nil, just don't put it in the JSON string.
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
type AutoGenerated struct {
Response string `json:"response"`
Number string `json:"number"`
ID interface{} `json:"id"`
}
func main() {
jsonStream := `
{ "response": "1", "number": "1234" }
{ "response": "1", "number": "1234", "id": "nil" }
`
decoder := json.NewDecoder(strings.NewReader(jsonStream))
for {
var m AutoGenerated
if err := decoder.Decode(&m); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(m)
}
}
The output of the program is:
{1 1234 <nil>}
{1 1234 nil}