How to unmarshal/parse a request using Go correctly? - go

How to parse a curl request response in Golang correctly? I have tried the following where I send a request to an api and its response is:
{
"Certificates": [
{
"Name": "some-name.here",
.......
}
],
"DataRange": "Certificates 1 - 1",
"TotalCount": 1
}
Now I want to use the Name in the Certificates in a string variable. i.e match. Before even I get to the looping through of the response, I get the error: json: cannot unmarshal object into Go value of type []program.Item. This error is coming from json.Unmarshal function where I pass my []bytes and the struct to use the bytes. Am I doing this correctly?
type Item struct {
Certificates []CertificatesInfo
}
type CertificatesInfo struct {
Name string
}
func main() {
url := .....
req, err := http.NewRequest("GET", url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
// handle err
continue
}
defer resp.Body.Close()
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
continue
}
var ItemInfo []Item
if err := json.Unmarshal(bodyBytes, &ItemInfo); err != nil {
return nil, fmt.Errorf("failed to parse %v", err)
}
for _, infos := range ItemInfo {
for _, names := range infos.Certificates {
infraId:= names.Name
match:= display_name
}
}
}

There's a mismatch between your sample JSON response data (a single object containing a list of certificates), and your code, which is expecting a list of objects, each of which contain a list of certificates).
Assuming that your JSON example is correct, this bit:
var ItemInfo []Item
if err := json.Unmarshal(bodyBytes, &ItemInfo); err != nil {
. . .
should probably be
var itemInfo Item
if err := json.Unmarshal(bodyBytes, &ItemInfo); err != nil {
. . .
Works on my machine:
https://go.dev/play/p/8FqIif1zzsQ
package main
import (
"encoding/json"
"fmt"
)
func main() {
resp := Item{}
err := json.Unmarshal([]byte(data), &resp)
if err != nil {
fmt.Print(err)
panic(err)
}
fmt.Printf("resp.DataRange: %s\n", resp.DataRange)
fmt.Printf("resp.TotalCount: %d\n", resp.TotalCount)
fmt.Printf("len(resp.Certificates: %d\n", len(resp.Certificates))
for i, c := range resp.Certificates {
fmt.Printf("resp.Certificates[%d].Name: %s\n", i, c.Name)
}
}
type Item struct {
Certificates []CertificatesInfo
DataRange string
TotalCount int
}
type CertificatesInfo struct {
Name string
}
const data = `
{
"Certificates": [
{ "Name": "alpha" },
{ "Name": "bravo" },
{ "Name": "charlie" }
],
"DataRange": "Certificates 1 - 1",
"TotalCount": 3
}
`

Related

Go elasticsearch bulk insert

I have been unable to solve the problem into elasticsearch Bulk method for several days, since I am not strong in Go and started learning it not so long ago, while executing the code :
package main
import (
"bytes"
json "encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type BulkInsertMetaData struct {
Index []BulkInsertIndex `json:"index"`
}
type BulkInsertIndex struct {
Index string `json:"_index"`
ID string `json:"_id"`
}
type BulInsertData struct {
Url string `json:"url"`
}
func main() {
dataMeta := BulkInsertMetaData{
Index: []BulkInsertIndex{
{
Index: "Test",
ID: "1234567890",
},
},
}
data := BulInsertData{
Url: "http://XXXX.XX",
}
TojsBulInsertData, _ := json.Marshal(data)
TojsBulkInsertMetaData, _ := json.Marshal(dataMeta)
BulkMetaData := bytes.NewBuffer(append(TojsBulkInsertMetaData, []byte("\n")...))
BulkData := bytes.NewBuffer(append(TojsBulInsertData, []byte("\n")...))
log.Println(BulkMetaData)
log.Println(BulkData)
respMetaData, err := http.Post("http://127.0.0.1:9200/_bulk", "application/json", BulkMetaData)
if err != nil {
log.Println(err)
}
body, err := ioutil.ReadAll(respMetaData.Body)
if err != nil {
log.Println(err)
}
fmt.Println(string(body))
respBulkData, err := http.Post("http://127.0.0.1:9200/_bulk", "application/json", BulkData)
if err != nil {
log.Println(err)
}
body2, err := ioutil.ReadAll(respBulkData.Body)
if err != nil {
log.Println(err)
}
fmt.Println(string(body2))
}
but i get an error:
2022/02/09 14:37:02 {"index":[{"_index":"Test","_id":"1234567890"}]}
2022/02/09 14:37:02 {"url":"http://XXXX.XX"}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"},"status":400}
please help and explain what I'm doing wrong, I searched the Internet for the answer to my question but did not find
I test insert when using REST client passes without problems
BulkMetaData should be {"index":{"_index":"Test","_id":"1234567890"}} (without []) and it should be sent to /_bulk together with BulkData, as a single payload:
{"index":{"_index":"Test","_id":"1234567890"}}
{"url":"http://XXXX.XX"}
Sorry for kinda necroing but I also recently needed to design a Bulk connector in our codebase and the fact that there are NO NDJSON encoder/decoders out on the web is appalling. Here is my implementation:
func ParseToNDJson(data []map[string]interface{}, dst *bytes.Buffer) error {
enc := json.NewEncoder(dst)
for _, element := range data {
if err := enc.Encode(element); err != nil {
if err != io.EOF {
return fmt.Errorf("failed to parse NDJSON: %v", err)
}
break
}
}
return nil
}
Driver code to test:
func main() {
var b bytes.Buffer
var data []map[string]interface{}
// pointless data generation...
for i, name := range []string{"greg", "sergey", "alex"} {
data = append(data, map[string]interface{}{name: i})
}
if err := ParseToNDJson(query, &body); err != nil {
return nil, fmt.Errorf("error encoding request: %s", err)
}
res, err := esapi.BulkRequest{
Index: "tasks",
Body: strings.NewReader(body.String()),
}.Do(ctx, q.es)
Hope this helps someone

how to fetch data from another api with body raw json

i have default body raw json and want to paste it into a struct so it can fetch data automatically and save it into a struct
Body Raw Json
{
"jsonrpc": "2.0",
"params": {
}
}
Response from api
{
"jsonrpc": "2.0",
"id": null,
"result": {
"status": 200,
"response": [
{
"service_id": 1129,
"service_name": "Adobe Illustrator",
"service_category_id": 28,
"service_category_name": "License Software",
"service_type_id": 25,
"service_type_name": "Software",
"create_date": "2020-03-09 03:47:44"
},
],
"message": "Done All User Returned"
}
}
I want to put it in the repository file so I can get data automatically
Repo file
// Get request
resp, err := http.Get("look at API Response Example")
if err != nil {
fmt.Println("No response from request")
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body) // response body is []byte
if err != nil {
return err
}
// data that already fetch accomadate to struct
var result models.OdooRequest
if err := json.Unmarshal(body, &result); err != nil {
fmt.Println("Can not unmarshal JSON")
}
for _, rec := range result.Response {
fmt.Println(rec.ServiceName)
}
return err
after being fetched then accommodated into a struct
struct
type OdooRequest struct {
Response []UpsertFromOdooServices
}
Sure, here's a rough way to make that request and read the response:
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)
type OdooRequest struct {
Result struct {
Status int `json:"status"`
Response []struct {
ServiceID int `json:"service_id"`
ServiceName string `json:"service_name"`
ServiceCategoryID int `json:"service_category_id"`
ServiceCategoryName string `json:"service_category_name"`
ServiceTypeID int `json:"service_type_id"`
ServiceTypeName string `json:"service_type_name"`
CreateDate string `json:"create_date"`
} `json:"response"`
Message string `json:"message"`
} `json:"result"`
}
func main() {
if err := run(); err != nil {
panic(err)
}
}
func run() error {
resp, err := http.Post(
"ADD_URL_HERE",
"application/json",
bytes.NewBufferString(`{"jsonrpc": "2.0","params": {}}`),
)
if err != nil {
return err
}
defer resp.Body.Close()
var odooResp OdooRequest
if err := json.NewDecoder(resp.Body).Decode(&odooResp); err != nil {
return err
}
for _, rec := range odooResp.Result.Response {
fmt.Println(rec.ServiceName)
}
return nil
}

Go - How can I make sure there aren't empty fields being sent in this code via gRPC?

I am writing some code to send logs with gRPC to a server. The problem I'm having is that the logs are different and not all have every field, but . I'm wondering how I can set only the fields they do have inside logpb.Log short of creating several different types of logs in the proto file?
This is all one methods, but the formatting on StackOverflow isn't with it.
The code:
func (*Client) SendWithgRPC(entry *log.Entry) error {
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
if err != nil {
log.Fatalf("Could not connect: %v", err)
}
defer conn.Close()
c := logpb.NewLogServiceClient(conn)
stream, err := c.SendLogs(context.Background())
if err != nil {
log.Fatalf("Error while calling SendLogs: %v", err)
}
logFields := entry.Data
category := ""
debug_id := ""
request_id := ""
tipi := ""
uri := ""
var user_id int32 = 0
ip := ""
if logFields["category"] != nil {
category = logFields["category"].(string)
}
if logFields["debug_id"] != nil {
debug_id = logFields["debug_id"].(string)
}
if logFields["request_id"] != nil {
request_id = logFields["request_id"].(string)
}
if logFields["type"] != nil {
tipi = logFields["type"].(string)
}
if logFields["uri"] != nil {
uri = logFields["uri"].(string)
}
if logFields["user_id"] != nil {
user_id = int32(logFields["user_id"].(int))
}
if logFields["ip"] != nil {
ip = logFields["ip"].(string)
}
logEntry := &logpb.Log{
Time: entry.Time.String(),
Level: entry.Level.String(),
Msg: entry.Message,
Category: category,
DebugId: debug_id,
Ip: ip,
RequestId: request_id,
Type: tipi,
Uri: uri,
Id: user_id,
}
logs := []*logpb.Log{logEntry}
logarray := []*logpb.SendLogsRequest{
&logpb.SendLogsRequest{
Logs: logs,
},
}
fmt.Print(logarray)
for _, req := range logarray {
stream.Send(req)
}
_, err = stream.CloseAndRecv()
if err != nil {
fmt.Printf("Error with response: %v", err)
}
return nil
}
You can use something like https://github.com/go-playground/validator
It offers validation using tags. There is a lot more to this lib apart from basic validation. Do check it out.
Sample example https://github.com/go-playground/validator/blob/master/_examples/simple/main.go
As easy as it gets :
package main
import (
"fmt"
"github.com/go-playground/validator/v10"
)
type Name struct {
First string `validate:"required"`
Last string
}
func main() {
validName := Name{First: "John"} // Last name is empty and is not required
invalidName := Name{Last: "Doe"} // first name is empty but is required
validate := validator.New()
err := validate.Struct(invalidName)
if err != nil {
fmt.Println("invalid name struct caused error")
}
err = validate.Struct(validName)
if err != nil {
fmt.Println("valid name struct caused error")
}
fmt.Println("GoodBye")
}
The output of above :
invalid name struct caused error
GoodBye
working code on go-playground : https://play.golang.org/p/S4Yj1yr16t2

How to save multi JSON API response to MongoDB using Golang

I am fetching data from an api which comes in this format (10+ individual orders):
{
"orders": [
{
"orderId": "2738183220",
"dateTimeOrderPlaced": "2019-12-11T20:19:19+01:00",
"orderItems": [
{
"orderItemId": "BFC0000347027892",
"ean": "8720165489992",
"cancelRequest": false,
"quantity": 1
}
]
},
{
"orderId": "2737526230",
"dateTimeOrderPlaced": "2019-12-11T17:18:42+01:00",
"orderItems": [
{
"orderItemId": "BFC0000346926381",
"ean": "8719326639581",
"cancelRequest": false,
"quantity": 1
}
]
}
]
}
I have this struct for that:
type OpenOrder struct {
Orders []struct {
OrderID string `json:"orderId"`
DateTimeOrderPlaced time.Time `json:"dateTimeOrderPlaced"`
OrderItems []struct {
OrderItemID string `json:"orderItemId"`
Ean string `json:"ean"`
CancelRequest bool `json:"cancelRequest"`
Quantity int `json:"quantity"`
} `json:"orderItems"`
} `json:"orders"`
}
But I am confused after looking at this for too long. How can I save this into MongoDB with Golang where so it will get saved as single documents? So one document is:
{
"orderId": "2738183220",
"dateTimeOrderPlaced": "2019-12-11T20:19:19+01:00",
"orderItems": [
{
"orderItemId": "BFC0000347027892",
"ean": "8720165489992",
"cancelRequest": false,
"quantity": 1
}
]
}
I wrote this code:
func GetOpenOrder(x string) {
apiURL := "https://api.bol.com/retailer/orders?fulfilment-method=" + x
req, err := http.NewRequest("GET", apiURL, nil)
req.Header.Add("Authorization", "Bearer "+auth.GetToken())
req.Header.Add("Accept", "application/vnd.retailer.v3+json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
// Read the response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(body, &o)
if err != nil {
fmt.Println(string(body))
}
c := auth.GetClient()
collection := c.Database("goprac").Collection("openorders")
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
result, _ := collection.InsertOne(ctx, o)
fmt.Println(result)
}
But right now the whole JSON response gets saved in one document. Can someone please point me in the correct direction? I have been searching SO for the last hour and found some similar questions, but still did not get the answer.

Index out of Range with array of structs in Go

I am new to Go so hopefully I'm making myself clear with this issue I'm having. My problem is that I am trying to iterate over an array of structs but I keep running into an index out of range issue. For the purposes of this problem, I have already verified that my array is not empty but that it in fact does contain at least one Services struct and file_content is the string that contains my valid JSON
Here is the snippet of code that represents the problem I'm having:
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
err_json := json.Unmarshal(file_content, &ServiceConf)
for _, s := range ServiceConf.Services {
log.Println(s)
}
So every time I run my code I get:
2014/03/14 18:19:53 http: panic serving [::1]:65448: runtime error: index out of range
{
"services" : [
{
"name": "translation",
"command": "to german",
"request": {
"key": "XXX",
"url": "https://www.googleapis.com/language/translate/v2?"
}
}
]
}
If you're interested in the complete source file:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
)
type SlackResponse struct {
token string
team_id string
channel_id string
channel_name string
timestamp string
user_id string
user_name string
text string
}
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
func main() {
content, err_read := ioutil.ReadFile("config.ini")
if err_read != nil {
log.Println("Could not read config")
return
}
log.Println(string(content))
err_json := json.Unmarshal(content, &ServiceConf)
if err_json != nil {
log.Println(err_json)
}
http.HandleFunc("/", handler)
http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
slack_response := SlackResponse{
r.FormValue("token"),
r.FormValue("team_id"),
r.FormValue("channel_id"),
r.FormValue("channel_name"),
r.FormValue("timestamp"),
r.FormValue("user_id"),
r.FormValue("user_name"),
r.FormValue("text"),
}
// log.Println(ServiceConf.Services[0].Request["key"])
// loop through services to find command phrases
for _, s := range ServiceConf.Services {
log.Println(s)
}
if slack_response.user_name == "slackbot" {
return
}
// fmt.Fprintf(w, "{ \"text\": \"Master %s! You said: '%s'\" }", slack_response.user_name, slack_response.text)
content, err := getContent("https://www.googleapis.com/language/translate/v2?key=&source=en&target=de&q=" + url.QueryEscape(slack_response.text))
if err != nil {
fmt.Fprintf(w, "{ \"text\": \"Huh?!\" }")
} else {
type trans struct {
Data struct {
Translations []struct {
TranslatedText string `json:"translatedText"`
} `json:"translations"`
} `json:"data"`
}
f := trans{}
err := json.Unmarshal(content, &f)
if err != nil {
log.Println(err)
}
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }", f.Data.Translations[0].TranslatedText)
}
}
// array of bytes if retrieved successfully.
func getContent(url string) ([]byte, error) {
// Build the request
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
// Send the request via a client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
// Defer the closing of the body
defer resp.Body.Close()
// Read the content into a byte array
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
// At this point we're done - simply return the bytes
return body, nil
}
Here is the stack trace:
2014/03/21 23:21:29 http: panic serving [::1]:59508: runtime error: index out of range
goroutine 3 [running]:
net/http.funcĀ·009()
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x215f80, 0x4b6537)
/usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/panic.c:248 +0x106
main.handler(0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/Users/et/src/go/src/github.com/etdebruin/gojacques/main.go:100 +0x81b
net/http.HandlerFunc.ServeHTTP(0x2cbc60, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc21001e5d0, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1496 +0x163
net/http.serverHandler.ServeHTTP(0xc21001f500, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc210058300)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1644 +0x28b
The error comes from this line
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }",
f.Data.Translations[0].TranslatedText)
So you didn't get any Translations back - that array is empty.
You might want to check resp.Status to see if an error was returned. This isn't returned as an error - you need to check it yourself.

Resources