Index out of Range with array of structs in Go - go

I am new to Go so hopefully I'm making myself clear with this issue I'm having. My problem is that I am trying to iterate over an array of structs but I keep running into an index out of range issue. For the purposes of this problem, I have already verified that my array is not empty but that it in fact does contain at least one Services struct and file_content is the string that contains my valid JSON
Here is the snippet of code that represents the problem I'm having:
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
err_json := json.Unmarshal(file_content, &ServiceConf)
for _, s := range ServiceConf.Services {
log.Println(s)
}
So every time I run my code I get:
2014/03/14 18:19:53 http: panic serving [::1]:65448: runtime error: index out of range
{
"services" : [
{
"name": "translation",
"command": "to german",
"request": {
"key": "XXX",
"url": "https://www.googleapis.com/language/translate/v2?"
}
}
]
}
If you're interested in the complete source file:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
)
type SlackResponse struct {
token string
team_id string
channel_id string
channel_name string
timestamp string
user_id string
user_name string
text string
}
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
func main() {
content, err_read := ioutil.ReadFile("config.ini")
if err_read != nil {
log.Println("Could not read config")
return
}
log.Println(string(content))
err_json := json.Unmarshal(content, &ServiceConf)
if err_json != nil {
log.Println(err_json)
}
http.HandleFunc("/", handler)
http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
slack_response := SlackResponse{
r.FormValue("token"),
r.FormValue("team_id"),
r.FormValue("channel_id"),
r.FormValue("channel_name"),
r.FormValue("timestamp"),
r.FormValue("user_id"),
r.FormValue("user_name"),
r.FormValue("text"),
}
// log.Println(ServiceConf.Services[0].Request["key"])
// loop through services to find command phrases
for _, s := range ServiceConf.Services {
log.Println(s)
}
if slack_response.user_name == "slackbot" {
return
}
// fmt.Fprintf(w, "{ \"text\": \"Master %s! You said: '%s'\" }", slack_response.user_name, slack_response.text)
content, err := getContent("https://www.googleapis.com/language/translate/v2?key=&source=en&target=de&q=" + url.QueryEscape(slack_response.text))
if err != nil {
fmt.Fprintf(w, "{ \"text\": \"Huh?!\" }")
} else {
type trans struct {
Data struct {
Translations []struct {
TranslatedText string `json:"translatedText"`
} `json:"translations"`
} `json:"data"`
}
f := trans{}
err := json.Unmarshal(content, &f)
if err != nil {
log.Println(err)
}
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }", f.Data.Translations[0].TranslatedText)
}
}
// array of bytes if retrieved successfully.
func getContent(url string) ([]byte, error) {
// Build the request
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
// Send the request via a client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
// Defer the closing of the body
defer resp.Body.Close()
// Read the content into a byte array
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
// At this point we're done - simply return the bytes
return body, nil
}
Here is the stack trace:
2014/03/21 23:21:29 http: panic serving [::1]:59508: runtime error: index out of range
goroutine 3 [running]:
net/http.func·009()
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x215f80, 0x4b6537)
/usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/panic.c:248 +0x106
main.handler(0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/Users/et/src/go/src/github.com/etdebruin/gojacques/main.go:100 +0x81b
net/http.HandlerFunc.ServeHTTP(0x2cbc60, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc21001e5d0, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1496 +0x163
net/http.serverHandler.ServeHTTP(0xc21001f500, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc210058300)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1644 +0x28b

The error comes from this line
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }",
f.Data.Translations[0].TranslatedText)
So you didn't get any Translations back - that array is empty.
You might want to check resp.Status to see if an error was returned. This isn't returned as an error - you need to check it yourself.

Related

How to unmarshal/parse a request using Go correctly?

How to parse a curl request response in Golang correctly? I have tried the following where I send a request to an api and its response is:
{
"Certificates": [
{
"Name": "some-name.here",
.......
}
],
"DataRange": "Certificates 1 - 1",
"TotalCount": 1
}
Now I want to use the Name in the Certificates in a string variable. i.e match. Before even I get to the looping through of the response, I get the error: json: cannot unmarshal object into Go value of type []program.Item. This error is coming from json.Unmarshal function where I pass my []bytes and the struct to use the bytes. Am I doing this correctly?
type Item struct {
Certificates []CertificatesInfo
}
type CertificatesInfo struct {
Name string
}
func main() {
url := .....
req, err := http.NewRequest("GET", url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
// handle err
continue
}
defer resp.Body.Close()
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
continue
}
var ItemInfo []Item
if err := json.Unmarshal(bodyBytes, &ItemInfo); err != nil {
return nil, fmt.Errorf("failed to parse %v", err)
}
for _, infos := range ItemInfo {
for _, names := range infos.Certificates {
infraId:= names.Name
match:= display_name
}
}
}
There's a mismatch between your sample JSON response data (a single object containing a list of certificates), and your code, which is expecting a list of objects, each of which contain a list of certificates).
Assuming that your JSON example is correct, this bit:
var ItemInfo []Item
if err := json.Unmarshal(bodyBytes, &ItemInfo); err != nil {
. . .
should probably be
var itemInfo Item
if err := json.Unmarshal(bodyBytes, &ItemInfo); err != nil {
. . .
Works on my machine:
https://go.dev/play/p/8FqIif1zzsQ
package main
import (
"encoding/json"
"fmt"
)
func main() {
resp := Item{}
err := json.Unmarshal([]byte(data), &resp)
if err != nil {
fmt.Print(err)
panic(err)
}
fmt.Printf("resp.DataRange: %s\n", resp.DataRange)
fmt.Printf("resp.TotalCount: %d\n", resp.TotalCount)
fmt.Printf("len(resp.Certificates: %d\n", len(resp.Certificates))
for i, c := range resp.Certificates {
fmt.Printf("resp.Certificates[%d].Name: %s\n", i, c.Name)
}
}
type Item struct {
Certificates []CertificatesInfo
DataRange string
TotalCount int
}
type CertificatesInfo struct {
Name string
}
const data = `
{
"Certificates": [
{ "Name": "alpha" },
{ "Name": "bravo" },
{ "Name": "charlie" }
],
"DataRange": "Certificates 1 - 1",
"TotalCount": 3
}
`

Go elasticsearch bulk insert

I have been unable to solve the problem into elasticsearch Bulk method for several days, since I am not strong in Go and started learning it not so long ago, while executing the code :
package main
import (
"bytes"
json "encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type BulkInsertMetaData struct {
Index []BulkInsertIndex `json:"index"`
}
type BulkInsertIndex struct {
Index string `json:"_index"`
ID string `json:"_id"`
}
type BulInsertData struct {
Url string `json:"url"`
}
func main() {
dataMeta := BulkInsertMetaData{
Index: []BulkInsertIndex{
{
Index: "Test",
ID: "1234567890",
},
},
}
data := BulInsertData{
Url: "http://XXXX.XX",
}
TojsBulInsertData, _ := json.Marshal(data)
TojsBulkInsertMetaData, _ := json.Marshal(dataMeta)
BulkMetaData := bytes.NewBuffer(append(TojsBulkInsertMetaData, []byte("\n")...))
BulkData := bytes.NewBuffer(append(TojsBulInsertData, []byte("\n")...))
log.Println(BulkMetaData)
log.Println(BulkData)
respMetaData, err := http.Post("http://127.0.0.1:9200/_bulk", "application/json", BulkMetaData)
if err != nil {
log.Println(err)
}
body, err := ioutil.ReadAll(respMetaData.Body)
if err != nil {
log.Println(err)
}
fmt.Println(string(body))
respBulkData, err := http.Post("http://127.0.0.1:9200/_bulk", "application/json", BulkData)
if err != nil {
log.Println(err)
}
body2, err := ioutil.ReadAll(respBulkData.Body)
if err != nil {
log.Println(err)
}
fmt.Println(string(body2))
}
but i get an error:
2022/02/09 14:37:02 {"index":[{"_index":"Test","_id":"1234567890"}]}
2022/02/09 14:37:02 {"url":"http://XXXX.XX"}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"},"status":400}
please help and explain what I'm doing wrong, I searched the Internet for the answer to my question but did not find
I test insert when using REST client passes without problems
BulkMetaData should be {"index":{"_index":"Test","_id":"1234567890"}} (without []) and it should be sent to /_bulk together with BulkData, as a single payload:
{"index":{"_index":"Test","_id":"1234567890"}}
{"url":"http://XXXX.XX"}
Sorry for kinda necroing but I also recently needed to design a Bulk connector in our codebase and the fact that there are NO NDJSON encoder/decoders out on the web is appalling. Here is my implementation:
func ParseToNDJson(data []map[string]interface{}, dst *bytes.Buffer) error {
enc := json.NewEncoder(dst)
for _, element := range data {
if err := enc.Encode(element); err != nil {
if err != io.EOF {
return fmt.Errorf("failed to parse NDJSON: %v", err)
}
break
}
}
return nil
}
Driver code to test:
func main() {
var b bytes.Buffer
var data []map[string]interface{}
// pointless data generation...
for i, name := range []string{"greg", "sergey", "alex"} {
data = append(data, map[string]interface{}{name: i})
}
if err := ParseToNDJson(query, &body); err != nil {
return nil, fmt.Errorf("error encoding request: %s", err)
}
res, err := esapi.BulkRequest{
Index: "tasks",
Body: strings.NewReader(body.String()),
}.Do(ctx, q.es)
Hope this helps someone

how to fetch data from another api with body raw json

i have default body raw json and want to paste it into a struct so it can fetch data automatically and save it into a struct
Body Raw Json
{
"jsonrpc": "2.0",
"params": {
}
}
Response from api
{
"jsonrpc": "2.0",
"id": null,
"result": {
"status": 200,
"response": [
{
"service_id": 1129,
"service_name": "Adobe Illustrator",
"service_category_id": 28,
"service_category_name": "License Software",
"service_type_id": 25,
"service_type_name": "Software",
"create_date": "2020-03-09 03:47:44"
},
],
"message": "Done All User Returned"
}
}
I want to put it in the repository file so I can get data automatically
Repo file
// Get request
resp, err := http.Get("look at API Response Example")
if err != nil {
fmt.Println("No response from request")
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body) // response body is []byte
if err != nil {
return err
}
// data that already fetch accomadate to struct
var result models.OdooRequest
if err := json.Unmarshal(body, &result); err != nil {
fmt.Println("Can not unmarshal JSON")
}
for _, rec := range result.Response {
fmt.Println(rec.ServiceName)
}
return err
after being fetched then accommodated into a struct
struct
type OdooRequest struct {
Response []UpsertFromOdooServices
}
Sure, here's a rough way to make that request and read the response:
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)
type OdooRequest struct {
Result struct {
Status int `json:"status"`
Response []struct {
ServiceID int `json:"service_id"`
ServiceName string `json:"service_name"`
ServiceCategoryID int `json:"service_category_id"`
ServiceCategoryName string `json:"service_category_name"`
ServiceTypeID int `json:"service_type_id"`
ServiceTypeName string `json:"service_type_name"`
CreateDate string `json:"create_date"`
} `json:"response"`
Message string `json:"message"`
} `json:"result"`
}
func main() {
if err := run(); err != nil {
panic(err)
}
}
func run() error {
resp, err := http.Post(
"ADD_URL_HERE",
"application/json",
bytes.NewBufferString(`{"jsonrpc": "2.0","params": {}}`),
)
if err != nil {
return err
}
defer resp.Body.Close()
var odooResp OdooRequest
if err := json.NewDecoder(resp.Body).Decode(&odooResp); err != nil {
return err
}
for _, rec := range odooResp.Result.Response {
fmt.Println(rec.ServiceName)
}
return nil
}

variable is empty but later has a value

I'm trying to develop a Terraform provider but I have a problem of the first request body. Here is the code:
type Body struct {
id string
}
func resourceServerCreate(d *schema.ResourceData, m interface{}) error {
key := d.Get("key").(string)
token := d.Get("token").(string)
workspace_name := d.Get("workspace_name").(string)
board_name := d.Get("board_name").(string)
resp, err := http.Post("https://api.trello.com/1/organizations?key="+key+"&token="+token+"&displayName="+workspace_name,"application/json",nil)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
//lettura body.
body := new(Body)
json.NewDecoder(resp.Body).Decode(body)
log.Println("[ORCA MADONNA] il log funzia "+body.id)
d.Set("board_id",body.id)
resp1, err1 := http.Post("https://api.trello.com/1/boards?key="+key+"&token="+token+"&idOrganization="+body.id+"&=&name="+board_name,"application/json",nil)
if err1 != nil {
log.Fatalln(resp1)
}
defer resp1.Body.Close()
d.SetId(board_name)
return resourceServerRead(d, m)
}
In the log is empty, but the second call have it and work fine. How is it possible?
Go doesn't force you to check error responses, therefore it's easy to make silly mistakes. Had you checked the return value from Decode(), you would have immediately discovered a problem.
err := json.NewDecoder(resp.Body).Decode(body)
if err != nil {
log.Fatal("Decode error: ", err)
}
Decode error: json: Unmarshal(non-pointer main.Body)
So your most immediate fix is to use & to pass a pointer to Decode():
json.NewDecoder(resp.Body).Decode(&body)
Also of note, some programming editors will highlight this mistake for you:
Here's a working demonstration, including a corrected Body structure as described at json.Marshal(struct) returns “{}”:
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
"net/http"
"time"
)
type JSON = map[string]interface{}
type JSONArray = []interface{}
func ErrFatal(err error, msg string) {
if err != nil {
log.Fatal(msg+": ", err)
}
}
func handleTestRequest(w http.ResponseWriter, req *http.Request) {
w.Write(([]byte)("{\"id\":\"yourid\"}"))
}
func launchTestServer() {
http.HandleFunc("/", handleTestRequest)
go http.ListenAndServe(":8080", nil)
time.Sleep(1 * time.Second) // allow server to get started
}
// Medium: "Don’t use Go’s default HTTP client (in production)"
var restClient = &http.Client{
Timeout: time.Second * 10,
}
func DoREST(method, url string, headers, payload JSON) *http.Response {
requestPayload, err := json.Marshal(payload)
ErrFatal(err, "json.Marshal(payload")
request, err := http.NewRequest(method, url, bytes.NewBuffer(requestPayload))
ErrFatal(err, "NewRequest "+method+" "+url)
for k, v := range headers {
request.Header.Add(k, v.(string))
}
response, err := restClient.Do(request)
ErrFatal(err, "DoRest client.Do")
return response
}
type Body struct {
Id string `json:"id"`
}
func clientDemo() {
response := DoREST("POST", "http://localhost:8080", JSON{}, JSON{})
defer response.Body.Close()
var body Body
err := json.NewDecoder(response.Body).Decode(&body)
ErrFatal(err, "Decode")
fmt.Printf("Body: %#v\n", body)
}
func main() {
launchTestServer()
for i := 0; i < 5; i++ {
clientDemo()
}
}

Golang Unmarshal an JSON response, then marshal with Struct field names

So I am hitting an API that returns a JSON response and I am unmarshalling it into a struct like so:
package main
type ProcessedRecords struct {
SLMIndividualID string `json:"individual_id"`
HouseholdPosition int `json:"Household Position"`
IndividualFirstName string `json:"individual_first_name"`
}
func main() {
req, _ := http.NewRequest(method, url, payload)
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(body)
var responseObject Response
json.Unmarshal(body, &responseObject)
fmt.Println(responseObject)
which works great. However I need to marshal this struct again but I want to use the Struct Fields as keys instead of the json: ... fields. I am using the following code:
recordsInput := []*firehose.Record{}
for i := 0; i < len(records); i++ {
if len(recordsInput) == 500 {
* code to submit records, this part works fine *
}
b, err := json.Marshal(records[i])
if err != nil {
log.Printf("Error: %v", err)
}
record := &firehose.Record{Data: b}
recordsInput = append(recordsInput, record)
}
This does submit records successfully but it's in the format:
{"individual_id":"33c05b49-149b-480f-b1c2-3a3b30e0cb6f","Household Position":1...}
and I'd like it in the format:
{"SLMIndividualId":"33c05b49-149b-480f-b1c2-3a3b30e0cb6f","HouseholdPosition":1...}
How can I achieve this?
Those tags say how the struct should be marshalled, so if they are present, that is how the output will be. You'll need to convert it to a matching struct that does not have the json: tags:
type ProcessedRecords struct {
SLMIndividualID string `json:"individual_id"`
HouseholdPosition int `json:"Household Position"`
IndividualFirstName string `json:"individual_first_name"`
}
type ProcessedRecordsOut struct {
SLMIndividualID string
HouseholdPosition int
IndividualFirstName string
}
func process() {
var in ProcessedRecords
json.Unmarshal(data, &in)
// Convert to same type w/o tags
out := ProcessedRecordsOut(in)
payload, _ := json.Marshal(out)
// ...
}
See a working example here: https://play.golang.org/p/p0Fc8DJotYE
You can omit fields one-way by defining a custom type and implementing the correct interface, e.g.
package main
import (
"encoding/json"
"fmt"
)
type Animal struct {
Name ReadOnlyString
Order string
}
type ReadOnlyString string
func (ReadOnlyString) UnmarshalJSON([]byte) error { return nil }
func main() {
x := Animal{"Bob", "First"}
js, err := json.Marshal(&x)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%s\n", js)
var jsonBlob = []byte(`{"Name": "Platypus", "Order": "Monotremata"}`)
if err := json.Unmarshal(jsonBlob, &x); err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%#v\n\n", x)
}
https://go.dev/play/p/-mwBL0kIqM
Found this answer here: https://github.com/golang/go/issues/19423#issuecomment-284607677

Resources