Golang Unmarshal an JSON response, then marshal with Struct field names - go

So I am hitting an API that returns a JSON response and I am unmarshalling it into a struct like so:
package main
type ProcessedRecords struct {
SLMIndividualID string `json:"individual_id"`
HouseholdPosition int `json:"Household Position"`
IndividualFirstName string `json:"individual_first_name"`
}
func main() {
req, _ := http.NewRequest(method, url, payload)
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(body)
var responseObject Response
json.Unmarshal(body, &responseObject)
fmt.Println(responseObject)
which works great. However I need to marshal this struct again but I want to use the Struct Fields as keys instead of the json: ... fields. I am using the following code:
recordsInput := []*firehose.Record{}
for i := 0; i < len(records); i++ {
if len(recordsInput) == 500 {
* code to submit records, this part works fine *
}
b, err := json.Marshal(records[i])
if err != nil {
log.Printf("Error: %v", err)
}
record := &firehose.Record{Data: b}
recordsInput = append(recordsInput, record)
}
This does submit records successfully but it's in the format:
{"individual_id":"33c05b49-149b-480f-b1c2-3a3b30e0cb6f","Household Position":1...}
and I'd like it in the format:
{"SLMIndividualId":"33c05b49-149b-480f-b1c2-3a3b30e0cb6f","HouseholdPosition":1...}
How can I achieve this?

Those tags say how the struct should be marshalled, so if they are present, that is how the output will be. You'll need to convert it to a matching struct that does not have the json: tags:
type ProcessedRecords struct {
SLMIndividualID string `json:"individual_id"`
HouseholdPosition int `json:"Household Position"`
IndividualFirstName string `json:"individual_first_name"`
}
type ProcessedRecordsOut struct {
SLMIndividualID string
HouseholdPosition int
IndividualFirstName string
}
func process() {
var in ProcessedRecords
json.Unmarshal(data, &in)
// Convert to same type w/o tags
out := ProcessedRecordsOut(in)
payload, _ := json.Marshal(out)
// ...
}
See a working example here: https://play.golang.org/p/p0Fc8DJotYE

You can omit fields one-way by defining a custom type and implementing the correct interface, e.g.
package main
import (
"encoding/json"
"fmt"
)
type Animal struct {
Name ReadOnlyString
Order string
}
type ReadOnlyString string
func (ReadOnlyString) UnmarshalJSON([]byte) error { return nil }
func main() {
x := Animal{"Bob", "First"}
js, err := json.Marshal(&x)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%s\n", js)
var jsonBlob = []byte(`{"Name": "Platypus", "Order": "Monotremata"}`)
if err := json.Unmarshal(jsonBlob, &x); err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%#v\n\n", x)
}
https://go.dev/play/p/-mwBL0kIqM
Found this answer here: https://github.com/golang/go/issues/19423#issuecomment-284607677

Related

How to set slice interface values with reflection

I would like to build a function that takes a generic pointer array and fill that list based on mongo results.
I don't know how to set the value I got from mongo into my pointer array. In the below attempt, program panics with following error : reflect.Set: value of type []interface {} is not assignable to type []Person
When I print total / documents found, it corresponds to what I am expecting. So I think question is about reflection.
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
var mongoResp struct {
Total int `bson:"total"`
Documents interface{} `bson:"documents"`
}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
receiverValue := reflect.ValueOf(receiver)
docs := []interface{}(mongoResp.Documents.(primitive.A))
receiverValue.Elem().Set(reflect.ValueOf(docs))
return mongoResp.Total, nil
}
type Person struct {
Name string `bson:"name"`
}
func main() {
var persons []Person
count, err := getListWithCount(context.Background(), &persons)
if err != nil {
log.Fatal(err)
}
fmt.Println(count)
fmt.Println(persons)
}
You should be able to decode first into bson.RawValue and then Unmarshal it into the receiver.
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
var mongoResp struct {
Total int `bson:"total"`
Documents bson.RawValue `bson:"documents"`
}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
if err := mongoResp.Documents.Unmarshal(receiver); err != nil {
return 0, err
}
return mongoResp.Total, nil
}
You can also implement it as a custom bson.Unmarshaler.
type MongoResp struct {
Total int `bson:"total"`
Documents interface{} `bson:"documents"`
}
func (r *MongoResp) UnmarshalBSON(data []byte) error {
var temp struct {
Total int `bson:"total"`
Documents bson.RawValue `bson:"documents"`
}
if err := bson.Unmarshal(data, &temp); err != nil {
return err
}
r.Total = temp.Total
return temp.Documents.Unmarshal(r.Documents)
}
With that you would use it in the function like so:
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
mongoResp := MongoResp{Documents: receiver}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
return mongoResp.Total, nil
}
Dynamically create a struct type that matches the queried document. See commentary below for details.
func getListWithCount(receiver interface{}) (int, error) {
dst := reflect.ValueOf(receiver).Elem()
// Your mongo query here
// Create a struct type that matches the document.
doct := reflect.StructOf([]reflect.StructField{
reflect.StructField{Name: "Total", Type: reflect.TypeOf(0), Tag: `bson:"total"`},
reflect.StructField{Name: "Documents", Type: dst.Type(), Tag: `bson:"documents"`},
})
// Decode to a value of the type.
docp := reflect.New(doct)
if err := cursor.Decode(docp.Interface()); err != nil {
return 0, err
}
docv := docp.Elem()
// Copy the Documents field to *receiver.
dst.Set(docv.Field(1))
// Return the total
return docv.Field(0).Interface().(int), nil
}
there is no need to use reflect here, you can decode it directly to your Person slices
func getPersons(ctx context.Context, coll *mongo.Collection, results interface{}) error {
cur, err := coll.Find(ctx, bson.D{})
if err != nil {
return err
}
err = cur.All(ctx, results)
if err != nil {
return err
}
return nil
}
and the len is the count of the results.
err = getPersons(ctx, coll, &persons)
require.NoError(t, err)
t.Logf("Got %d persons: %v", len(persons), persons)
see https://gist.github.com/xingyongtao/459f92490bdcbf7d5afe9f5d1ae6c04a

How to access a nested field in struct in Golang

Let's say I have this structblock:
type AllShipments struct {
Shipments []struct {
ShipmentID int `json:"shipmentId"`
ShipmentDate time.Time `json:"shipmentDate"`
ShipmentItems []struct {
OrderItemID string `json:"orderItemId"`
OrderID string `json:"orderId"`
} `json:"shipmentItems"`
Transport struct {
TransportID int `json:"transportId"`
} `json:"transport"`
} `json:"shipments"`
}
I use:
func main() {
var t AllShipments
.....
.....
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(body, &t)
if err != nil {
fmt.Println(string(body))
}
prettyJSON, err := json.MarshalIndent(t, "", " ")
if err != nil {
log.Fatal("Failed to generate json", err)
}
fmt.Printf("%s\n", string(prettyJSON))
How to get for example only the json field "shipmentDate", I tried t.Shipments.ShipmentDate but that did not work. t.Shipments did work to filter out the first array. So what is the best way to do that? I know I can disable the other json fields with "json:-" but that is not what I am looking for. Just a way to access that field
t.Shipments is a slice - so to access any of it's elements you must pass an index - and then you can access that element's fields e.g.
if len(t.Shipments) > 0 {
fmt.Println(v.Shipments[0].ShipmentDate)
}
or to range over the slice:
for i, v := range t.Shipments {
fmt.Println(i, v.ShipmentDate)
}

Map response to a struct using Golang

I am attempting to map a response from an API to a struct using Golang.
The JSON that comes back when I view the link in the browser is below:
{
"GBP": 657.54
}
And I just want to map it to a simple struct like so:
type Price struct {
Name string
Value float64
}
Here is my current code.
func FetchCoinPrice(fsym string, tsyms string) Price {
url := fmt.Sprintf("https://min-api.cryptocompare.com/data/price?fsym=" + fsym + "&tsyms=" + tsyms)
fmt.Println("Requesting data from " + url)
price := Price{}
// getting the data using http
request, err := http.Get(url)
if err != nil {
log.Fatal(err.Error())
}
// Read the response body using ioutil
body, err := ioutil.ReadAll(request.Body)
if err != nil {
log.Fatal(err.Error())
}
defer request.Body.Close()
if request.StatusCode == http.StatusOK {
json.Unmarshal(body, &price)
}
return price
}
At the moment all I receive is an empty struct, I know the link is bringing back the correct data and I've tested it in my browser.
The mapping doesn't work that way. Instead, you should use a map:
data := []byte(`{
"GBP": 657.54
}`)
priceMap := map[string]float64{}
err := json.Unmarshal(data, &priceMap)
// Check your errors!
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(priceMap)
This will print:
map[GBP:657.54]
You can then iterate over the map and build the struct you mentioned above, or just access the entry directly if you know the currency. eg: priceMap["GBP"]
You should really check your errors, especially if you're not getting the output you expect from Unmarshal.
The problem is that the unmarshaler cannot guess that keys in a JSON object should correspond to some value in a struct. Golang JSON mapping simply doesn't work that way.
However, you can make your "Price" type implement json.Unmarshaler to deserialize a message into a map of floats (map[string]float64) then ensure the shape is right and populate the struct accordingly:
func (p *Price) UnmarshalJSON(bs []byte) error {
kvs := map[string]float64{}
err := json.Unmarshal(bs, &kvs)
if err != nil {
return err
}
if len(kvs) != 1 {
return fmt.Errorf("expected 1 key, got %d", len(kvs))
}
for name, value := range kvs {
p.Name, p.Value = name, value
}
return nil
}
func main() {
jsonstr := `[{"GBP":657.54},{"USD":123.45}]`
ps := []Price{}
err := json.Unmarshal([]byte(jsonstr), &ps)
if err != nil {
panic(err)
}
// ps=[]main.Price{
// main.Price{Name:"GBP", Value:657.54},
// main.Price{Name:"USD", Value:123.45}
// }
}

How to judge unmarshal json interface{} type in golang?

I want to judge json type,but it always return "I don't know about type map[string]interface {}!",How to resolve it.
=========================================================================
type getRemoteCardInfo struct {
Code int
Msg string
Data []*remoteCardInfo
}
type remoteCardInfo struct {
Sn string
RemoteCardIp string
RemoteCardMac string
}
func Get_json_data(url string) (interface{}, error) {
client := &http.Client{}
req, err := http.NewRequest("GET", url, nil)
req.Header.Add("X-MYCMDB-Auth-Token", "sPf98SMBWzOZJEJB8KWltbJyKvFYPauu")
if err != nil {
return nil, err
}
resp, _ := client.Do(req)
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
return nil, fmt.Errorf("query failed: %s", resp.Status)
}
var result interface{}
body, err := ioutil.ReadAll(resp.Body)
if err := json.Unmarshal(body, &result); err != nil {
log.Fatalf("JSON unmarshaling failed: %s", err)
}
resp.Body.Close()
fmt.Println(result)
return result, nil
}
func main() {
jsondata, err := Get_json_data(DHCPURL)
if err != nil {
log.Fatal(err)
}
switch v := jsondata.(type) {
case getRemoteCardInfo:
fmt.Println("aaaa")
default:
fmt.Printf("I don't know about type %T!\n", v)
}
The go JSON unmarshaler doesn't know about types, as you can tell by the fact that it stores the result into an interface{} value:
func Unmarshal(data []byte, v interface{}) error
// "v" can be any type -------^
So it's up to you to use the unmarshaler to populate your structure and determine if the result is valid or not.
In your example it looks like you're trying to unmarshal a remoteCardInfo from an HTTP response. To do this you should unmarshal into an empty remoteCardInfo struct and determine if the required fields were populated.
For example, suppose you expect a JSON document like so:
{
"sn": "123",
"ip": "0.0.0.0",
"mac": "ff:ff:ff:ff:ff:ff"
}
Then you should define your "remoteCardInfo" struct as below:
type remoteCardInfo struct {
Sn string `json:"sn"`
RemoteCardIp string `json:"ip"`
RemoteCardMac string `json:"mac"`
}
And then unmarshal and validate it like so:
func getRemoteCardInfo(bs []byte) (*remoteCardInfo, error) {
rci := remoteCardInfo{}
err := json.Unmarshal(bs, &rci)
if err != nil {
return nil, err
}
// Validate the expected fields
if rci.Sn == "" {
return nil, fmt.Errorf(`missing "sn"`)
}
if rci.RemoteCardIp == "" {
return nil, fmt.Errorf(`missing "ip"`)
}
if rci.RemoteCardMac == "" {
return nil, fmt.Errorf(`missing "mac"`)
}
return &rci, nil
}
Of course, you can validate the fields any way you like but the main thing to remember is that the unmarshaler only does the job of ensuring that the input byte array is a valid JSON document and populates the fields from the document into the fields defined by the value.
It cannot tell you what "type" of object the JSON document represents.

Index out of Range with array of structs in Go

I am new to Go so hopefully I'm making myself clear with this issue I'm having. My problem is that I am trying to iterate over an array of structs but I keep running into an index out of range issue. For the purposes of this problem, I have already verified that my array is not empty but that it in fact does contain at least one Services struct and file_content is the string that contains my valid JSON
Here is the snippet of code that represents the problem I'm having:
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
err_json := json.Unmarshal(file_content, &ServiceConf)
for _, s := range ServiceConf.Services {
log.Println(s)
}
So every time I run my code I get:
2014/03/14 18:19:53 http: panic serving [::1]:65448: runtime error: index out of range
{
"services" : [
{
"name": "translation",
"command": "to german",
"request": {
"key": "XXX",
"url": "https://www.googleapis.com/language/translate/v2?"
}
}
]
}
If you're interested in the complete source file:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
)
type SlackResponse struct {
token string
team_id string
channel_id string
channel_name string
timestamp string
user_id string
user_name string
text string
}
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
func main() {
content, err_read := ioutil.ReadFile("config.ini")
if err_read != nil {
log.Println("Could not read config")
return
}
log.Println(string(content))
err_json := json.Unmarshal(content, &ServiceConf)
if err_json != nil {
log.Println(err_json)
}
http.HandleFunc("/", handler)
http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
slack_response := SlackResponse{
r.FormValue("token"),
r.FormValue("team_id"),
r.FormValue("channel_id"),
r.FormValue("channel_name"),
r.FormValue("timestamp"),
r.FormValue("user_id"),
r.FormValue("user_name"),
r.FormValue("text"),
}
// log.Println(ServiceConf.Services[0].Request["key"])
// loop through services to find command phrases
for _, s := range ServiceConf.Services {
log.Println(s)
}
if slack_response.user_name == "slackbot" {
return
}
// fmt.Fprintf(w, "{ \"text\": \"Master %s! You said: '%s'\" }", slack_response.user_name, slack_response.text)
content, err := getContent("https://www.googleapis.com/language/translate/v2?key=&source=en&target=de&q=" + url.QueryEscape(slack_response.text))
if err != nil {
fmt.Fprintf(w, "{ \"text\": \"Huh?!\" }")
} else {
type trans struct {
Data struct {
Translations []struct {
TranslatedText string `json:"translatedText"`
} `json:"translations"`
} `json:"data"`
}
f := trans{}
err := json.Unmarshal(content, &f)
if err != nil {
log.Println(err)
}
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }", f.Data.Translations[0].TranslatedText)
}
}
// array of bytes if retrieved successfully.
func getContent(url string) ([]byte, error) {
// Build the request
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
// Send the request via a client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
// Defer the closing of the body
defer resp.Body.Close()
// Read the content into a byte array
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
// At this point we're done - simply return the bytes
return body, nil
}
Here is the stack trace:
2014/03/21 23:21:29 http: panic serving [::1]:59508: runtime error: index out of range
goroutine 3 [running]:
net/http.funcĀ·009()
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x215f80, 0x4b6537)
/usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/panic.c:248 +0x106
main.handler(0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/Users/et/src/go/src/github.com/etdebruin/gojacques/main.go:100 +0x81b
net/http.HandlerFunc.ServeHTTP(0x2cbc60, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc21001e5d0, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1496 +0x163
net/http.serverHandler.ServeHTTP(0xc21001f500, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc210058300)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1644 +0x28b
The error comes from this line
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }",
f.Data.Translations[0].TranslatedText)
So you didn't get any Translations back - that array is empty.
You might want to check resp.Status to see if an error was returned. This isn't returned as an error - you need to check it yourself.

Resources