How can I create a nested JSON response from a response I received from an API?
For example, I got a resp (*http.Response) from an API which I expect to be a list of objects ([{},{},{},...])
I want to create a response like so
{
total: 1234,
addresses: [{},{},{}]
}
I'm not quite sure how to deal with this. I got close as my code below returns a similar structure but my addresses section returns an escaped string like so
"[{\"access\":\"INTERNAL\",\"address\":\"1P9SnpTait5bS\"}]"
func (h *Handler) getAddresses(w http.ResponseWriter, r *http.Request) {
type Message struct {
Total int `json:total`
Addresses string `json:addresses`
}
resp, _ := h.Search(address, page, pageOffset) // *http.Response
body, _ := ioutil.ReadAll(resp.Body)
res := Message{
Total: total,
Addresses: string(body),
}
m, _ := json.Marshal(res)
w.Write(m)
}
You can use json.RawMessage if all you need is to pass the json along.
func (h *Handler) getAddresses(w http.ResponseWriter, r *http.Request) {
type Message struct {
Total int `json:"total"`
Addresses json.RawMessage `json:"addresses"`
}
resp, err := h.Search(address, page, pageOffset) // *http.Response
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
res := Message{
Total: total,
Addresses: json.RawMessage(body),
}
if err := json.NewEncoder(w).Encode(res); err != nil {
log.Println("failed to respond with json:", err)
}
}
According to your desired response format, Addresses should be defined as type []map[string]interface rather than of type string.
type Message struct {
Total int `json:total`
Addresses []map[string]interface{} `json:addresses`
}
Then you would json unmarshal the response bytes into a temporary variable of type []map[string]interface{} before assigning the temporary variable to Message.Address, and finally json marshal Message as you were doing already.
Your question currently doesn't demonstrate where total is coming from. I assume you know how to deal with total.
Related
Let's say I have the following struct
type Response struct {
ID string `json:"id"`
Edited int `json:"edited"`
}
type Responses []Response
Then lets say I send a request to an API, but my issue is the API docs and from testing have told me that the edited value can come back as a bool or as a int. This obviously upsets Go and it throws an error when decoding the response body into the struct.
// ... http GET
// response [{"edited": true, id: 1}, {"edited": 1683248234.0, id: 2}]
r := Responses{}
err := json.NewDecoder(resp.Body).Decode(&r)
if err != nil {
return nil, err
}
defer resp.Body.Close()
How can I handle the above situation where I can't automatically load it into a struct? I'm assuming I'd need to do it into an interface first then filter the slice and handle the two different Response types in their own structs? But then I can't combine them!
So I'm thinking of conforming the field to one or the other, bool or int.
n.b. this relates to the Reddit API where some of the fields such as edited, created, created_utc don't have conforming types.
As told #mkopriva, the simplest way to handle different type of variable is use interface{} type:
const resp = `[{"edited": true, "id": 1}, {"edited": 1683248234.0, "id": 2}, {"id": 3}]`
type Response struct {
ID int `json:"id"`
Edited interface{} `json:"edited"`
}
type Responses []Response
r := Responses{}
err := json.Unmarshal([]byte(resp), &r)
if err != nil {
log.Fatal(err)
}
for _, response := range r {
switch response.Edited.(type) {
case float64:
fmt.Println("float64")
case bool:
fmt.Println("bool")
default:
fmt.Println("invalid")
}
}
PLAYGROUND
Also you can define new type with custom json.Unmarshaler implementation:
type Response struct {
ID int `json:"id"`
Edited Edited `json:"edited"`
}
type Edited struct {
BoolVal *bool
FloatVal *float64
}
func (e *Edited) UnmarshalJSON(data []byte) error {
boolVal, err := strconv.ParseBool(string(data))
if err == nil {
e.BoolVal = &boolVal
return nil
}
floatVal, err := strconv.ParseFloat(string(data), 64)
if err == nil {
e.FloatVal = &floatVal
return nil
}
return errors.New("undefined type")
}
r := Responses{}
err := json.Unmarshal([]byte(resp), &r)
if err != nil {
log.Fatal(err)
}
for _, response := range r {
edited := response.Edited
switch {
case edited.FloatVal != nil:
fmt.Println("float64")
case edited.BoolVal != nil:
fmt.Println("bool")
default:
fmt.Println("invalid")
}
}
PLAYGROUND
So I am hitting an API that returns a JSON response and I am unmarshalling it into a struct like so:
package main
type ProcessedRecords struct {
SLMIndividualID string `json:"individual_id"`
HouseholdPosition int `json:"Household Position"`
IndividualFirstName string `json:"individual_first_name"`
}
func main() {
req, _ := http.NewRequest(method, url, payload)
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(body)
var responseObject Response
json.Unmarshal(body, &responseObject)
fmt.Println(responseObject)
which works great. However I need to marshal this struct again but I want to use the Struct Fields as keys instead of the json: ... fields. I am using the following code:
recordsInput := []*firehose.Record{}
for i := 0; i < len(records); i++ {
if len(recordsInput) == 500 {
* code to submit records, this part works fine *
}
b, err := json.Marshal(records[i])
if err != nil {
log.Printf("Error: %v", err)
}
record := &firehose.Record{Data: b}
recordsInput = append(recordsInput, record)
}
This does submit records successfully but it's in the format:
{"individual_id":"33c05b49-149b-480f-b1c2-3a3b30e0cb6f","Household Position":1...}
and I'd like it in the format:
{"SLMIndividualId":"33c05b49-149b-480f-b1c2-3a3b30e0cb6f","HouseholdPosition":1...}
How can I achieve this?
Those tags say how the struct should be marshalled, so if they are present, that is how the output will be. You'll need to convert it to a matching struct that does not have the json: tags:
type ProcessedRecords struct {
SLMIndividualID string `json:"individual_id"`
HouseholdPosition int `json:"Household Position"`
IndividualFirstName string `json:"individual_first_name"`
}
type ProcessedRecordsOut struct {
SLMIndividualID string
HouseholdPosition int
IndividualFirstName string
}
func process() {
var in ProcessedRecords
json.Unmarshal(data, &in)
// Convert to same type w/o tags
out := ProcessedRecordsOut(in)
payload, _ := json.Marshal(out)
// ...
}
See a working example here: https://play.golang.org/p/p0Fc8DJotYE
You can omit fields one-way by defining a custom type and implementing the correct interface, e.g.
package main
import (
"encoding/json"
"fmt"
)
type Animal struct {
Name ReadOnlyString
Order string
}
type ReadOnlyString string
func (ReadOnlyString) UnmarshalJSON([]byte) error { return nil }
func main() {
x := Animal{"Bob", "First"}
js, err := json.Marshal(&x)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%s\n", js)
var jsonBlob = []byte(`{"Name": "Platypus", "Order": "Monotremata"}`)
if err := json.Unmarshal(jsonBlob, &x); err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%#v\n\n", x)
}
https://go.dev/play/p/-mwBL0kIqM
Found this answer here: https://github.com/golang/go/issues/19423#issuecomment-284607677
I have data returned from Elasticsearch, using "github.com/olivere/elastic". That sort of works, when i add it to my struct and string it, like so,
data := Api {
Total: myTotal,
Data: string(result),
}
c.JSON(http.StatusOK, totalData)
the api is a struct like so,
type Api struct {
Total interface{}
Data interface{}
}
This returns data ok, from 1 to any number of results on request. How the results loaded into the data interface are not escaped or something, e.g.
"Data":"{\"CID\":\"XXXXXXXXXX\",\"Link\":\"XXXXXXXXX\",
So I have tried to unmarshal the data before adding it to the api struct.
var p DataApi
err := json.Unmarshal(result, &p)
if err != nil {
panic(err)
}
totalData := Api {
Total: myTotal,
Data: p,
}
c.JSON(http.StatusOK, totalData)
This sort of works fine, returns the data in the correct way, but only when loading one result. When 2 or more results are requested, I get this error from the unmarshal panic
invalid character '{' after top-level value
I have tried and google all over but can not find a solution to this? I am not sure what I am doing wrong? The DataApi is a nested set of structs, I was not sure if there was anything I should be being because of that?
This is being run within the Gin framework.
Thanks.
EDIT
So when I use fmt.Println on the string(result) I can print any number of results on the screen. How can I add this to the API struct and then I need the struct converted into JSON data. Is there some way of appending this string data on the JSON converted API struct?
Try to unmarshal multiple results into a slice:
var q []Api
err = json.Unmarshal(result, &q)
See on playground https://play.golang.org/p/D_bVAd4jBlI
package main
import (
"encoding/json"
"fmt"
)
type Api struct {
Total interface{}
Data interface{}
}
func main() {
data := Api{
Total: 1,
Data: "2",
}
result, err := json.Marshal(data)
if err != nil {
panic(err)
}
fmt.Printf("single data: %s\n", result)
var p Api
err = json.Unmarshal(result, &p)
if err != nil {
panic(err)
}
dataSlice := []Api{data}
result, err = json.Marshal(dataSlice)
if err != nil {
panic(err)
}
fmt.Printf("slice of data: %s\n", result)
var q []Api
err = json.Unmarshal(result, &q)
if err != nil {
panic(err)
}
}
Use json.RawMessage to store arbitrary JSON documents:
var p json.RawMessage
err := json.Unmarshal(result, &p)
if err != nil {
panic(err)
}
totalData := Api {
Total: myTotal,
Data: p,
}
c.JSON(http.StatusOK, totalData)
I have a working solution to my problem, I just use the Hits Hits From the data returned by elastic search, I would like just the source data but i think it does what I need it to do... for now.
Thanks.
I am attempting to map a response from an API to a struct using Golang.
The JSON that comes back when I view the link in the browser is below:
{
"GBP": 657.54
}
And I just want to map it to a simple struct like so:
type Price struct {
Name string
Value float64
}
Here is my current code.
func FetchCoinPrice(fsym string, tsyms string) Price {
url := fmt.Sprintf("https://min-api.cryptocompare.com/data/price?fsym=" + fsym + "&tsyms=" + tsyms)
fmt.Println("Requesting data from " + url)
price := Price{}
// getting the data using http
request, err := http.Get(url)
if err != nil {
log.Fatal(err.Error())
}
// Read the response body using ioutil
body, err := ioutil.ReadAll(request.Body)
if err != nil {
log.Fatal(err.Error())
}
defer request.Body.Close()
if request.StatusCode == http.StatusOK {
json.Unmarshal(body, &price)
}
return price
}
At the moment all I receive is an empty struct, I know the link is bringing back the correct data and I've tested it in my browser.
The mapping doesn't work that way. Instead, you should use a map:
data := []byte(`{
"GBP": 657.54
}`)
priceMap := map[string]float64{}
err := json.Unmarshal(data, &priceMap)
// Check your errors!
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(priceMap)
This will print:
map[GBP:657.54]
You can then iterate over the map and build the struct you mentioned above, or just access the entry directly if you know the currency. eg: priceMap["GBP"]
You should really check your errors, especially if you're not getting the output you expect from Unmarshal.
The problem is that the unmarshaler cannot guess that keys in a JSON object should correspond to some value in a struct. Golang JSON mapping simply doesn't work that way.
However, you can make your "Price" type implement json.Unmarshaler to deserialize a message into a map of floats (map[string]float64) then ensure the shape is right and populate the struct accordingly:
func (p *Price) UnmarshalJSON(bs []byte) error {
kvs := map[string]float64{}
err := json.Unmarshal(bs, &kvs)
if err != nil {
return err
}
if len(kvs) != 1 {
return fmt.Errorf("expected 1 key, got %d", len(kvs))
}
for name, value := range kvs {
p.Name, p.Value = name, value
}
return nil
}
func main() {
jsonstr := `[{"GBP":657.54},{"USD":123.45}]`
ps := []Price{}
err := json.Unmarshal([]byte(jsonstr), &ps)
if err != nil {
panic(err)
}
// ps=[]main.Price{
// main.Price{Name:"GBP", Value:657.54},
// main.Price{Name:"USD", Value:123.45}
// }
}
I have this simple generic Request struct to make get requests in my app:
package api
import (
"net/http"
"time"
"log"
"app/errors"
)
type Request struct {
Url string
}
func (request *Request) Run(responseObject *AppStatusInfo) *errors.Error {
req, requestErr := http.NewRequest(http.MethodGet, request.Url, nil)
req.Header.Set("Content-Type", "application/json")
timeout := time.Duration(5 * time.Second)
client := &http.Client{
Timeout: timeout,
}
resp, requestErr := client.Do(req)
if requestErr != nil {
return &errors.UnknownError
}
decodeError := DecodeJsonRequestBody(resp, &responseObject)
if (decodeError != nil) {
return &errors.UnknownError
}
defer resp.Body.Close()
return nil
}
Here responseObject has pointer of type AppStatusInfo which is a struct with some fields.
I run it like this to get app status information and put it inside appStatusInfo object:
var appStatusInfo AppStatusInfo
req := Request{
Url:config.Config.ApiUrl,
}
req.Run(&appStatusInfo)
So, this code runs fine.
But, when I want to generalize Request to accept other types of responses, like UserProducts, I don't know how to do it without replacing responseObject *AppStatusInfo with responseObject interface{}, then casting it with responseObject.(UserProducts) which I think can be improved.
So, as soon as there are no generics, how do I make Request.Run() accept different types and return respective objects?
Assuming that DecodeJsonRequestBody passes the second argument to json.Unmarshal or json.Decoder.Decode, then write it like this. I show the changed lines only:
func (request *Request) Run(responseObject interface{}) *errors.Error {
...
resp, requestErr := client.Do(req)
if requestErr != nil {
return &errors.UnknownError
}
defer resp.Body.Close() // defer close before doing anything else
...
decodeError := DecodeJsonRequestBody(resp, responseObject) // don't take address of responseObject
...
}
You can call it like this:
var up UserProducts
err = r.Run(&up)
var asi AppStatusInfo
err = r.Run(&asi)
Type assertions and type conversions are not required.