Trying to get the value of "Total" from JSON response - go

Response:
{
"meta": {
"query_time": 0.039130201,
"pagination": {
"offset": 1345,
"limit": 5000,
"total": 1345
},
Structs:
type InLicense struct {
Total int16 json:"total,omitempty"
}
type OutLicense struct {
Pagination []InLicense json:"pagination,omitempty"
}
type MetaLicense struct {
Meta []OutLicense json:"meta,omitempty"
}
Code snip inside function:
req, err := http.NewRequest("GET", , nil)
if err != nil {
//handle error
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Println("Error: ", err)
}
defer resp.Body.Close()
val := &MetaLicense{}
err = json.NewDecoder(resp.Body).Decode(&val)
if err != nil {
log.Fatal(err)
}
for _, s := range val.Meta {
for _, a := range s.Pagination {
fmt.Println(a.Total)
}
}
}
After I run this code I get the following error:
json: cannot unmarshal object into Go struct field MetaLicense.meta of type []OutLicense
Which type would []OutLicense need to be in order for this to be unmarshaled correctly? I cant print it another way, but it prints with the {} and Strings.Trim will not work.

You should use just a simple field declaration with actual type, not a [] of the type as it is done below:
type InLicense struct {
Total int16 json:"total,omitempty"
}
type OutLicense struct {
Pagination InLicense json:"pagination,omitempty"
}
type MetaLicense struct {
Meta OutLicense json:"meta,omitempty"
}

I simplified the parsing a bit and just used the json.Unmarshal() function instead.
raw := "{\n \"meta\": {\n \"query_time\": 0.039130201,\n \"pagination\": {\n \"offset\": 1345,\n \"limit\": 5000,\n \"total\": 1345\n }\n }\n}"
parsed := &MetaLicense{}
err := json.Unmarshal([]byte(raw), parsed)
if err != nil {
log.Fatal(err)
}
fmt.Println(parsed.Meta.Pagination.Total) // Prints: 1345
Here's the types I used
type InLicense struct {
Total int16 `json:"total,omitempty"`
}
type OutLicense struct {
Pagination InLicense `json:"pagination,omitempty"`
}
type MetaLicense struct {
Meta OutLicense `json:"meta,omitempty"`
}
As written your provided JSON has a extra , which makes your json unparsable (assuming you add the missing }'s too.
There are no lists in your JSON. Lists are denoted with the [] symbols. For your types to work it, your JSON would have to look like this:
{
"meta": [{
"query_time": 0.039130201,
"pagination": [{
"offset": 1345,
"limit": 5000,
"total": 1345
}]
}]
}

Related

Unmarshal nested GRPC structure in go

We want to unmarshal (in golang) a GRPC message and transform it into a map[string]interface{} to further process it. After using this code:
err := ptypes.UnmarshalAny(resource, config)
configMarshal, err := json.Marshal(config)
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
we get the following structure:
{
"name": "envoy.filters.network.tcp_proxy",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"value": "ChBCbGFja0hvbGVDbHVzdGVyEhBCbGFja0hvbGVDbHVzdGVy"
}
}
}
Where the TypedConfig field remains encoded. How can we decode the TypedConfig field? We know the type_url and we know the value, but to unmarshal the field, it needs to be of the pbany.Any type. But because the TypedConfig structure is a map[string] interface {}, our program either fails to compile, or it crashes, complaining that it is expecting a pbany.Any type, but instead it is getting a map[string] interface {}.
We have the following questions:
Is there a way to turn the structure under TypedConfig into a pbany.Any type that can be subsequently unmarshalled?
Is there a way to recursively unmarshal the entire GRPC message?
Edit (provide more information about the code, schemas/packages used)
We are looking at the code of xds_proxy.go here: https://github.com/istio/istio/blob/master/pkg/istio-agent/xds_proxy.go
This code uses a *discovery.DiscoveryResponse structure in this function:
func forwardToEnvoy(con *ProxyConnection, resp *discovery.DiscoveryResponse) {
The protobuf schema for discovery.DiscoveryResponse (and every other structure used in the code) is in the https://github.com/envoyproxy/go-control-plane/ repository in this file: https://github.com/envoyproxy/go-control-plane/blob/main/envoy/service/discovery/v3/discovery.pb.go
We added code to the forwardToEnvoy function to see the entire unmarshalled contents of the *discovery.DiscoveryResponse structure:
var config proto.Message
switch resp.TypeUrl {
case "type.googleapis.com/envoy.config.route.v3.RouteConfiguration":
config = &route.RouteConfiguration{}
case "type.googleapis.com/envoy.config.listener.v3.Listener":
config = &listener.Listener{}
// Six more cases here, truncated to save space
}
for _, resource := range resp.Resources {
err := ptypes.UnmarshalAny(resource, config)
if err != nil {
proxyLog.Infof("UnmarshalAny err %v", err)
return false
}
configMarshal, err := json.Marshal(config)
if err != nil {
proxyLog.Infof("Marshal err %v", err)
return false
}
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
if err != nil {
proxyLog.Infof("Unmarshal err %v", err)
return false
}
}
And this works well, except that now we have these TypedConfig fields that are still encoded:
{
"name": "virtualOutbound",
"address": {
"Address": {
"SocketAddress": {
"address": "0.0.0.0",
"PortSpecifier": {
"PortValue": 15001
}
}
}
},
"filter_chains": [
{
"filter_chain_match": {
"destination_port": {
"value": 15001
}
},
"filters": [
{
"name": "istio.stats",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/udpa.type.v1.TypedStruct",
"value": "CkF0eXBlLmdvb2dsZWFwaXMuY29tL2Vudm95LmV4dGVuc2lvbnMuZmlsdG"
}
}
},
One way to visualize the contents of the TypedConfig fields is to use this code:
for index1, filterChain := range listenerConfig.FilterChains {
for index2, filter := range filterChain.Filters {
proxyLog.Infof("Listener %d: Handling filter chain %d, filter %d", i, index1, index2)
switch filter.ConfigType.(type) {
case *listener.Filter_TypedConfig:
proxyLog.Infof("Found TypedConfig")
typedConfig := filter.GetTypedConfig()
proxyLog.Infof("typedConfig.TypeUrl = %s", typedConfig.TypeUrl)
switch typedConfig.TypeUrl {
case "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy":
tcpProxyConfig := &tcp_proxy.TcpProxy{}
err := typedConfig.UnmarshalTo(tcpProxyConfig)
if err != nil {
proxyLog.Errorf("Failed to unmarshal TCP proxy configuration")
} else {
proxyLog.Infof("TcpProxy Config for filter chain %d filter %d: %s", index1, index2, prettyPrint(tcpProxyConfig))
}
But then the code becomes very complex, as we have a large number of structures, and these structures can occur in different order in the messages.
So we wanted to get a generic way of unmarshalling these TypedConfig message by using pbAny, and hence our questions.

How can I put a format specifier in Elasticsearch query using Go?

In the following code, I want to put a variable id, like some format specifier %d. How do I do this for the following elasticsearch query with Golang?
str := `{
"query": {
"match": {
"id": 123
}
}
}`
s := []byte(str)
url := "http://localhost:9200/student/_delete_by_query"
_, err = helper.GoDelete(url, s)
if err != nil {
return err
}
return nil
Using fmt.Sprintf may be the simplest way to do that, not the fastest. but simplest.
d := 123
id := fmt.Sprintf(`{"query": {"match": {"id": %d}}}`, d)
fmt.Sprintf could work, but is also prone to errors. I would create the appropriate structures, then use json.Marshal to do it:
type (
Match struct {
ID int `json:"id"`
}
Query struct {
Match Match `json:"match"`
}
MyStruct struct {
Query Query `json:"query"`
}
)
func main() {
s := MyStruct{
Query: Query{
Match: Match{
ID: 123,
},
},
}
bytes, err := json.Marshal(s)
}

gin bindJson array of objects

I would like to bind a json array of objects like this one :
[
{
"id": "someid"
},
{
"id": "anotherid"
}
]
Here my model
type DeleteByID struct {
ID string `json:"id" binding:"required"`
}
I use gin to handle the object
var stock []DeleteByID
if err := ctx.ShouldBindJSON(&stock); err != nil {
return err
}
The problem is that it does not bind/check my object.
You can achieve this by using json.Unmarshal() like this:
var stock []DeleteByID
body, err := ioutil.ReadAll(c.Request.Body)
if err != nil {
c.AbortWithError(400, err)
return
}
err = json.Unmarshal(body, &stock)
if err != nil {
c.AbortWithError(400, err)
return
}
c.String(200, fmt.Sprintf("%#v", stock))
The alternative is to pass the array as a nested field. When marked with "dive", gin will bind and validate. These ones will cause an error:
{
"Deletions": [ {
"id": 13
},
{
}
]
}
This is acceptable input:
{
"Deletions": [
{
"id": "someid"
},
{
"id": "anotherid"
}
]
}
Here my model
type DeleteByID struct {
ID string `json:"id" binding:"required"`
}
type DeletePayload struct {
Deletions []DeleteByID `binding:"dive"`
}
The dive keyword will ensure that the JSON array is validated as it becomes a slice, map or array.
var stock DeletePayload
if err := ctx.ShouldBindJSON(&stock); err != nil {
return err
}
See this issue for some more details: https://github.com/gin-gonic/gin/issues/3238

Sorting not working in golang chaincode hyperledger fabric

I'm trying to sort the result in golang chaincode, but the result is random, below is my chaincode sample:
package main
import (
"bytes"
"encoding/json"
"fmt"
"time"
"github.com/hyperledger/fabric/core/chaincode/shim"
"github.com/hyperledger/fabric/protos/peer"
)
type itemStruct struct {
ID string `json:"id"`
Status string `json:"status"`
CreatedAt time.Time `json:"created_at"`
}
func createItem(stub shim.ChaincodeStubInterface, args []string) peer.Response {
if len(args) != 3 {
return shim.Error(fmt.Sprintf("Expecting %v arguments {id, status, created_at}, but got %v", 3, len(args)))
}
itemID := args[0]
if len(itemID) == 0 {
return shim.Error("id field is required")
}
status := args[1]
if len(status) == 0 {
return shim.Error("status field is required")
}
createdAt, err := time.Parse(time.RFC3339, args[2])
if err != nil {
return shim.Error("created_at is not a valid datetime string")
}
item := itemStruct{
ID: itemID,
CreatedAt: createdAt,
}
itemAsJSONBytes, err := json.Marshal(item)
if err != nil {
return shim.Error(err.Error())
}
return shim.Success(itemAsJSONBytes)
}
func getPendingItems(stub shim.ChaincodeStubInterface, args []string) peer.Response {
var bookmark string
if len(args) > 0 && len(args[0]) > 0 {
bookmark = args[0]
}
queryString := `{
"selector": {
"status": "pending"
},
"sort": [
{"created_at": "desc"}
]
}`
result, pagination, err := queryWithPagination(stub, queryString, 20, bookmark)
if err != nil {
return shim.Error(err.Error())
}
return shim.Success(constructResponse(result, pagination).Bytes())
}
func queryWithPagination(stub shim.ChaincodeStubInterface, queryString string, pageSize int32, bookmark string) (map[string]string, string, error) {
var pagination string
iterator, meta, err := stub.GetQueryResultWithPagination(queryString, pageSize, bookmark)
if err != nil {
return nil, pagination, err
}
defer iterator.Close()
result, err := iterateResult(iterator)
if err != nil {
return nil, pagination, err
}
pagination = fmt.Sprintf(`{"count": %v, "next_page_token": "%v"}`, meta.FetchedRecordsCount, meta.Bookmark)
return result, pagination, nil
}
func constructResponse(items map[string]string, pagination string) *bytes.Buffer {
// buffer is a JSON array containing QueryResults
var buffer bytes.Buffer
if len(pagination) > 0 {
buffer.WriteString(`{"data":`)
}
buffer.WriteString(`[`)
bArrayMemberAlreadyWritten := false
for _, val := range items {
// Add a comma before array members, suppress it for the first array member
if bArrayMemberAlreadyWritten == true {
buffer.WriteString(",")
}
buffer.WriteString(val)
bArrayMemberAlreadyWritten = true
}
buffer.WriteString("]")
if len(pagination) > 0 {
buffer.WriteString(`,"pagination":`)
buffer.WriteString(pagination)
buffer.WriteString("}")
}
return &buffer
}
func iterateResult(iterator shim.StateQueryIteratorInterface) (map[string]string, error) {
result := map[string]string{}
for iterator.HasNext() {
queryResponse, err := iterator.Next()
if err != nil {
return nil, err
}
result[queryResponse.Key] = string(queryResponse.Value)
}
return result, nil
}
// SmartContract : Smart contract struct
type SmartContract struct {
}
// Init : This method is called when chaincode is initialized or updated.
func (s *SmartContract) Init(stub shim.ChaincodeStubInterface) peer.Response {
return shim.Success(nil)
}
// Invoke : This method is called when any transaction or query fired
func (s *SmartContract) Invoke(stub shim.ChaincodeStubInterface) peer.Response {
// Retrieve the requested Smart Contract function and arguments
function, args := stub.GetFunctionAndParameters()
// Route to the appropriate handler function to interact with the ledger appropriately
// Transactions
if function == "createItem" {
return createItem(stub, args)
}
// Queries
if function == "getItems" {
return getItems(stub, args)
}
return shim.Error("Invalid function")
}
func main() {
// Create a new Smart Contract
err := shim.Start(new(SmartContract))
if err != nil {
fmt.Printf("Error creating new Smart Contract: %s", err)
}
}
It creates and asset which can have different status based on what's passed, and I have defined one query function which fetches only pending items.
I have applied sort, but the result is still random, can anyone help me here and guide me where I'm going wrong in this?
the sort field has to be present in the selector !
something like:
queryString := `{
"selector": {
"created_at": "$gt": null
"status": "pending"
},
"sort": [
{"created_at": "desc"}
]
}`
or (range)
queryString := `{
"selector": {
"created_at": {
"$gt": "2015-01-01T00:00:00Z",
"$lt": "2019-01-01T00:00:00Z"
},
"status": "pending"
},
"sort": [
{"created_at": "desc"}
]
}`
Per the docs - to use sorting, ensure that:
At least one of the sort fields is included in the selector.
There is an index already defined, with all the sort fields in the same
order.
Each object in the sort array has a single key.
If an object in the sort array does not have a single key, the resulting sort order is implementation specific and might change.
Also, as an fyi Find does not support multiple fields with different sort orders, so the directions must be either all ascending or all descending.
See also CouchDB sort doesn't work

unexpected end of JSON input while unmarshal to struct

Could you help me to understand why I'm always getting the error unexpected end of JSON input while trying to unmarshal the following json to the LimitOrder struct?
P.S.: if I use map[string]json.RawMessage instead of LimitOrder struct I'm able to execute the unmarshal.
{
"response_data": {
"order": {
"order_id": 3,
"coin_pair": "BRLBTC",
"order_type": 1,
"status": 4,
"has_fills": true,
"quantity": "1.00000000",
"limit_price": "900.00000",
"executed_quantity": "1.00000000",
"executed_price_avg": "900.00000",
"fee": "0.00300000",
"created_timestamp": "1453835329",
"updated_timestamp": "1453835329",
"operations": [
{
"operation_id": 1,
"quantity": "1.00000000",
"price": "900.00000",
"fee_rate": "0.30",
"executed_timestamp": "1453835329"
}
]
}
},
"status_code": 100,
"server_unix_timestamp": "1453835329"
}
LimitOrder struct
type LimitOrder struct {
OrderId int `json:"order_id"`
CoinPair string `json:"coin_pair"`
OrderType int `json:"order_type"`
Status int `json:"status"`
HasFills bool `json:"has_fills"`
Quantity float64 `json:"quantity,string"`
LimitPrice float64 `json:"limit_price,string"`
ExecutedQuantity float64 `json:"executed_quantity,string"`
ExecutedPriceAvg float64 `json:"executed_price_avg,string"`
Fee float64 `json:"fee,string"`
Operations []*Operation `json:"operations"`
CreatedTimestamp string `json:"created_timestamp"`
UpdatedTimestamp string `json:"updated_timestamp"`
}
and this is how I'm trying to unmarshal it
func (limitOrder *LimitOrder) UnmarshalJSON(buf []byte) error {
tmp := make(map[string]json.RawMessage)
if err := json.Unmarshal(buf, &tmp); err != nil {
return err
}
tmp2 := make(map[string]json.RawMessage)
if err := json.Unmarshal(tmp["response_data"], &tmp2); err != nil {
return err
}
if err := json.Unmarshal(tmp2["order"], limitOrder); err != nil {
return err
}
return nil
}
I found some help on golang-nuts group. You can check the answer here https://groups.google.com/forum/#!topic/golang-nuts/SZXBcXgUoo0. To make a long story short, the problem was with my structures, I've built structures only for a small piece of the whole json, so I fixed it building structures for the whole json response.

Resources