Extracting Data From Kafka REST Proxy in Go - go

I am using the REST proxy instance of Kafka for producing and consuming messages.Using the API to get new messages but I am not able to convert those messages to a struct model in Go. For example:
// Get records
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf(FETCH_CONSUMER, URL, GROUP, CONSUMER), nil)
if err != nil {
panic(err)
}
req.Header.Add("Accept", CONTENT_TYPE)
respRecords, err := client.Do(req)
if err != nil {
panic(err)
}
defer respRecords.Body.Close()
fmt.Printf("Response %s\n", respRecords.Status)
fmt.Println(respRecords.Body)
recordsBodyResp := bufio.NewScanner(respRecords.Body)
for recordsBodyResp.Scan() {
fmt.Printf("<--Body %s\n", recordsBodyResp.Text())
}
The value returned is in the following format:
[{"topic":"backward","key":null,"value":{"AdoptionID":"abcd123","IPAddress":"8.8.8.8","Port":"80","Status":"requested"},"partition":0,"offset":7}]
Since it's an array of objects, I want to extract the value portion of the key "value" into a struct.
That is where I am stuck.

You can create a struct like:
type AutoGenerated []struct {
Topic string `json:"topic"`
Key interface{} `json:"key"`
Value Value `json:"value"`
Partition int `json:"partition"`
Offset int `json:"offset"`
}
type Value struct {
AdoptionID string `json:"AdoptionID"`
IPAddress string `json:"IPAddress"`
Port string `json:"Port"`
Status string `json:"Status"`
}
And Unmarshal in that struct.
See this sample code:
package main
import (
"fmt"
"encoding/json"
)
func main() {
type Value struct {
AdoptionID string `json:"AdoptionID"`
IPAddress string `json:"IPAddress"`
Port string `json:"Port"`
Status string `json:"Status"`
}
type AutoGenerated []struct {
Topic string `json:"topic"`
Key interface{} `json:"key"`
Value Value `json:"value"`
Partition int `json:"partition"`
Offset int `json:"offset"`
}
byt := []byte(`[{"topic":"backward","key":null,"value":{"AdoptionID":"abcd123","IPAddress":"8.8.8.8","Port":"80","Status":"requested"},"partition":0,"offset":7}]`)
var dat AutoGenerated
if err := json.Unmarshal(byt, &dat); err != nil {
panic(err)
}
fmt.Printf("%#v", dat)
}

Related

Reading and Unmarshalling API results in Golang

In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}

Deserialize proto with oneof from BSON fails

Deserializing a BSON into a structure created by protobuf with a oneof property fails:
panic: no decoder found for test.isTest_Entry
This is probably because the oneof field is registered as a plain interface (from the generated code):
type Test struct {
// Types that are valid to be assigned to Entry:
// *Test_S1
// *Test_S2
Entry isTest_Entry `protobuf_oneof:"entry"`
XXX_NoUnkeyedLiteral struct{} `json:"-" bson:"-"`
XXX_unrecognized []byte `json:"-" bson:"-"`
XXX_sizecache int32 `json:"-" bson:"-"`
}
//...
type isTest_Entry interface {
isTest_Entry()
}
package main
import (
"fmt"
"go.mongodb.org/mongo-driver/bson"
"test"
)
func main() {
entry := test.Test{}
data, err := bson.Marshal(&entry)
if nil != err {
panic(err)
}
fmt.Println(data)
newMessage := &test.Test{}
err = bson.Unmarshal(data, newMessage)
if err != nil {
panic(err)
}
fmt.Println(newMessage.GetEntry())
}
Is there any tags I can add to make this work or is this a bug?

How to omit empty json fields using json.decoder

I try to understand why both functions return the same output.
As far as I understood, the point of omit empty is to not add that key to the result struct.
I wrote this example, I was expecting the first output not to have the "Empty" key, but for some reason its value still shows as 0.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type agentOmitEmpty struct {
Alias string `json:"Alias,omitempty"`
Skilled bool `json:"Skilled,omitempty"`
FinID int32 `json:"FinId,omitempty"`
Empty int `json:"Empty,omitempty"`
}
type agent struct {
Alias string `json:"Alias"`
Skilled bool `json:"Skilled"`
FinID int32 `json:"FinId"`
Empty int `json:"Empty"`
}
func main() {
jsonString := `{
"Alias":"Robert",
"Skilled":true,
"FinId":12345
}`
fmt.Printf("output with omit emtpy: %v\n", withEmpty(strings.NewReader(jsonString)))
// output with omit emtpy: {Robert true 12345 0}
fmt.Printf("output regular: %v\n", withoutEmpty(strings.NewReader(jsonString)))
// output without omit: {Robert true 12345 0}
}
func withEmpty(r *strings.Reader) agentOmitEmpty {
dec := json.NewDecoder(r)
body := agentOmitEmpty{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
func withoutEmpty(r *strings.Reader) agent {
dec := json.NewDecoder(r)
body := agent{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
You need to define Empty as *int so it will be replaced with nil when there is no value. Then it will not be saved in the database.

How to check if a specific property of a struct is null?

Forgive me, I come from a c# background!
I have following struct in Go. We populate this struct by reading in the config from a file, which works well. However Im trying to find out a way to tell if a particular property in the struct, when passed in through the config file is null. As in, explicitly not set at all.
I've struggled for about 3 hours on this. I can do it for type strings etc, but I can't find out how to do it generically, across all types?
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
// Config type for configuration
type Config struct {
BatchSize int `json:"batchSize"`
BatchTime int `json:"batchTime"`
DataFolder string `json:"dataFolder"`
TempFolder string `json:"tempFolder"`
//Kafka configuration
Brokers []string `json:"streamBrokers"`
TopicJoined string `json:"streamTopicJoined"`
TopicRemoved string `json:"streamTopicRemoved"`
Group string `json:"streamGroup"`
ClientName string `json:"streamClientName"`
// Stats configuration
StatsPrefix string `json:"statsPrefix"`
//AWS S3 configuration
AccessKey string `json:"amazonAccessKey"`
SecretKey string `json:"amazonSecretKey"`
Region string `json:"amazonRegion"`
Endpoint string `json:"amazonEndpoint"`
S3Bucket string `json:"amazonS3Bucket"`
S3UploadBufferSize int32 `json:"amazonS3UploadBufferSize"`
S3UploadConcurrentSize int32 `json:"amazonS3UploadConcurrentSize"`
S3UploadRetries int32 `json:"amazonS3UploadRetries"`
S3UploadRetryTime int32 `json:"amazonS3UploadRetryTime"`
//Logging
StatsdHost string `json:"statsdHost"`
StatsdPort int `json:"statsdPort"`
StatsdRate float64 `json:"statsdRate"`
//Test Publishing
TestMode bool `json:"testMode"`
TestCount int `json:"testCount"`
}
// LoadConfig load config from file
func LoadConfig(configFile string) *Config {
if _, err := os.Stat(configFile); os.IsNotExist(err) {
panic(err)
}
if config, err := loadFromFile(configFile); nil != err {
panic(err)
} else {
fmt.Println("OneDrive", os.Getenv("OneDrive"))
msValuePtr := reflect.ValueOf(config)
msValue := msValuePtr.Elem()
typeOfT := msValue.Type()
for i := 0; i < msValue.NumField(); i++ {
field := msValue.Field(i)
// TODO: Check if field is null, regardless of type and the value from OS env variables...
}
return config
}
}
func loadFromFile(path string) (*Config, error) {
var config Config
file, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("could not open config path %q: %v", path, err)
}
defer file.Close()
decoder := json.NewDecoder(file)
err = decoder.Decode(&config)
if err != nil {
return nil, fmt.Errorf("could not parse config path %q: %v", path, err)
}
return &config, nil
}
In go, a value's default value is its zero value. You may want to make all of your types pointers (eg: *string rather than string) since the zero value of a pointer is nil. Unmarshaling your config file into a struct would preserve nil values for the keys that are missing / have null values.
Note that since slices (eg: []string) are reference types, they act as pointers and are nullable (meaning you wouldn't need to declare the type as *[]string).
I've used this library in the past to help with merging config / setting required keys (and many others exist):
https://github.com/jinzhu/configor
Example of encoding/decoding json - https://play.golang.org/p/DU_5Tuvm5-

Use string as struct value

I have this code. What I need is to get the transaction details from the transaction ID returned from blockchain
package main
import (
"encoding/base64"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"strings"
)
type Transaction struct {
Bid string `json:"bid"`
Fun string `json:"fun"`
ID string `json:"id"`
Timestamp string `json:"timestamp"`
TraderA string `json:"traderA"`
TraderB string `json:"traderB"`
Seller string `json:"seller"`
PointAmount string `json:"pointAmount"`
PrevTransactionID string `json:"prevTransactionId"`
}
type AllTxs struct {
TXs []Transaction `json:"tx"`
}
type Transact struct {
Cert string `json:"cert"`
ChaincodeID string `json:"chaincodeID"`
Nonce string `json:"nonce"`
Payload string `json:"payload"`
Signature string `json:"signature"`
Timestamp string `json:"nanos"`
Txid string `json:"txid"`
Type int `json:"type"`
}
func main() {
resp, err := http.Get("http://blockchain_transactions_url/trans_id")
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
byteArray := []byte(body)
var t Transact
json.Unmarshal(byteArray, &t)
//I get all the values base64 encoded
st, err := base64.StdEncoding.DecodeString(t.Payload)
if err != nil {
log.Fatal(err)
}
trd := string(st)
sp := strings.Split(trd, "\n")
result := strings.Join(sp, ",")
res := strings.Replace(result, ",", `", "`, -1)
ret := strings.Replace(res, `", "`, `{"`, 1) + `"}`
byteA := []byte(ret)
var tf AllTxs
json.Unmarshal(byteA, &tf)
//the tf...
ref := Transaction{}
fmt.Println(ref.Id)
}
the t.Payload I get is
"CsYBCAESgwESgAE3ZjFhY2Y2MTgxMGRhODMyMTA5NjZiNGYzNjc2NWU5NmIxY2Q0OTliODkyNmY0MDU0YWQ5NzhlNzhkZjczMDRhOGZlMDM1ZjZhYTBhODE2YzdmNjFlNGZkZDQ1MjM4M2Q5ZmU5ZDQxNmIyZGI4YTE1YmRkMjAzZmU2N2I5OTYyZho8ChBpbml0X3RyYW5zYWN0aW9uCgYwMDAxMTcKA2dpbwoEbW9oYQoBNQoCMTIKBTk4NzczCgcyMDE3NDIy"
the tf I get is
{"??7f1acf61810da83210966b4f36765e96b1cd499b8926f4054ad978e78df7304a8fe035f6aa0a816c7f61e4fdd452383d9fe9d416b2db8a15bdd203fe67b9962f<", "init_transaction", "000117", "gio", "moha", "5", "12", "98773", "2017422"}
At the last how can I get the JSON of Transaction/AllTxs type?
Given the comment, it looks like you are trying to unmarshal the string as JSON, but as it stands, the string is not valid JSON; if it were, you would be able to unmarshal it as follows, ignoring the fact that the fields to not appear to match up with the desired structure;
package main
import (
"encoding/json"
"fmt"
)
type Transaction struct {
Bin string `json:"bin"`
Fun string `json:"fun"`
ID string `json:"id"`
Timestamp string `json:"timestamp"`
TraderA string `json:"traderA"`
TraderB string `json:"traderB"`
Seller string `json:"seller"`
PointAmount string `json:"pointAmount"`
PrevTransactionID string `json:"prevTransactionId"`
}
func main() {
snippet := `{
"bin": "7f1acf61810da83210966b4f36765e96b1cd499b8926f4054ad978e78df7304a8fe035f6aa0a816c7f61e4fdd452383d9fe9d416b2db8a15bdd203fe67b9962f",
"fun": "init_transaction",
"id": "000117",
"timestamp": "gio",
"traderA": "moha",
"traderB": "5",
"seller": "12",
"pointAmount": "98773",
"prevTransactionId": "2017422"
}`
t := Transaction{}
json.Unmarshal([]byte(snippet), &t)
fmt.Println(t)
}

Resources