Protocol buffer serialization Golang - go

I am using DialogFlow V2 official GoLang SDK. In my webhook, I am returning a payload, which I'm obtaining using the function GetWebhookPayload().
This returns *google_protobuf4.Struct. I would like to turn this struct into a map[string]interface{}. How is this possible?
This is what the struct looks like when serialized:
"payload": {
"fields": {
"messages": {
"Kind": {
"ListValue": {
"values": [
{
"Kind": {
"StructValue": {
"fields": {
"title": {
"Kind": {
"StringValue": "Hi! How can I help?"
}
},
"type": {
"Kind": {
"StringValue": "message"
}
}
}
}
}
}
]
}
}
}
}
What I essentially need is for it to be serialized as such:
"payload": {
"messages": [
{
"title": "Hi! How can I help?",
"type": "message"
}
]
}

This can be solved using jsonpb.
package main
import (
"bytes"
"encoding/json"
"github.com/golang/protobuf/jsonpb"
)
func main() {
...
payload := qr.GetWebhookPayload()
b, marshaler := bytes.Buffer{}, jsonpb.Marshaler{}
if err := marshaler.Marshal(&b, payload.GetFields()["messages"]); err != nil {
// handle err
}
msgs := []interface{}{}
if err := json.Unmarshal(b.Bytes(), &msgs); err != nil {
// handle err
}
// msgs now populated
}

Related

goavro and original Go data structure

How can I recreate original data structure in Golang serialized with avro using goavro?
With this library https://github.com/hamba/avro it's quite easy.
out := SimpleRecord{}
err = avro.Unmarshal(schema, data, &out)
type of variable out is SimpleRecord.
Let's say I have this struct and avro schema:
type SimpleRecord struct {
F1 int `avro:"f1"`
F2 string `avro:"f2"`
F3 string `avro:"f3"`
Dependencies []string `avro:"dependencies"`
}
func main() {
avro_schema_txt := `{
"type": "record",
"name": "AvroData",
"namespace": "data.avro",
"doc": "docstring",
"fields": [
{
"name": "f1",
"type": "int"
},
{
"name": "f2",
"type": "string"
},
{
"name": "f3",
"type": "string"
},
{
"name": "dependencies",
"type": {
"type": "array",
"items": "string"
}
}
]
}`
}
and then
codec, err := goavro.NewCodec(avro_schema_txt)
if err != nil {
log.Fatal(err.Error())
}
out, _, err := codec.NativeFromBinary(data)
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(out)
where data is marshaled with avro, out is of type interface{}, so how can I "make" it SimpleRecord?
There are two ways you can do it, either by casting out and doing a 'manual mapping' or by using codec.TextualFromNative. Both approaches are shown below for completeness,
Approach 1
Cast out from interface{} to map[string]interface{} and retrieve the values
...
simpleMap := out.(map[string]interface{})
f1 := simpleMap["f1"]
f2 := simpleMap["f2"]
...
SimpleRecord {
F1: f1,
F2: f2,
...
}
Approach 2
Use TextualFromNative, the below code shows both encode decode process
var avro_schema_txt = `{
"type": "record",
"name": "AvroData",
"namespace": "data.avro",
"doc": "docstring",
"fields": [
{
"name": "f1",
"type": "int"
},
{
"name": "f2",
"type": "string"
},
{
"name": "f3",
"type": "string"
},
{
"name": "dependencies",
"type": {
"type": "array",
"items": "string"
}
}
]
}`
// added json to match field names in avro
type SimpleRecord struct {
F1 int `avro:"f1" json:"f1"`
F2 string `avro:"f2" json:"f2"`
F3 string `avro:"f3" json:"f3"`
Dependencies []string `avro:"dependencies" json:"dependencies"`
}
func encodeDecode() {
data := SimpleRecord{
F1: 1,
F2: "tester2",
F3: "tester3",
Dependencies: []string { "tester4", "tester5" },
}
codec, err := goavro.NewCodec(avro_schema_txt)
if err != nil {
log.Fatal(err.Error())
}
// encode
textualIn, err := json2.Marshal(data)
if err != nil {
log.Fatal(err.Error())
}
nativeIn, _, err := codec.NativeFromTextual(textualIn)
if err != nil {
log.Fatal(err.Error())
}
binaryIn, err := codec.BinaryFromNative(nil, nativeIn)
if err != nil {
log.Fatal(err.Error())
}
// decode
nativeOut, _, err := codec.NativeFromBinary(binaryIn)
if err != nil {
log.Fatal(err.Error())
}
textualOut, err := codec.TextualFromNative(nil, nativeOut)
if err != nil {
log.Fatal(err.Error())
}
var out = SimpleRecord{}
err = json2.Unmarshal(textualOut, &out)
if err != nil {
log.Fatal(err.Error())
}
if !reflect.DeepEqual(data, out) {
log.Fatal("should be equal")
}
}

How to replace json values from key-value map in GOlang

I have below json & I need to replace employee_names to emp_id from a map.
Tried this GolangPlay , but not sure how to replace values from the given map and error handling if the value is not present in given map.
Json data:
[
{
"dept": "IT",
"condition": {
"employee": [
"emp1"
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
"emp2",
"emp3"
]
}
}
]
Map data:
[{emp1 14325} {emp3 49184} {emp2 21518}]
Expected output:
[
{
"dept": "IT",
"condition": {
"employee": [
14325
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
21518,
49184
]
}
}
]
code :
Started with below code , but not sure how to use the given map to replace with error handling.
package main
import (
"encoding/json"
"fmt"
//"strconv"
//"log"
)
func main() {
jsonStr := `[
{
"dept": "IT",
"condition": {
"employee": [
"emp1"
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
"emp2",
"emp3"
]
}
}
]`
empMap := `[{emp1 14325} {emp3 49184} {emp2 21518}]`
type GetEmployee []struct {
Dept string `json:"dept"`
Condition struct {
Employee []string `json:"employee"`
} `json:"condition"`
}
var empResponse GetEmployee
unmarshallingError := json.Unmarshal([]byte(string(jsonStr)), &empResponse)
if unmarshallingError != nil {
fmt.Println(unmarshallingError.Error())
}
fmt.Println(empResponse)
fmt.Println(empMap)
for i := range empResponse {
fmt.Println(i)
}
}
Instead of storing the ids in an array of {ids: value}, it will be better for them to be in a map instead.
The above will range over the Employees' name and change it to the id. A check is made to see if there is a certain id key in the map.
for i, e := range empResponse {
fmt.Println(e)
for j,val := range empResponse[i].Condition.Employee {
if _, ok := ids[val]; ok {
empResponse[i].Condition.Employee[j] = ids[val]
}
}
}
Full code
package main
import (
"encoding/json"
"fmt"
//"strconv"
//"log"
)
func main() {
jsonStr := `[
{
"dept": "IT",
"condition": {
"employee": [
"emp1"
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
"emp2",
"emp3"
]
}
}
]`
empMap := `{"emp1": "14325", "emp3": "49184", "emp2": "21518"}`
type GetEmployee []struct {
Dept string `json:"dept"`
Condition struct {
Employee []string `json:"employee"`
} `json:"condition"`
}
var empResponse GetEmployee
var ids map[string]string
unmarshallingError := json.Unmarshal([]byte(string(jsonStr)), &empResponse)
if unmarshallingError != nil {
fmt.Println(unmarshallingError.Error())
}
json.Unmarshal([]byte(empMap), &ids)
fmt.Println(empResponse)
fmt.Println(ids)
for i, e := range empResponse {
fmt.Println(e)
for j,val := range empResponse[i].Condition.Employee {
if _, ok := ids[val]; ok {
empResponse[i].Condition.Employee[j] = ids[val]
}
}
}
fmt.Println(empResponse)
}
playground
In the above, the id are strings since the name to be replaced are strings. Actually Employee are of type []string. If a string is to be replaced by an int, then Employee type needs to be changed to []interface{}.
playground

Unmarshalling a complicated JSON ad bin it with a struct

I have the following JSOn response from a webhook call
{
"responseId": "d5c70d8b-e8ad-41df-bb3b-26b0e51d60ca-a14fa99c",
"queryResult": {
"queryText": "1111111111",
"parameters": {
"phone-number": "1111111111"
},
"allRequiredParamsPresent": true,
"fulfillmentText": "Thats great! You payment link has been sent to Gaf ( Mobile number 1111111111 )",
"fulfillmentMessages": [{
"text": {
"text": ["Thats great! You payment link has been sent to Far ( Mobile number 1111111111 )"]
}
}],
"outputContexts": [{
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_name",
"lifespanCount": 2,
"parameters": {
"name": ["Gar"],
"name.original": ["Gar"],
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}, {
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_number",
"lifespanCount": 4,
"parameters": {
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}, {
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_name_confirm",
"lifespanCount": 3,
"parameters": {
"name": ["Gaf"],
"name.original": ["Far"],
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}, {
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/__system_counters__",
"parameters": {
"no-input": 0.0,
"no-match": 0.0,
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}],
"intent": {
"name": "projects/open-prod-bot-pfgibi/agent/intents/d21f7be5-0f77-4cb8-9857-26ba04964317",
"displayName": "GetMobileNumber"
},
"intentDetectionConfidence": 1.0,
"languageCode": "en"
},
"originalDetectIntentRequest": {
"payload": {
}
},
"session": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829"
}
I wanted to extract out
"outputContexts": [{
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_name",
"lifespanCount": 2,
"parameters": {
"name": ["Gar"],
"name.original": ["Gar"],
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}
out of this and bind this with a struct. But I couldn't do it. I am trying to loop it through the map as below
var f interface{}
json.Unmarshal(b, &f)
for k, v := range f.(map[string]interface{}) {
if k == "queryResultmap" {
fmt.Println(v)
}
}
but not working. I am new to Go. Tried few examples in google but since this one is a complicated JSON I am unable to do it. Please help
I would suggest you declare a struct with the fields you care about and unmarshal into that, but if you want to stick to interface{} try this:
m := f.(map[string]interface{})
r := m["queryResult"].(map[string]interface{})
fmt.Println(r["outputContext"])
https://play.golang.com/p/gA5wWcPyd6E
Using struct:
type OutputContext struct {
Name string `json:"name"`
LifespanCount int `json:"lifespanCount"`
Parameters struct {
Name []string `json:"name"`
NameOriginal []string `json:"name.original"`
PhoneNumber string `json:"phone-number"`
PhoneNumberOriginal string `json:"phone-number.original"`
NoInput float64 `json:"no-input"`
NoMatch float64 `json:"no-match"`
} `json:"parameters"`
}
type QueryResult struct {
OutputContexts []OutputContext `json:"outputContexts"`
}
// ...
var dest struct {
QueryResult QueryResult `json:"queryResult"`
}
if err := json.Unmarshal(data, &dest); err != nil {
panic(err)
}
for _, v := range dest.QueryResult.OutputContexts {
fmt.Printf("%+v\n", v)
}
https://play.golang.com/p/EfIugDQJ651

Unmarshalling nested json object from http request returns nil

I've been going through other similar questions here but I don't know what I'm doing wrong.
I am calling this API:
https://coronavirus-tracker-api.herokuapp.com/v2/locations
Which returns a JSON object like this one:
{
"latest": {
"confirmed": 272166,
"deaths": 11299,
"recovered": 87256
},
"locations": [
{
"id": 0,
"country": "Thailand",
"country_code": "TH",
"province": "",
"last_updated": "2020-03-21T06:59:11.315422Z",
"coordinates": {
"latitude": "15",
"longitude": "101"
},
"latest": {
"confirmed": 177,
"deaths": 1,
"recovered": 41
}
},
{
"id": 39,
"country": "Norway",
"country_code": "NO",
"province": "",
"last_updated": "2020-03-21T06:59:11.315422Z",
"coordinates": {
"latitude": "60.472",
"longitude": "8.4689"
},
"latest": {
"confirmed": 1463,
"deaths": 3,
"recovered": 1
}
}
]
}
So I have written a small program to parse it but I can only parse the outer object ("latest") while the inner array ("locations") always returns nil.
Code is here (even if TCP calls don't work on the playground):
https://play.golang.org/p/ma225d07iRA
and here:
package main
import (
"encoding/json"
"fmt"
"net/http"
"time"
)
type AutoGenerated struct {
Latest Latest `json:"latest"`
Locations []Locations `json:"locations"`
}
type Latest struct {
Confirmed int `json:"confirmed"`
Deaths int `json:"deaths"`
Recovered int `json:"recovered"`
}
type Coordinates struct {
Latitude string `json:"latitude"`
Longitude string `json:"longitude"`
}
type Locations struct {
ID int `json:"id"`
Country string `json:"country"`
CountryCode string `json:"country_code"`
Province string `json:"province"`
LastUpdated time.Time `json:"last_updated"`
Coordinates Coordinates `json:"coordinates"`
Latest Latest `json:"latest"`
}
var latestUrl = "https://coronavirus-tracker-api.herokuapp.com/v2/latest"
func getJson(url string, target interface{}) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
fmt.Println(err)
}
req.Header.Add("content-type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Println(err)
}
decoder := json.NewDecoder(res.Body)
var data AutoGenerated
err = decoder.Decode(&data)
if err != nil {
fmt.Println(err)
}
for i, loc := range data.Locations {
fmt.Printf("%d: %s", i, loc.Country)
}
defer res.Body.Close()
}
func main() {
var a AutoGenerated
getJson(latestUrl, &a)
}
The problem is that the endpoint https://coronavirus-tracker-api.herokuapp.com/v2/latest does not return locations. This is the response I get by calling it:
{
"latest": {
"confirmed": 304524,
"deaths": 12973,
"recovered": 91499
}
}
However if you call the correct endpoint https://coronavirus-tracker-api.herokuapp.com/v2/locations, it might work.

goavro not able to validate json data with schema

I'm new to go and avro and struggling with validating data.I have this avro schema
{
"namespace": "com.input",
"name": "parent",
"type": "record",
"fields": [
{
"name": "field1",
"type": [
"null",
{
"type": "record",
"name": "child",
"fields": [
{
"name": "child1",
"type": "string"
},
{
"name": "child2",
"type": "string"
}
]
}
]
}
]
}
Data to be validated:
{
"field1": {
"child1": "1",
"child2": "abc"
}
}
This is the code which i'm using to validate using goavro library:
func loadMockData() (stringFormatData string) {
mockData, err := ioutil.ReadFile("sample.json")
if err != nil {
log.Println(err)
}
return string(mockData)
}
func loadSchema() (stringFormatData string) {
mockSchema, err := ioutil.ReadFile("schema.avsc")
if err != nil {
log.Println(err)
}
return string(mockSchema)
}
avroSchema := loadSchema()
jsonString := loadMockData()
codec, err := goavro.NewCodec(avroSchema)
decoded, _, err := codec.NativeFromTextual([]byte(jsonString))
This gives me following error:
NativeFromTextual error: cannot decode textual record "com.input.parent": cannot decode textual union: cannot decode textual map: cannot determine codec: "parentid"
I tried debugging the goavro library, seems that fieldcodec remains nil for parentid in map.go. With simple fields this setting works fine. Problem is with nested data its not able to get codec for parentid.
Any help is much appreciated.

Resources