How to replace json values from key-value map in GOlang - go

I have below json & I need to replace employee_names to emp_id from a map.
Tried this GolangPlay , but not sure how to replace values from the given map and error handling if the value is not present in given map.
Json data:
[
{
"dept": "IT",
"condition": {
"employee": [
"emp1"
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
"emp2",
"emp3"
]
}
}
]
Map data:
[{emp1 14325} {emp3 49184} {emp2 21518}]
Expected output:
[
{
"dept": "IT",
"condition": {
"employee": [
14325
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
21518,
49184
]
}
}
]
code :
Started with below code , but not sure how to use the given map to replace with error handling.
package main
import (
"encoding/json"
"fmt"
//"strconv"
//"log"
)
func main() {
jsonStr := `[
{
"dept": "IT",
"condition": {
"employee": [
"emp1"
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
"emp2",
"emp3"
]
}
}
]`
empMap := `[{emp1 14325} {emp3 49184} {emp2 21518}]`
type GetEmployee []struct {
Dept string `json:"dept"`
Condition struct {
Employee []string `json:"employee"`
} `json:"condition"`
}
var empResponse GetEmployee
unmarshallingError := json.Unmarshal([]byte(string(jsonStr)), &empResponse)
if unmarshallingError != nil {
fmt.Println(unmarshallingError.Error())
}
fmt.Println(empResponse)
fmt.Println(empMap)
for i := range empResponse {
fmt.Println(i)
}
}

Instead of storing the ids in an array of {ids: value}, it will be better for them to be in a map instead.
The above will range over the Employees' name and change it to the id. A check is made to see if there is a certain id key in the map.
for i, e := range empResponse {
fmt.Println(e)
for j,val := range empResponse[i].Condition.Employee {
if _, ok := ids[val]; ok {
empResponse[i].Condition.Employee[j] = ids[val]
}
}
}
Full code
package main
import (
"encoding/json"
"fmt"
//"strconv"
//"log"
)
func main() {
jsonStr := `[
{
"dept": "IT",
"condition": {
"employee": [
"emp1"
]
}
},
{
"dept": "HR",
"condition": {
"employee": [
"emp2",
"emp3"
]
}
}
]`
empMap := `{"emp1": "14325", "emp3": "49184", "emp2": "21518"}`
type GetEmployee []struct {
Dept string `json:"dept"`
Condition struct {
Employee []string `json:"employee"`
} `json:"condition"`
}
var empResponse GetEmployee
var ids map[string]string
unmarshallingError := json.Unmarshal([]byte(string(jsonStr)), &empResponse)
if unmarshallingError != nil {
fmt.Println(unmarshallingError.Error())
}
json.Unmarshal([]byte(empMap), &ids)
fmt.Println(empResponse)
fmt.Println(ids)
for i, e := range empResponse {
fmt.Println(e)
for j,val := range empResponse[i].Condition.Employee {
if _, ok := ids[val]; ok {
empResponse[i].Condition.Employee[j] = ids[val]
}
}
}
fmt.Println(empResponse)
}
playground
In the above, the id are strings since the name to be replaced are strings. Actually Employee are of type []string. If a string is to be replaced by an int, then Employee type needs to be changed to []interface{}.
playground

Related

goavro and original Go data structure

How can I recreate original data structure in Golang serialized with avro using goavro?
With this library https://github.com/hamba/avro it's quite easy.
out := SimpleRecord{}
err = avro.Unmarshal(schema, data, &out)
type of variable out is SimpleRecord.
Let's say I have this struct and avro schema:
type SimpleRecord struct {
F1 int `avro:"f1"`
F2 string `avro:"f2"`
F3 string `avro:"f3"`
Dependencies []string `avro:"dependencies"`
}
func main() {
avro_schema_txt := `{
"type": "record",
"name": "AvroData",
"namespace": "data.avro",
"doc": "docstring",
"fields": [
{
"name": "f1",
"type": "int"
},
{
"name": "f2",
"type": "string"
},
{
"name": "f3",
"type": "string"
},
{
"name": "dependencies",
"type": {
"type": "array",
"items": "string"
}
}
]
}`
}
and then
codec, err := goavro.NewCodec(avro_schema_txt)
if err != nil {
log.Fatal(err.Error())
}
out, _, err := codec.NativeFromBinary(data)
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(out)
where data is marshaled with avro, out is of type interface{}, so how can I "make" it SimpleRecord?
There are two ways you can do it, either by casting out and doing a 'manual mapping' or by using codec.TextualFromNative. Both approaches are shown below for completeness,
Approach 1
Cast out from interface{} to map[string]interface{} and retrieve the values
...
simpleMap := out.(map[string]interface{})
f1 := simpleMap["f1"]
f2 := simpleMap["f2"]
...
SimpleRecord {
F1: f1,
F2: f2,
...
}
Approach 2
Use TextualFromNative, the below code shows both encode decode process
var avro_schema_txt = `{
"type": "record",
"name": "AvroData",
"namespace": "data.avro",
"doc": "docstring",
"fields": [
{
"name": "f1",
"type": "int"
},
{
"name": "f2",
"type": "string"
},
{
"name": "f3",
"type": "string"
},
{
"name": "dependencies",
"type": {
"type": "array",
"items": "string"
}
}
]
}`
// added json to match field names in avro
type SimpleRecord struct {
F1 int `avro:"f1" json:"f1"`
F2 string `avro:"f2" json:"f2"`
F3 string `avro:"f3" json:"f3"`
Dependencies []string `avro:"dependencies" json:"dependencies"`
}
func encodeDecode() {
data := SimpleRecord{
F1: 1,
F2: "tester2",
F3: "tester3",
Dependencies: []string { "tester4", "tester5" },
}
codec, err := goavro.NewCodec(avro_schema_txt)
if err != nil {
log.Fatal(err.Error())
}
// encode
textualIn, err := json2.Marshal(data)
if err != nil {
log.Fatal(err.Error())
}
nativeIn, _, err := codec.NativeFromTextual(textualIn)
if err != nil {
log.Fatal(err.Error())
}
binaryIn, err := codec.BinaryFromNative(nil, nativeIn)
if err != nil {
log.Fatal(err.Error())
}
// decode
nativeOut, _, err := codec.NativeFromBinary(binaryIn)
if err != nil {
log.Fatal(err.Error())
}
textualOut, err := codec.TextualFromNative(nil, nativeOut)
if err != nil {
log.Fatal(err.Error())
}
var out = SimpleRecord{}
err = json2.Unmarshal(textualOut, &out)
if err != nil {
log.Fatal(err.Error())
}
if !reflect.DeepEqual(data, out) {
log.Fatal("should be equal")
}
}

Unmarshalling a complicated JSON ad bin it with a struct

I have the following JSOn response from a webhook call
{
"responseId": "d5c70d8b-e8ad-41df-bb3b-26b0e51d60ca-a14fa99c",
"queryResult": {
"queryText": "1111111111",
"parameters": {
"phone-number": "1111111111"
},
"allRequiredParamsPresent": true,
"fulfillmentText": "Thats great! You payment link has been sent to Gaf ( Mobile number 1111111111 )",
"fulfillmentMessages": [{
"text": {
"text": ["Thats great! You payment link has been sent to Far ( Mobile number 1111111111 )"]
}
}],
"outputContexts": [{
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_name",
"lifespanCount": 2,
"parameters": {
"name": ["Gar"],
"name.original": ["Gar"],
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}, {
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_number",
"lifespanCount": 4,
"parameters": {
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}, {
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_name_confirm",
"lifespanCount": 3,
"parameters": {
"name": ["Gaf"],
"name.original": ["Far"],
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}, {
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/__system_counters__",
"parameters": {
"no-input": 0.0,
"no-match": 0.0,
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}],
"intent": {
"name": "projects/open-prod-bot-pfgibi/agent/intents/d21f7be5-0f77-4cb8-9857-26ba04964317",
"displayName": "GetMobileNumber"
},
"intentDetectionConfidence": 1.0,
"languageCode": "en"
},
"originalDetectIntentRequest": {
"payload": {
}
},
"session": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829"
}
I wanted to extract out
"outputContexts": [{
"name": "projects/open-prod-bot-pfgibi/agent/sessions/80f2fb70-01d0-fc1d-200a-ccbae5572829/contexts/awaiting_name",
"lifespanCount": 2,
"parameters": {
"name": ["Gar"],
"name.original": ["Gar"],
"phone-number": "1111111111",
"phone-number.original": "1111111111"
}
}
out of this and bind this with a struct. But I couldn't do it. I am trying to loop it through the map as below
var f interface{}
json.Unmarshal(b, &f)
for k, v := range f.(map[string]interface{}) {
if k == "queryResultmap" {
fmt.Println(v)
}
}
but not working. I am new to Go. Tried few examples in google but since this one is a complicated JSON I am unable to do it. Please help
I would suggest you declare a struct with the fields you care about and unmarshal into that, but if you want to stick to interface{} try this:
m := f.(map[string]interface{})
r := m["queryResult"].(map[string]interface{})
fmt.Println(r["outputContext"])
https://play.golang.com/p/gA5wWcPyd6E
Using struct:
type OutputContext struct {
Name string `json:"name"`
LifespanCount int `json:"lifespanCount"`
Parameters struct {
Name []string `json:"name"`
NameOriginal []string `json:"name.original"`
PhoneNumber string `json:"phone-number"`
PhoneNumberOriginal string `json:"phone-number.original"`
NoInput float64 `json:"no-input"`
NoMatch float64 `json:"no-match"`
} `json:"parameters"`
}
type QueryResult struct {
OutputContexts []OutputContext `json:"outputContexts"`
}
// ...
var dest struct {
QueryResult QueryResult `json:"queryResult"`
}
if err := json.Unmarshal(data, &dest); err != nil {
panic(err)
}
for _, v := range dest.QueryResult.OutputContexts {
fmt.Printf("%+v\n", v)
}
https://play.golang.com/p/EfIugDQJ651

Unmarshalling nested json object from http request returns nil

I've been going through other similar questions here but I don't know what I'm doing wrong.
I am calling this API:
https://coronavirus-tracker-api.herokuapp.com/v2/locations
Which returns a JSON object like this one:
{
"latest": {
"confirmed": 272166,
"deaths": 11299,
"recovered": 87256
},
"locations": [
{
"id": 0,
"country": "Thailand",
"country_code": "TH",
"province": "",
"last_updated": "2020-03-21T06:59:11.315422Z",
"coordinates": {
"latitude": "15",
"longitude": "101"
},
"latest": {
"confirmed": 177,
"deaths": 1,
"recovered": 41
}
},
{
"id": 39,
"country": "Norway",
"country_code": "NO",
"province": "",
"last_updated": "2020-03-21T06:59:11.315422Z",
"coordinates": {
"latitude": "60.472",
"longitude": "8.4689"
},
"latest": {
"confirmed": 1463,
"deaths": 3,
"recovered": 1
}
}
]
}
So I have written a small program to parse it but I can only parse the outer object ("latest") while the inner array ("locations") always returns nil.
Code is here (even if TCP calls don't work on the playground):
https://play.golang.org/p/ma225d07iRA
and here:
package main
import (
"encoding/json"
"fmt"
"net/http"
"time"
)
type AutoGenerated struct {
Latest Latest `json:"latest"`
Locations []Locations `json:"locations"`
}
type Latest struct {
Confirmed int `json:"confirmed"`
Deaths int `json:"deaths"`
Recovered int `json:"recovered"`
}
type Coordinates struct {
Latitude string `json:"latitude"`
Longitude string `json:"longitude"`
}
type Locations struct {
ID int `json:"id"`
Country string `json:"country"`
CountryCode string `json:"country_code"`
Province string `json:"province"`
LastUpdated time.Time `json:"last_updated"`
Coordinates Coordinates `json:"coordinates"`
Latest Latest `json:"latest"`
}
var latestUrl = "https://coronavirus-tracker-api.herokuapp.com/v2/latest"
func getJson(url string, target interface{}) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
fmt.Println(err)
}
req.Header.Add("content-type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Println(err)
}
decoder := json.NewDecoder(res.Body)
var data AutoGenerated
err = decoder.Decode(&data)
if err != nil {
fmt.Println(err)
}
for i, loc := range data.Locations {
fmt.Printf("%d: %s", i, loc.Country)
}
defer res.Body.Close()
}
func main() {
var a AutoGenerated
getJson(latestUrl, &a)
}
The problem is that the endpoint https://coronavirus-tracker-api.herokuapp.com/v2/latest does not return locations. This is the response I get by calling it:
{
"latest": {
"confirmed": 304524,
"deaths": 12973,
"recovered": 91499
}
}
However if you call the correct endpoint https://coronavirus-tracker-api.herokuapp.com/v2/locations, it might work.

Protocol buffer serialization Golang

I am using DialogFlow V2 official GoLang SDK. In my webhook, I am returning a payload, which I'm obtaining using the function GetWebhookPayload().
This returns *google_protobuf4.Struct. I would like to turn this struct into a map[string]interface{}. How is this possible?
This is what the struct looks like when serialized:
"payload": {
"fields": {
"messages": {
"Kind": {
"ListValue": {
"values": [
{
"Kind": {
"StructValue": {
"fields": {
"title": {
"Kind": {
"StringValue": "Hi! How can I help?"
}
},
"type": {
"Kind": {
"StringValue": "message"
}
}
}
}
}
}
]
}
}
}
}
What I essentially need is for it to be serialized as such:
"payload": {
"messages": [
{
"title": "Hi! How can I help?",
"type": "message"
}
]
}
This can be solved using jsonpb.
package main
import (
"bytes"
"encoding/json"
"github.com/golang/protobuf/jsonpb"
)
func main() {
...
payload := qr.GetWebhookPayload()
b, marshaler := bytes.Buffer{}, jsonpb.Marshaler{}
if err := marshaler.Marshal(&b, payload.GetFields()["messages"]); err != nil {
// handle err
}
msgs := []interface{}{}
if err := json.Unmarshal(b.Bytes(), &msgs); err != nil {
// handle err
}
// msgs now populated
}

How to populate an arbitrary struct in Go?

I want to create a function (in the example code called gremlinResultToStruct()) that takes an arbitrary struct and populates the values from a database query. Please note that the "Properties" key in the GremlinResultStruct are arbitrary too (that is why it is a Properties map[string][]struct)
GOAL:
I want gremlinResultToStruct() to populate the given struct based on its keys if they are available in the database results.
WHY:
I have hundreds of structs like Foo and Bar in my codebase and I want to have a single function that populates the struct needed in the code instead of creating semi-duplicate code.
CODE:
My code looks like this, it also contains the question:
// Gremlin Query result struct
type GremlinResult struct {
AtType string `json:"#type"`
AtValue []struct {
AtType string `json:"#type"`
AtValue struct {
ID struct {
AtType string `json:"#type"`
AtValue int `json:"#value"`
} `json:"id"`
Label string `json:"label"`
Properties map[string][]struct { // <== IMPORTANT, CONTAINS A VALUE THAT IS THE EQUIVALENT OF THE KEY IN THE STRUCT (like Something, SomethingElse, EvenMoreSomethingElse or AndMoreSomethingElse)
AtType string `json:"#type"`
AtValue struct {
ID struct {
AtType string `json:"#type"`
AtValue int `json:"#value"`
} `json:"id"`
Label string `json:"label"`
Value string `json:"value"`
} `json:"#value"`
} `json:"#properties"`
} `json:"#value"`
} `json:"#value"`
}
type Foo struct {
Something string
SomethingElse bool
}
type Bar struct {
EvenMoreSomethingElse string
AndMoreSomethingElse int
}
func gremlinResultToStruct(result *GremlinResult, ???){ // What to do at ???
// How to itterate over the ??? key values (like: Something, SomethingElse, EvenMoreSomethingElse or AndMoreSomethingElse)
// How to populate:
// VARIABLE := Something, SomethingElse, EvenMoreSomethingElse or AndMoreSomethingElse
// result.AtValue[0].AtValue.Properties[ VARIABLE ][0].AtValue.Value
}
func main {
// Do stuff to get JSON results. HOW is not relevant for question
dbResults := doDbStuffToPopulateResuts()
// Create the results variable as gremlinResults struct
results := GremlinResult{}
// Unmarshall to this struct
err = json.Unmarshal(dbResults, &results)
// 1. Expected result should be "result-a" (see below)
// Create the Foo struct
foo := Foo{}
// HERE IS MY QUESTION ABOUT (see function above)
gremlinResultToStruct(&results, &foo)
// NOW foo SHOULD HAVE Something and SomethingElse set
fmt.Println(foo.Something, foo.SomethingElse)
// 2. Expected result should be "result-b" (see below)
// Create the Bar struct
bar := Bar{}
// HERE IS MY QUESTION ABOUT (see function above)
gremlinResultToStruct(&results, &bar)
// NOW foo SHOULD HAVE Something and SomethingElse set
fmt.Println(bar.EvenMoreSomethingElse, bar.AndMoreSomethingElse)
}
Example JSON results.
result-A:
{
"#type": "g:List",
"#value": [{
"#type": "g:Vertex",
"#value": {
"id": {
"#type": "g:Int64",
"#value": 200
},
"label": "key",
"properties": {
"SomeThing": [{ // <== IMPORTANT, same as struct field in Foo{}
"#type": "g:VertexProperty",
"#value": {
"id": {
"#type": "g:Int64",
"#value": 201
},
"value": true,
"label": "Write"
}
}],
"SomeThingElse": [{ // <== IMPORTANT, same as struct field in Foo{}
"#type": "g:VertexProperty",
"#value": {
"id": {
"#type": "g:Int64",
"#value": 202
},
"value": true,
"label": "Read"
}
}]
}
}
}]
}
result-B:
{
"#type": "g:List",
"#value": [{
"#type": "g:Vertex",
"#value": {
"id": {
"#type": "g:Int64",
"#value": 200
},
"label": "key",
"properties": {
"EvenMoreSomethingElse": [{ // <== IMPORTANT, same as struct field in Bar{}
"#type": "g:VertexProperty",
"#value": {
"id": {
"#type": "g:Int64",
"#value": 201
},
"value": true,
"label": "Write"
}
}],
"AndEvenMoreSomethingElse": [{ // <== IMPORTANT, same as struct field in Bar{}
"#type": "g:VertexProperty",
"#value": {
"id": {
"#type": "g:Int64",
"#value": 202
},
"value": true,
"label": "Read"
}
}]
}
}
}]
}

Resources