Json data adapter - go

I want to take JSON in
{ name: "http://example.com", ...}
and map it to
{ url: "http://example.com", ...}
I want to do this mapping and be able to Marshal back to json as Url. The struct is rather large, so is it possible to do this with one struct instead creating multiple struct types?
This is the solution I have come to so far: http://play.golang.org/p/wBPbSjkTYF
After seeing answers and how I failed to accurately describe my use case. Updating to the actual mapping I'm trying to do:
{ "data": { "name": "http://example.com", "key": "value" } }
To:
{ "url": "http://example.com", { "data": "key": "value" } }

The json package doesn't have any tags that allows you to use one key name of a field on Marshal and a different on Unmarshal.
It is still possible to Marshal a struct to differently, but in order to do that you will have to implement your own Marshaller (MarshalJSON function). A working example:
package main
import (
"bytes"
"encoding/json"
"fmt"
)
type Data struct {
Url string `json:"name"`
}
// marshalObject takes a slice of key values
// (should be correctly escaped json strings without "")
// and another slice of interface{} values and marshals
// them into a JSON object string
func marshalObject(keys []string, values []interface{}) ([]byte, error) {
if len(keys) != len(values) {
panic("Different length of keys and values slices")
}
if len(keys) == 0 {
return []byte(`{}`), nil
}
var b bytes.Buffer
b.Write([]byte(`{"`))
for i, key := range keys {
if i != 0 {
b.Write([]byte(`,"`))
}
b.WriteString(key)
b.Write([]byte(`":`))
j, err := json.Marshal(values[i])
if err != nil {
return nil, err
}
b.Write(j)
}
b.Write([]byte(`}`))
return b.Bytes(), nil
}
func (d *Data) MarshalJSON() ([]byte, error) {
// Here you can add a list of keys and values.
// Currently it is only `url` mapped to d.Url
return marshalObject(
[]string{
`url`,
},
[]interface{}{
d.Url,
},
)
}
func main() {
i := []byte(`{"name":"http://example.com"}`)
fmt.Printf("Json Input: %+s\n", i)
var d *Data
err := json.Unmarshal(i, &d)
if err != nil {
panic(err)
}
fmt.Printf("Data: %#v\n", d)
o, err := json.Marshal(d)
if err != nil {
panic(err)
}
fmt.Printf("Json Output: %+s\n", o)
}
Result:
Json Input: {"name":"http://example.com"}
Data: &main.Data{Url:"http://example.com"}
Json Output: {"url":"http://example.com"}
Playground:
http://play.golang.org/p/u6ExI9V95D

If you unmarshal a JSON object into an map[string]interface{} variable, you will be given everything in the object rather than just a few fields. It should be fairly straight forward to make the required modifications, like so:
func nameToUrl(data []byte) ([]byte, error) {
var obj map[string]interface{}
if err := json.Unmarshal(data, &obj); err != nil {
return nil, err
}
if name, ok := obj["name"]; ok {
delete(obj, "name")
obj["url"] = name
}
return json.Marshal(obj)
}
You can experiment with this here: http://play.golang.org/p/0jz_HAGg3E

Related

Cast an interface using a pointer of a Slice to iterate its values

Context:
Using a custom Binder and Validator in Echo Framework. The binder use a signature of (interface{}, echo.Context), but a pointer is always passed and checked by echo.DefaultBinder.
I'm having trouble validating an array of struct for some reason, when an array is passed for some unknown reason. Therefore, I'm trying to validate each elements in the interface, if this interface is an Array or a Slice.
Problem:
I cannot find a way to both cast the interface to a value instead of a pointer and iterate trough the values of this array to validate each of them.
My Code so far:
func (cb *CustomBinder) Bind(i interface{}, c echo.Context) error {
db := new(echo.DefaultBinder)
validate := validator.New()
if err := db.Bind(i, c); err != nil {
return err
}
kind := reflect.ValueOf(i).Elem().Kind()
if kind == reflect.Array || kind == reflect.Slice {
// ... Iteration and Validation
} else {
if err := validate.Struct(i); err != nil {
return err
}
}
return nil
}
I would rather use type assertions than reflection because reflection is slow in terms of performance and not friendly to use.
To illustrate what I mean, check this example code where I have a function that accepts an argument of type interface{} and prints the values according to the data type,
package main
import (
"fmt"
)
func main() {
var dynamicValue interface{}
dynamicValue = []interface{}{"value1", "value2"}
printValue(dynamicValue)
dynamicValue = map[string]interface{}{"key1": "value1"}
printValue(dynamicValue)
dynamicValue = "value1"
printValue(dynamicValue)
}
func printValue(i interface{}) {
if arrayValue, isArray := i.([]interface{}); isArray {
for index, value := range arrayValue {
fmt.Printf("Index: %d Value: %v \n", index, value)
}
} else if mapValue, isMap := i.(map[string]interface{}); isMap {
for key, value := range mapValue {
fmt.Printf("Key: %s Value: %v \n", key, value)
}
} else if stringValue, isString := i.(string); isString {
fmt.Println(stringValue)
} else {
fmt.Println("Invalid data type! Only supports string, arrays and maps ")
}
}
Output:
Index: 0 Value: value1
Index: 1 Value: value2
Key: key1 Value: value1
value1
Playground: https://play.golang.org/p/TMfojVdoi5b
You can use this logic of type assertion in your code to check if the interface is a slice and iterate over it for validation.
Something like this,
func (cb *CustomBinder) Bind(i interface{}, c echo.Context) error {
db := new(echo.DefaultBinder)
validate := validator.New()
if err := db.Bind(i, c); err != nil {
return err
}
if arrayValue, isArray := i.([]interface{}); isArray {
// Iteration
for index, value := range arrayValue {
// Validation
}
} else {
if err := validate.Struct(i); err != nil {
return err
}
}
return nil
}

How to receive a json to insert

I am receiving some values per post and I have a json type field but it arrives empty and if I enter a normal text it works and I do not see the error in the field
the model was updated so that it receives the fields and allows inserting in mysql
POSTman
{
"Code":"1234",//it works
"Desc":"desc",//it works
"Config":{"link":"https://stackoverflow.com/" }, //not works
"Dev":[ {"item":1},{"item":2}]//not works
}
type User struct {
gorm.Model
Code string `gorm:"type:varchar(100);unique_index"`
Desc string `gorm:"type:varchar(255);"`
Config JSON `json:"currencies" gorm:"type:varchar(255);"`
Dev JSON `json:"currencies" gorm:"type:varchar(255);"`
}
func CreateUser(c *gin.Context) {
var usuario models.User
var bodyBytes []byte
if c.Request.Body != nil {
bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
}
data := bytes.NewBuffer(bodyBytes)
fmt.Println(data.Config)
c.BindJSON(&usuario)
db.DB.Create(&usuario)
c.JSON(200, usuario)
}
Model update. receive post form with json fields and insert in mysql
package models
import (
"bytes"
"database/sql/driver"
"errors"
)
type JSON []byte
func (j JSON) Value() (driver.Value, error) {
if j.IsNull() {
return nil, nil
}
return string(j), nil
}
func (j *JSON) Scan(value interface{}) error {
if value == nil {
*j = nil
return nil
}
s, ok := value.([]byte)
if !ok {
errors.New("error")
}
*j = append((*j)[0:0], s...)
return nil
}
func (m JSON) MarshalJSON() ([]byte, error) {
if m == nil {
return []byte("null"), nil
}
return m, nil
}
func (m *JSON) UnmarshalJSON(data []byte) error {
if m == nil {
return errors.New("error")
}
*m = append((*m)[0:0], data...)
return nil
}
func (j JSON) IsNull() bool {
return len(j) == 0 || string(j) == "null"
}
func (j JSON) Equals(j1 JSON) bool {
return bytes.Equal([]byte(j), []byte(j1))
}
Thank you very much to everyone who helped me, I consider that the functionality of receiving a json and saving it in mysql is very common and this can be useful to many people
You can change the JSON like below or You can change the Struct like below (I prefer struct approach)
{
"Code": "1234",
"Desc": "desc",
"Config": {
"Link": "https://stackoverflow.com/"
},
"Dev": [
{
"Item": 1
},
{
"Item": 2
}
]
}
Struct:
type User struct {
gorm.Model
Code string `json:"Code" gorm:"type:varchar(100);unique_index"`
Desc string `json:"Desc" gorm:"type:varchar(255);"`
Config []struct {
Link string `json:"link" gorm:"type:varchar(255);"`
Title string `json:"title" gorm:"type:varchar(255);"`
}
Dev []struct {
Item string `json:"item" gorm:"type:varchar(255);"`
}
}
You have made two kind of mistakes
Your json decoding cannot work because your struct does not match your json. Config is defined as a array of something but in your json you have an object not array, and in Dev the property Item is a int not a string
Your model may not be well defined as you have not defined you joined table. Well I never seen a working example with this kind of definition. I suggest you to declare your nested struct as independent struct.
Here a full working example :
package main
import (
"database/sql"
"encoding/json"
"fmt"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
const data = `{
"Code":"1234",
"Desc":"desc",
"Config":{"link":"https://stackoverflow.com/" },
"Dev":[ {"item":1},{"item":2}]
}`
type Config struct {
Id int `gorm:"primaryKey"`
Link string `json:"link"`
Title string
UserId int
}
type Item struct {
Id int `gorm:"primaryKey"`
Item int `json:"item"`
UserId int
}
type User struct {
Id int `gorm:"primaryKey"`
Code string
Desc string
Config Config `gorm:"foreignkey:UserId"`
Dev []Item `gorm:"foreignkey:UserId"`
}
func initDb(url string) (*gorm.DB, *sql.DB, error) {
connexion := sqlite.Open(url)
db, err := gorm.Open(connexion, &gorm.Config{})
if err != nil {
return nil, nil, err
}
sql, err := db.DB()
if err != nil {
return nil, nil, err
}
err = db.AutoMigrate(&User{})
if err != nil {
return nil, nil, err
}
err = db.AutoMigrate(&Item{})
if err != nil {
return nil, nil, err
}
err = db.AutoMigrate(&Config{})
if err != nil {
return nil, nil, err
}
return db, sql, nil
}
func run() error {
db, sql, err := initDb("file::memory:?cache=shared")
if err != nil {
return err
}
defer sql.Close()
var user User
err = json.Unmarshal([]byte(data), &user)
fmt.Printf("%#v\n", user)
err = db.Create(&user).Error
if err != nil {
return err
}
var loaded User
db.Preload("Config").Preload("Dev").First(&loaded)
fmt.Printf("%#v\n", loaded)
return nil
}
func main() {
if err := run(); err != nil {
fmt.Println("failed", err)
}
}
try adding this JSON Field in your model
import (
"errors"
"database/sql/driver"
"encoding/json"
)
// JSON Interface for JSON Field of yourTableName Table
type JSON interface{}
// Value Marshal
func (a JSON) Value() (driver.Value, error) {
return json.Marshal(a)
}
// Scan Unmarshal
func (a *JSON) Scan(value interface{}) error {
b, ok := value.([]byte)
if !ok {
return errors.New("type assertion to byte failed")
}
return json.Unmarshal(b,&a)
}
All these answers didn't work for me, but this will work for everyone
Model
// This is the max Thing you need
import "gorm.io/datatypes"
import "encoding/json"
type CMSGenericModel struct {
gorm.Model
//... Other Posts
ExtraData datatypes.JSON `json:"data"`
}
In Handler Function
type CmsReqBody struct {
// ============= RAW ========
Data json.RawMessage `json:"data"`
// other props...
}
cmsBodyRecord := new(models.CMSGenericModel)
cmsBodyPayload := new(CmsReqBody)
if err := c.BodyParser(cmsBodyPayload); err != nil {
return c.Status(503).SendString(err.Error())
}
cmsBodyRecord.ExtraData = datatypes.JSON(cmsBodyPayload.Data)
My Sample Data
{
"title": "Blog Post 1",
"subtitle": "first",
"description": "Updated",
"type": "blog",
"isActive": true,
"uuid": "new",
"data": {
"complex1": ["kkkk", "yyyy"],
"complex2": [
{
"name": "sourav"
},
{
"name": "yahooo"
},
{
"yahoo": "name",
"kjk": ["abbsb", {"data": "abcd"}]
}
]
}
}

What's the purpose of gob.Register method?

I have read the documentation of ( gob) and I have some problems :
Now I know how to encode structure and decode like that:
func main() {
s1 := &S{
Field1: "Hello Gob",
Field2: 999,
}
log.Println("Original value:", s1)
buf := new(bytes.Buffer)
err := gob.NewEncoder(buf).Encode(s1)
if err != nil {
log.Println("Encode:", err)
return
}
s2 := &S{}
err = gob.NewDecoder(buf).Decode(s2)
if err != nil {
log.Println("Decode:", err)
return
}
log.Println("Decoded value:", s2)
}
But I don't know the purpose of this method gob.Register() can someone explain to me when to use it and why?
If you're dealing with concrete types (structs) only, you don't really need it. Once you're dealing with interfaces you must register your concrete type first.
For example, let's assume we have these struct and interface (the struct implements the interface):
type Getter interface {
Get() string
}
type Foo struct {
Bar string
}
func (f Foo)Get() string {
return f.Bar
}
To send a Foo over gob as a Getter and decode it back, we must first call
gob.Register(Foo{})
So the flow would be:
// init and register
buf := bytes.NewBuffer(nil)
gob.Register(Foo{})
// create a getter of Foo
g := Getter(Foo{"wazzup"})
// encode
enc := gob.NewEncoder(buf)
enc.Encode(&g)
// decode
dec := gob.NewDecoder(buf)
var gg Getter
if err := dec.Decode(&gg); err != nil {
panic(err)
}
Now try removing the Register and this won't work because gob wouldn't know how to map things back to their appropriate type.
As http://golang.org/pkg/encoding/gob/#Register said:
Only types that will be transferred as implementations of interface
values need to be registered.
So it doesn't needed by your demo.
If you want to encode / decode a map[string]interface{}, since the field of the map is enclosed as interface type, then we need to register the specific type before.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type SomeStruct struct {
Text string
}
func main() {
var bytes bytes.Buffer
// Remove one of these, then the decoding will produce error
gob.Register(SomeStruct{})
gob.Register([]interface{}{})
gob.Register([]SomeStruct{})
gob.Register(map[string]SomeStruct{})
writer := gob.NewEncoder(&bytes)
err := writer.Encode(map[string]interface{}{
"SomeStruct": SomeStruct{"Halo"},
"SomeSlice": []interface{}{},
"SomeSliceStruct": []SomeStruct{
{
Text: "SomeText",
},
},
"SomeMapStruct": map[string]SomeStruct{
"S": {"Test"},
},
})
if err != nil {
log.Fatalf("Error on encode process: %v\n", err)
return
}
reader := gob.NewDecoder(&bytes)
var aMap map[string]interface{}
err = reader.Decode(&aMap)
if err != nil {
log.Fatalf("Error on decode process: %v\n", err)
return
}
fmt.Printf("Decode is successful: %+v\n", aMap)
}

How do I convert this cache item back to a slice of maps?

I'm still new to Go and trying to use Beego's cache. I can put a []map[string]string into the cache but can't figure out how to convert the value back to a []map[string]string.
For instance, to put the item in the cache:
m:=make([]map[string]string)
// add items to the slice of maps
.......
// cache it
if err := c.Put("key", m, 100); err != nil {
fmt.Println(err)
}
// retrieve it
n := c.Get("key")
fmt.Println(reflect.TypeOf(n)) // ==>string
// failed attempt
a := n.([]map[string]string)
fmt.Println(a) // panic: interface conversion: interface is string, not []map[string]string
How do I convert n to a slice of maps?
well digging into the code seems like even if it says interface{} what it does it actually squash everything to []byte
https://github.com/astaxie/beego/blob/master/cache/memcache/memcache.go#L77
and when it Does the Get convert everything to string
https://github.com/astaxie/beego/blob/master/cache/memcache/memcache.go#L61
So you need to Marshal / Unmarshal the data structure yourself.
See this example and substitute your calls and error check with the dummies one provided:
http://play.golang.org/p/9z3KcOlgAx
package main
import (
"bytes"
"encoding/json"
"log"
)
var cache map[string]string = make(map[string]string)
func Put(key, value string) {
cache[key] = value
}
func Get(key string) string {
return cache[key]
}
func main() {
m := map[string]string{
"A": "1",
"B": "2",
}
if b, err := json.Marshal(m); err != nil {
log.Fatal(err)
} else {
Put("myKey", string(b))
}
b := bytes.NewBufferString(Get("myKey"))
var mm map[string]string
if err := json.Unmarshal(b.Bytes(), &mm); err != nil {
log.Fatal(err)
}
log.Printf("%#v", mm)
}

Index out of Range with array of structs in Go

I am new to Go so hopefully I'm making myself clear with this issue I'm having. My problem is that I am trying to iterate over an array of structs but I keep running into an index out of range issue. For the purposes of this problem, I have already verified that my array is not empty but that it in fact does contain at least one Services struct and file_content is the string that contains my valid JSON
Here is the snippet of code that represents the problem I'm having:
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
err_json := json.Unmarshal(file_content, &ServiceConf)
for _, s := range ServiceConf.Services {
log.Println(s)
}
So every time I run my code I get:
2014/03/14 18:19:53 http: panic serving [::1]:65448: runtime error: index out of range
{
"services" : [
{
"name": "translation",
"command": "to german",
"request": {
"key": "XXX",
"url": "https://www.googleapis.com/language/translate/v2?"
}
}
]
}
If you're interested in the complete source file:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
)
type SlackResponse struct {
token string
team_id string
channel_id string
channel_name string
timestamp string
user_id string
user_name string
text string
}
type service_config struct {
Services []struct {
Name string
Command string
Request map[string]interface{}
}
}
var ServiceConf = service_config{}
func main() {
content, err_read := ioutil.ReadFile("config.ini")
if err_read != nil {
log.Println("Could not read config")
return
}
log.Println(string(content))
err_json := json.Unmarshal(content, &ServiceConf)
if err_json != nil {
log.Println(err_json)
}
http.HandleFunc("/", handler)
http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
slack_response := SlackResponse{
r.FormValue("token"),
r.FormValue("team_id"),
r.FormValue("channel_id"),
r.FormValue("channel_name"),
r.FormValue("timestamp"),
r.FormValue("user_id"),
r.FormValue("user_name"),
r.FormValue("text"),
}
// log.Println(ServiceConf.Services[0].Request["key"])
// loop through services to find command phrases
for _, s := range ServiceConf.Services {
log.Println(s)
}
if slack_response.user_name == "slackbot" {
return
}
// fmt.Fprintf(w, "{ \"text\": \"Master %s! You said: '%s'\" }", slack_response.user_name, slack_response.text)
content, err := getContent("https://www.googleapis.com/language/translate/v2?key=&source=en&target=de&q=" + url.QueryEscape(slack_response.text))
if err != nil {
fmt.Fprintf(w, "{ \"text\": \"Huh?!\" }")
} else {
type trans struct {
Data struct {
Translations []struct {
TranslatedText string `json:"translatedText"`
} `json:"translations"`
} `json:"data"`
}
f := trans{}
err := json.Unmarshal(content, &f)
if err != nil {
log.Println(err)
}
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }", f.Data.Translations[0].TranslatedText)
}
}
// array of bytes if retrieved successfully.
func getContent(url string) ([]byte, error) {
// Build the request
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
// Send the request via a client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
// Defer the closing of the body
defer resp.Body.Close()
// Read the content into a byte array
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
// At this point we're done - simply return the bytes
return body, nil
}
Here is the stack trace:
2014/03/21 23:21:29 http: panic serving [::1]:59508: runtime error: index out of range
goroutine 3 [running]:
net/http.funcĀ·009()
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x215f80, 0x4b6537)
/usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/panic.c:248 +0x106
main.handler(0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/Users/et/src/go/src/github.com/etdebruin/gojacques/main.go:100 +0x81b
net/http.HandlerFunc.ServeHTTP(0x2cbc60, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc21001e5d0, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1496 +0x163
net/http.serverHandler.ServeHTTP(0xc21001f500, 0x5a85e8, 0xc21000f6e0, 0xc210037dd0)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc210058300)
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1644 +0x28b
The error comes from this line
fmt.Fprintf(w, "{ \"text\": \"Translated to German you said: '%s'\" }",
f.Data.Translations[0].TranslatedText)
So you didn't get any Translations back - that array is empty.
You might want to check resp.Status to see if an error was returned. This isn't returned as an error - you need to check it yourself.

Resources