Is there something like sql.NullJson akin to sql.NullString in golang? - go

I am querying from postgres using golang, one of the field contains json that can sometimes be NULL
Like this
row := db.QueryRow(
"select my_string, my_json from my_table where my_string = $1",
my_id)
var my_string sql.NullString
var myjson MyJsonStruct
err := row.Scan(&my_string2, &myjson)
But I am getting
sql: Scan error on column index 2, name "my_json": unsupported Scan, storing driver.Value type <nil> into type *main.MyJsonStruct
I checked https://godoc.org/database/sql but didn't find sql.NullJson What is a go way of dealing with this situation?

No, there is no sql.json. I think the best way to dealing with json column in db is to implement valuer and scanner. so something like this :
// Scan implements database/sql.Scanner interface
func (m *MyJsonStruct) Scan(src interface{}) error {
if src == nil {
return nil
}
data, ok := src.([]byte)
if !ok {
return errors.New("type assertion to []byte failed")
}
var myJsonStruct MyJsonStruct
if err := json.Unmarshal(data, &myJsonStruct); err != nil {
return fmt.Errorf("unmarshal myJsonStruct: %w", err)
}
*m = myJsonStruct
return nil
}
// Value implements database/sql/driver.Valuer interface
func (m MyJsonStruct) Value() (driver.Value, error) {
data, err := json.Marshal(m)
if err != nil {
return nil, fmt.Errorf("marshal myJsonStruct: %w", err)
}
return data, nil
}

Related

How can I return two different concrete types from a single method in Go 1.18?

Let say that I have this code:
type Type1 struct {
Name string `json:"name,omitempty"`
Path string `json:"path"`
File string `json:"file"`
Tag int `json:"tag"`
Num int `json:"num"`
}
func LoadConfiguration(data []byte) (*Type1, error) {
config, err := loadConf1(data)
if err != nil {
return nil, err
}
confOther, err := loadConfOther1()
if err != nil {
return nil, err
}
// do something with confOther
fmt.Println("confOther", confOther)
if confOther.Tag == 0 {
config.Num = 5
}
// do something with config attributes of type1
if config.Tag == 0 {
config.Tag = 5
}
if config.Num == 0 {
config.Num = 4
}
return config, nil
}
func loadConf1(bytes []byte) (*Type1, error) {
config := &Type1{}
if err := json.Unmarshal(bytes, config); err != nil {
return nil, fmt.Errorf("cannot load config: %v", err)
}
return config, nil
}
func loadConfOther1() (*Type1, error) {
// return value of this specific type
flatconfig := &Type1{}
// read a file as []byte
// written as a fixed array to simplify this example
fileContent := []byte{10, 22, 33, 44, 55}
if err := json.Unmarshal(fileContent, flatconfig); err != nil {
return nil, fmt.Errorf("cannot read config %v", err)
}
return flatconfig, nil
}
The only public function is LoadConfiguration.
It's based on a real code and It's used to read a json data as a specific struct. If something seems useless, it's because I simplified the original code.
The code above is ok, but now I want to create another struct type called "Type2" and re-use the same methods to read data into Type2 without copying and pasting everything.
type Type2 struct {
Name string `json:"name,omitempty"`
Path string `json:"path"`
Map *map[string]interface{} `json:"map"`
Other string `json:"other"`
}
Basically, I want to be able to call LoadConfiguration to get also Type2. I can accept to call a specific method like LoadConfiguration2, but I don't want to copy and paste also loadConf1 and loadConfOther1.
Is there a way to do that in an idiomatic way in Go 1.18?
Actually the code shown in your question doesn't do anything more than passing a type into json.Unmarshal and format an error so you can rewrite your function to behave just like it:
func LoadConfiguration(data []byte) (*Type1, error) {
config := &Type1{}
if err := loadConf(data, config); err != nil {
return nil, err
}
// ...
}
// "magically" accepts any type
// you could actually get rid of the intermediate function altogether
func loadConf(bytes []byte, config any) error {
if err := json.Unmarshal(bytes, config); err != nil {
return fmt.Errorf("cannot load config: %v", err)
}
return nil
}
In case the code actually does something more than just passing a pointer into json.Unmarshal, it can benefit from type parameters.
type Configurations interface {
Type1 | Type2
}
func loadConf[T Configurations](bytes []byte) (*T, error) {
config := new(T)
if err := json.Unmarshal(bytes, config); err != nil {
return nil, fmt.Errorf("cannot load config: %v", err)
}
return config, nil
}
func loadConfOther[T Configurations]() (*T, error) {
flatconfig := new(T)
// ... code
return flatconfig, nil
}
In these cases you can create a new pointer of either type with new(T) and then json.Unmarshal will take care of deserializing the content of the byte slice or file into it — provided the JSON can be actually unmarshalled into either struct.
The type-specific code in the top-level function should still be different, especially because you want to instantiate the generic functions with an explicit concrete type. So I advise to keep LoadConfiguration1 and LoadConfiguration2.
func LoadConfiguration1(data []byte) (*Type1, error) {
config, err := loadConf[Type1](data)
if err != nil {
return nil, err
}
confOther, err := loadConfOther[Type1]()
if err != nil {
return nil, err
}
// ... type specific code
return config, nil
}
However if the type-specific code is a small part of it, you can probably get away with a type-switch for the specific part, though it doesn't seem a viable option in your case. I would look like:
func LoadConfiguration[T Configuration](data []byte) (*T, error) {
config, err := loadConf[T](data)
if err != nil {
return nil, err
}
// let's pretend there's only one value of type parameter type
// type-specific code
switch t := config.(type) {
case *Type1:
// ... some *Type1 specific code
case *Type2:
// ... some *Type2 specific code
default:
// can't really happen because T is restricted to Configuration but helps catch errors if you extend the union and forget to add a corresponding case
panic("invalid type")
}
return config, nil
}
Minimal example playground: https://go.dev/play/p/-rhIgoxINTZ

How to set slice interface values with reflection

I would like to build a function that takes a generic pointer array and fill that list based on mongo results.
I don't know how to set the value I got from mongo into my pointer array. In the below attempt, program panics with following error : reflect.Set: value of type []interface {} is not assignable to type []Person
When I print total / documents found, it corresponds to what I am expecting. So I think question is about reflection.
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
var mongoResp struct {
Total int `bson:"total"`
Documents interface{} `bson:"documents"`
}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
receiverValue := reflect.ValueOf(receiver)
docs := []interface{}(mongoResp.Documents.(primitive.A))
receiverValue.Elem().Set(reflect.ValueOf(docs))
return mongoResp.Total, nil
}
type Person struct {
Name string `bson:"name"`
}
func main() {
var persons []Person
count, err := getListWithCount(context.Background(), &persons)
if err != nil {
log.Fatal(err)
}
fmt.Println(count)
fmt.Println(persons)
}
You should be able to decode first into bson.RawValue and then Unmarshal it into the receiver.
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
var mongoResp struct {
Total int `bson:"total"`
Documents bson.RawValue `bson:"documents"`
}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
if err := mongoResp.Documents.Unmarshal(receiver); err != nil {
return 0, err
}
return mongoResp.Total, nil
}
You can also implement it as a custom bson.Unmarshaler.
type MongoResp struct {
Total int `bson:"total"`
Documents interface{} `bson:"documents"`
}
func (r *MongoResp) UnmarshalBSON(data []byte) error {
var temp struct {
Total int `bson:"total"`
Documents bson.RawValue `bson:"documents"`
}
if err := bson.Unmarshal(data, &temp); err != nil {
return err
}
r.Total = temp.Total
return temp.Documents.Unmarshal(r.Documents)
}
With that you would use it in the function like so:
func getListWithCount(ctx context.Context, receiver interface{}) (int, error) {
//my mongo query here
mongoResp := MongoResp{Documents: receiver}
if err := cursor.Decode(&mongoResp); err != nil {
return 0, err
}
return mongoResp.Total, nil
}
Dynamically create a struct type that matches the queried document. See commentary below for details.
func getListWithCount(receiver interface{}) (int, error) {
dst := reflect.ValueOf(receiver).Elem()
// Your mongo query here
// Create a struct type that matches the document.
doct := reflect.StructOf([]reflect.StructField{
reflect.StructField{Name: "Total", Type: reflect.TypeOf(0), Tag: `bson:"total"`},
reflect.StructField{Name: "Documents", Type: dst.Type(), Tag: `bson:"documents"`},
})
// Decode to a value of the type.
docp := reflect.New(doct)
if err := cursor.Decode(docp.Interface()); err != nil {
return 0, err
}
docv := docp.Elem()
// Copy the Documents field to *receiver.
dst.Set(docv.Field(1))
// Return the total
return docv.Field(0).Interface().(int), nil
}
there is no need to use reflect here, you can decode it directly to your Person slices
func getPersons(ctx context.Context, coll *mongo.Collection, results interface{}) error {
cur, err := coll.Find(ctx, bson.D{})
if err != nil {
return err
}
err = cur.All(ctx, results)
if err != nil {
return err
}
return nil
}
and the len is the count of the results.
err = getPersons(ctx, coll, &persons)
require.NoError(t, err)
t.Logf("Got %d persons: %v", len(persons), persons)
see https://gist.github.com/xingyongtao/459f92490bdcbf7d5afe9f5d1ae6c04a

How can I initialize struct with values from array of interface in Go?

I'm getting a message from server like [0,"on",[6,1,5,"market",45.7]] and save it to []interface{} variable. I want to initialize struct with values of this array.
I'm totally new in Go and try to do it like:
import "golang.org/x/net/websocket"
...
var msg []interface{}
// Server send response: `[0,"on",[6,1,5,"market",45.7]]`
if err := websocket.Message.Receive(ws, &msg); err != nil {
logger.Println(err)
} else {
type Order struct {
ID int32,
GID int32,
CID int32,
Type string,
Amount float64
}
// here msg is [0,"on",[6,1,5,"market",45.7]]
switch msg[1] {
case "on":
if rawOrder, ok := msg[2].([]interface{}); ok {
order := Order{int32(rawOrder[0]), int32(rawOrder[1]), int32(rawOrder[2]), string(rawOrder[3]), float64(rawOrder[4])}
}
}
But I'm getting an error "Cannot convert an expression of the type 'interface{}' to the type 'int32'" and the next step is use switch for every rawOrder[i] type, but it's toooo long.
How can I do it easilly?
If you know that the codec used on the websocket will always be json, you can formally define Order and give it an UnmarshalJSON function to do the decoding for you.
import "golang.org/x/net/websocket"
type Order struct {
ID, GID, CID int32
Type string
Amount float64
}
func (o *Order) UnmarshalJSON(data []byte) error {
var first []json.RawMessage
err := json.Unmarshal(data, &first)
if err != nil {
return fmt.Errorf("invalid order, must be array: %w", err)
}
if len(first) != 3 {
return fmt.Errorf("invalid order, length must be 3, got %d", len(first))
}
var second string
err = json.Unmarshal(first[1], &second)
if err != nil {
return fmt.Errorf("invalid order, second element must be string: %w", err)
}
switch second {
case "on":
var third []json.RawMessage
err = json.Unmarshal(first[2], &third)
if err != nil {
return fmt.Errorf("invalid order, third element must be array: %w", err)
}
if len(third) != 5 {
return fmt.Errorf("invalid order, element 3 length must be 5, got %d", len(third))
}
for i, f := range []interface{}{&o.ID, &o.GID, &o.CID, &o.Type, &o.Amount} {
err = json.Unmarshal(third[i], f)
if err != nil {
return fmt.Errorf("invalid order, wrong type for element 3[%d]: %w", i, err)
}
}
return nil
}
return fmt.Errorf("invalid order, unknown type %q", second)
}
...
var msg *Order
// Server send response: `[0,"on",[6,1,5,"market",45.7]]`
if err := websocket.JSON.Receive(ws, &msg); err != nil {
logger.Println(err)
}
// msg is now an &Order{ID:6, GID:1, CID:5, Type:"market", Amount:45.7}
}
The reason the UnmarshalJSON function is huge is because your API is bad. If you control the Server; then you should avoid using mixed types in the same array, and you should avoid using arrays for relational data.

Map response to a struct using Golang

I am attempting to map a response from an API to a struct using Golang.
The JSON that comes back when I view the link in the browser is below:
{
"GBP": 657.54
}
And I just want to map it to a simple struct like so:
type Price struct {
Name string
Value float64
}
Here is my current code.
func FetchCoinPrice(fsym string, tsyms string) Price {
url := fmt.Sprintf("https://min-api.cryptocompare.com/data/price?fsym=" + fsym + "&tsyms=" + tsyms)
fmt.Println("Requesting data from " + url)
price := Price{}
// getting the data using http
request, err := http.Get(url)
if err != nil {
log.Fatal(err.Error())
}
// Read the response body using ioutil
body, err := ioutil.ReadAll(request.Body)
if err != nil {
log.Fatal(err.Error())
}
defer request.Body.Close()
if request.StatusCode == http.StatusOK {
json.Unmarshal(body, &price)
}
return price
}
At the moment all I receive is an empty struct, I know the link is bringing back the correct data and I've tested it in my browser.
The mapping doesn't work that way. Instead, you should use a map:
data := []byte(`{
"GBP": 657.54
}`)
priceMap := map[string]float64{}
err := json.Unmarshal(data, &priceMap)
// Check your errors!
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(priceMap)
This will print:
map[GBP:657.54]
You can then iterate over the map and build the struct you mentioned above, or just access the entry directly if you know the currency. eg: priceMap["GBP"]
You should really check your errors, especially if you're not getting the output you expect from Unmarshal.
The problem is that the unmarshaler cannot guess that keys in a JSON object should correspond to some value in a struct. Golang JSON mapping simply doesn't work that way.
However, you can make your "Price" type implement json.Unmarshaler to deserialize a message into a map of floats (map[string]float64) then ensure the shape is right and populate the struct accordingly:
func (p *Price) UnmarshalJSON(bs []byte) error {
kvs := map[string]float64{}
err := json.Unmarshal(bs, &kvs)
if err != nil {
return err
}
if len(kvs) != 1 {
return fmt.Errorf("expected 1 key, got %d", len(kvs))
}
for name, value := range kvs {
p.Name, p.Value = name, value
}
return nil
}
func main() {
jsonstr := `[{"GBP":657.54},{"USD":123.45}]`
ps := []Price{}
err := json.Unmarshal([]byte(jsonstr), &ps)
if err != nil {
panic(err)
}
// ps=[]main.Price{
// main.Price{Name:"GBP", Value:657.54},
// main.Price{Name:"USD", Value:123.45}
// }
}

How to judge unmarshal json interface{} type in golang?

I want to judge json type,but it always return "I don't know about type map[string]interface {}!",How to resolve it.
=========================================================================
type getRemoteCardInfo struct {
Code int
Msg string
Data []*remoteCardInfo
}
type remoteCardInfo struct {
Sn string
RemoteCardIp string
RemoteCardMac string
}
func Get_json_data(url string) (interface{}, error) {
client := &http.Client{}
req, err := http.NewRequest("GET", url, nil)
req.Header.Add("X-MYCMDB-Auth-Token", "sPf98SMBWzOZJEJB8KWltbJyKvFYPauu")
if err != nil {
return nil, err
}
resp, _ := client.Do(req)
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
return nil, fmt.Errorf("query failed: %s", resp.Status)
}
var result interface{}
body, err := ioutil.ReadAll(resp.Body)
if err := json.Unmarshal(body, &result); err != nil {
log.Fatalf("JSON unmarshaling failed: %s", err)
}
resp.Body.Close()
fmt.Println(result)
return result, nil
}
func main() {
jsondata, err := Get_json_data(DHCPURL)
if err != nil {
log.Fatal(err)
}
switch v := jsondata.(type) {
case getRemoteCardInfo:
fmt.Println("aaaa")
default:
fmt.Printf("I don't know about type %T!\n", v)
}
The go JSON unmarshaler doesn't know about types, as you can tell by the fact that it stores the result into an interface{} value:
func Unmarshal(data []byte, v interface{}) error
// "v" can be any type -------^
So it's up to you to use the unmarshaler to populate your structure and determine if the result is valid or not.
In your example it looks like you're trying to unmarshal a remoteCardInfo from an HTTP response. To do this you should unmarshal into an empty remoteCardInfo struct and determine if the required fields were populated.
For example, suppose you expect a JSON document like so:
{
"sn": "123",
"ip": "0.0.0.0",
"mac": "ff:ff:ff:ff:ff:ff"
}
Then you should define your "remoteCardInfo" struct as below:
type remoteCardInfo struct {
Sn string `json:"sn"`
RemoteCardIp string `json:"ip"`
RemoteCardMac string `json:"mac"`
}
And then unmarshal and validate it like so:
func getRemoteCardInfo(bs []byte) (*remoteCardInfo, error) {
rci := remoteCardInfo{}
err := json.Unmarshal(bs, &rci)
if err != nil {
return nil, err
}
// Validate the expected fields
if rci.Sn == "" {
return nil, fmt.Errorf(`missing "sn"`)
}
if rci.RemoteCardIp == "" {
return nil, fmt.Errorf(`missing "ip"`)
}
if rci.RemoteCardMac == "" {
return nil, fmt.Errorf(`missing "mac"`)
}
return &rci, nil
}
Of course, you can validate the fields any way you like but the main thing to remember is that the unmarshaler only does the job of ensuring that the input byte array is a valid JSON document and populates the fields from the document into the fields defined by the value.
It cannot tell you what "type" of object the JSON document represents.

Resources