Query Document with different struct for results - go

I have a collection of documents that were inserted into Mongo looking something like this:
type Stats struct {
UserStatus string `json:"userStatus" bson:"userStatus"`
... a bunch more fields
}
type User struct {
ID bson.ObjectId `json:"-" bson:"_id"`
LastName string `json:"lastName" bson:"lastName"`
FirstName string `json:"firstName" bson:"firstName"`
Role string `json:"role" bson:"role"`
Tags []string `json:"tags" bson:"tags"`
... (a bunch more fields)
Stats UserStats `json:"stats" bson:"stats"`
}
I want to query it to get a specific report, so I tried this:
func UserNameReport() {
... get mongo session, etc.
// create struct of just the data I want returned
type UserNames struct {
LastName string `json:"lastName" bson:"lastName"`
FirstName string `json:"firstName" bson:"firstName"`
... etc
UserStats Stats `json:"stats" bson:"stats"`
}
projection := bson.M{"lastName":1, "firstName":1, etc}
result := []UserNames{}
err := x.Find({query user collection}).Select(projection).All(&result)
...
}
This works - my question is, how can I include just ONE field from the 'Stats' struct? In other words,
I essentially want the "projection" to be this:
projection := bson.M{"lastName":1, ..., "stats.userStatus":1} <-- stats.userStatus doesn't work
...
err := x.Find({query user collection}).Select(projection).All(&result)
I get the entire "Stats" embedded struct in the results - how can I filter out just one field from the sub-document in and put it into the result set?
Thanks!

It works perfectly for me, with MongoDB 2.6.5
The following code:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"log"
)
type Statistics struct {
Url string
Hits int
}
type Person struct {
Num int
Uuid string
Name string
Stats []Statistics
}
func main() {
// Connect to the database
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
// Remove people collection if any
c := session.DB("test").C("people")
c.DropCollection()
// Add some data
err = c.Insert(
&Person{1, "UUID1", "Joe", []Statistics{Statistics{"a", 1}, Statistics{"b", 2}}},
&Person{2, "UUID2", "Jane", []Statistics{Statistics{"c", 3}, Statistics{"d", 4}}},
&Person{3, "UUID3", "Didier", []Statistics{Statistics{"e", 5}, Statistics{"f", 6}}})
if err != nil {
log.Fatal(err)
}
result := []Person{}
err = c.Find(bson.M{"$or": []bson.M{bson.M{"uuid": "UUID3"}, bson.M{"name": "Joe"}}}).Select(bson.M{"num": 1, "name": 1, "stats.hits": 1}).All(&result)
if err != nil {
log.Fatal(err)
}
fmt.Println(result)
}
results in:
[{1 Joe [{ 1} { 2}]} {3 Didier [{ 5} { 6}]}]
... which is precisely what I would expect.

Maybe this will help others - essentially I was trying to take a document with an embedded document and return a result set like I would do in SQL with a select a.LastName + ', ' + a.FirstName as Name, b.OtherData and in essence have a different 'table' / 'document'.
So here is my current solution - love to get better ones (more performant?) though!
I created a new struct and I'm using the 'mapstructure' library
import "github.com/goinggo/mapstructure"
type Stats struct {
UserStatus string `json:"userStatus" bson:"userStatus"`
... a bunch more fields
}
type User struct {
ID bson.ObjectId `json:"-" bson:"_id"`
LastName string `json:"lastName" bson:"lastName"`
FirstName string `json:"firstName" bson:"firstName"`
Role string `json:"role" bson:"role"`
Tags []string `json:"tags" bson:"tags"`
... (a bunch more fields)
Stats UserStats `json:"stats" bson:"stats"`
}
type MyReportItem struct {
FirstName string `json:"firstName" jpath:"firstName"`
LastName string `json:"lastName" jpath:"lastName"`
Status string `json:"status" jpath:"stats.userStatus"`
}
type MyReport struct {
Results []MyReportItem `json:"results"`
}
func xxxx(w http.ResponseWriter, r *http.Request) {
var users MyReportItem
// the results will come back as a slice of map[string]interface{}
mgoResult := []map[string]interface{}{}
// execute the query
err := c.Find(finder).Select(projection).All(&mgoResult)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
user := MyReportItem{}
// iterate through the results and decode them into the MyReport struct
for _, x := range mgoResult {
docScript, _ := json.Marshal(x)
docMap := map[string]interface{}{}
json.Unmarshal(docScript, &docMap)
err := mapstructure.DecodePath(docMap, &user)
if err == nil {
users.Results = append(users.Results, user)
}
}
... send back the results ...
_ := json.NewEncoder(w).Encode(&users)
}
Now I get a slice of objects in the form:
results: [
{
firstName: "John",
lastName: "Doe",
status: "active"
}
...
]
Instead of:
{
firstName: "John",
lastName: "Doe",
stats: {
status: "active"
}
}

Related

How to return nested entities after creating a new object?

Model Account contains nested structures - Currency and User
When I create a new instance of Account in DB, and then return it in my response, nested entities are empty:
type Account struct {
BaseModel
Name string `gorm:"size:64;not null" json:"name"`
Balance decimal.Decimal `gorm:"type:decimal(16, 2);default:0;not null;" json:"balance"`
UserID int `gorm:"not null" json:"-"`
User User `gorm:"foreignKey:UserID" json:"user"`
CurrencyID int `gorm:"not null" json:"-"`
Currency Currency `gorm:"foreignKey:CurrencyID" json:"currency"`
}
type CreateAccountBody struct {
Name string `json:"name" binding:"required"`
Balance decimal.Decimal `json:"balance"`
CurrencyID int `json:"currency_id" binding:"required"`
}
func CreateAccount(ctx *gin.Context) {
body := CreateAccountBody{}
if err := ctx.Bind(&body); err != nil {
log.Println("Error while binding body:", err)
ctx.JSON(
http.StatusBadRequest,
gin.H{"error": "Wrong request parameters"},
)
return
}
account := Account {
Name: body.Name,
Balance: body.Balance,
CurrencyID: body.CurrencyID,
UserID: 1,
}
if result := db.DB.Create(&account); result.Error != nil {
log.Println("Unable to create an account:", result.Error)
}
ctx.JSON(http.StatusCreated, gin.H{"data": account})
}
To avoid this problem, I refresh account variable with separate query:
db.DB.Create(&account)
db.DB.Preload("User").Preload("Currency").Find(&account, account.ID)
ctx.JSON(http.StatusCreated, gin.H{"data": account})
Is this the most effective and correct way to achieve the desired result?
I'm gonna share you how usually I managed this scenario. First, let me share the code.
main.go file
package main
import (
"context"
"gogindemo/handlers"
"github.com/gin-gonic/gin"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
var (
db *gorm.DB
ctx *gin.Context
)
func init() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable"
var err error
db, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic(err)
}
db.AutoMigrate(&handlers.Currency{})
db.AutoMigrate(&handlers.User{})
db.AutoMigrate(&handlers.Account{})
}
func AddDb() gin.HandlerFunc {
return func(ctx *gin.Context) {
ctx.Request = ctx.Request.WithContext(context.WithValue(ctx.Request.Context(), "DB", db))
ctx.Next()
}
}
func main() {
db.Create(&handlers.User{Id: 1, Name: "john doe"})
db.Create(&handlers.User{Id: 2, Name: "mary hut"})
db.Create(&handlers.Currency{Id: 1, Name: "EUR"})
db.Create(&handlers.Currency{Id: 2, Name: "USD"})
r := gin.Default()
r.POST("/account", AddDb(), handlers.CreateAccount)
r.Run()
}
Here, I've just added the code for bootstrapping the database objects and add some dummy data to it.
handlers/handlers.go file
package handlers
import (
"net/http"
"github.com/gin-gonic/gin"
"github.com/shopspring/decimal"
"gorm.io/gorm"
)
type User struct {
Id int
Name string
}
type Currency struct {
Id int
Name string
}
type Account struct {
Id int
Name string `gorm:"size:64;not null" json:"name"`
Balance decimal.Decimal `gorm:"type:decimal(16, 2);default:0;not null;" json:"balance"`
UserID int `gorm:"not null" json:"-"`
User User `gorm:"foreignKey:UserID" json:"user"`
CurrencyID int `gorm:"not null" json:"-"`
Currency Currency `gorm:"foreignKey:CurrencyID" json:"currency"`
}
type CreateAccountBody struct {
Name string `json:"name" binding:"required"`
Balance decimal.Decimal `json:"balance"`
CurrencyID int `json:"currency_id" binding:"required"`
}
func CreateAccount(c *gin.Context) {
db, ok := c.Request.Context().Value("DB").(*gorm.DB)
if !ok {
c.JSON(http.StatusInternalServerError, gin.H{"error": "internal server error"})
return
}
var accountReq CreateAccountBody
if err := c.BindJSON(&accountReq); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "wrong request body payload"})
return
}
// create Account & update the "account" variable
account := Account{Name: accountReq.Name, Balance: accountReq.Balance, CurrencyID: accountReq.CurrencyID, UserID: 1}
db.Create(&account).Preload("Currency").Preload("User").Find(&account, account.Id)
c.IndentedJSON(http.StatusCreated, account)
}
Within this file, I actually talk with the database through the DB passed in the context. Now, back to your question.
If the relationship between the Currency/Account and User/Account is of type 1:1, then, you should rely on the Preload clause. This will load the related entity in a separate query instead of adding it in an INNER JOIN clause.
Let me know if this solves your issue, thanks!

Get data from two different struct

Have this struct of User and Post and I try to make Name from User to be included within Post Struct when a user create a new post.
type User struct {
ID int
Name string
Created time.Time
}
type Post struct {
ID int
PostTitle string
PostDesc string
Created time.Time
}
How can I create something connected between this two struct such as Author of the Post ?
The goal is try to get the name of the post author which from User struct with the code below:
post, err := app.Models.Posts.GetPost(id)
GetPost() just run SELECT query and scan row
This approach is without any ORM.
It's a simple query that can return multiple rows. You've to scan the whole resultset and map each column on the struct's fields.
Keep in mind to always check for errors.
Below you can find the solution:
package main
import (
"database/sql"
"fmt"
"time"
_ "github.com/lib/pq"
)
type Post struct {
ID int
PostTitle string
PostDesc string
Created time.Time
UserID int
User User
}
type User struct {
ID int
Name string
Created time.Time
}
func main() {
conn, err := sql.Open("postgres", "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable")
if err != nil {
panic(err)
}
defer conn.Close()
// get MAX id
var id int
conn.QueryRow(`SELECT MAX(id) FROM posts`).Scan(&id)
// insert
sqlInsertStmt := `INSERT INTO posts (id, post_title, post_desc, created, user_id) VALUES ($1,$2,$3,$4,$5)`
if _, err = conn.Exec(sqlInsertStmt, id+1, "TDD", "Introduction to Test-Driven-Development", time.Now(), 1); err != nil {
panic(err)
}
// read
rows, err := conn.Query(`SELECT posts.id, post_title, post_desc, posts.created, users.id, users.name, users.created FROM posts INNER JOIN users ON posts.user_id=users.id`)
if err != nil {
panic(err)
}
var posts []Post
for rows.Next() {
var post Post
if err = rows.Scan(&post.ID, &post.PostTitle, &post.PostDesc, &post.Created, &post.User.ID, &post.User.Name, &post.User.Created); err != nil {
panic(err)
}
posts = append(posts, post)
}
if err = rows.Err(); err != nil {
panic(err)
}
for _, v := range posts {
fmt.Printf("author name: %q\n", v.User.Name)
}
}
Let me know if this helps.
Edit
I've also included an example of INSERT in Postgres. To achieve it, we've to use the db.Exec() function, and provide the parameters.
Pay attention to how you construct the query as you can get a SQL-Injection vulnerability.
Lastly, in a real-world scenario, you shouldn't lookup for the MAX id in the posts table but should be automatically generated.
Give it a try to this solution, maybe it resolves your issue.
First, I defined the structs in this way:
// "Post" belongs to "User", "UserID" is the foreign key
type Post struct {
gorm.Model
ID int
PostTitle string
PostDesc string
Created time.Time
UserID int
User User
}
type User struct {
ID int
Name string
Created time.Time
}
In this way, you can say that Post belongs to User and access the User's information within the Post struct.
To query the records, you've to use Preload("User") to be sure to eager load the User records from the separate table.
Keep attention to the name you pass in as the argument in Preload, as it can be tricky.
Lastly, you'll be able to access data in the embedded struct (with the dot notation).
Below, you can find a complete working example (implemented with the use of Docker):
package main
import (
"fmt"
"time"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
// "Post" belongs to "User", "UserID" is the foreign key
type Post struct {
gorm.Model
ID int
PostTitle string
PostDesc string
Created time.Time
UserID int
User User
}
type User struct {
ID int
Name string
Created time.Time
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic(err)
}
db.AutoMigrate(&Post{})
newPost := &Post{ID: 1, PostTitle: "Golang", PostDesc: "Introduction to Golang", Created: time.Now(), UserID: 1, User: User{ID: 1, Name: "John Doe", Created: time.Now()}}
db.Create(newPost)
var post Post
db.Preload("User").Find(&post, 1)
fmt.Printf("author name: %q\n", post.User.Name)
}
Let me know if I answered your question!

Parsing Nested JSON string

I am trying to parse a nested json string
I did get it to work by using multiple structs, but I am wondering if I can parse the JSON without using an extra struct.
type Events struct {
Events []Event `json:"events"`
}
type Event struct {
Name string `json:"name"`
Url string `json:"url"`
Dates struct {
Start struct {
LocalDate string
LocalTime string
}
}
}
type Embed struct {
TM Events `json:"_embedded"`
}
func TMGetEventsByCategory(location string, category string) {
parameters := "city=" + location + "&classificationName=" + category + "&apikey=" + api_key
tmUrl := tmBaseUrl + parameters
resp, err := http.Get(tmUrl)
var embed Embed
var tm Event
if err != nil {
log.Printf("The HTTP request failed with error %s\n", err)
} else {
data, _ := ioutil.ReadAll(resp.Body)
err := json.Unmarshal(data, &embed)
json.Unmarshal(data, &tm)
}
}
JSON Data looks like this:
{
"_embedded": {
"events": [],
},
"OtherStuff": {
}
}
Is it possible to get rid of the Embed struct and read straight to the events part of the json string?
What you're looking for here is json.RawMessage. It can help delay parsing of certain values, and in you case map[string]json.RawMessage should represent the top-level object where you can selectively parse values. Here's a simplified example you can adjust to your case:
package main
import (
"encoding/json"
"fmt"
)
type Event struct {
Name string `json:"name"`
Url string `json:"url"`
}
func main() {
bb := []byte(`
{
"event": {"name": "joe", "url": "event://101"},
"otherstuff": 15.2,
"anotherstuff": 100
}`)
var m map[string]json.RawMessage
if err := json.Unmarshal(bb, &m); err != nil {
panic(err)
}
if eventRaw, ok := m["event"]; ok {
var event Event
if err := json.Unmarshal(eventRaw, &event); err != nil {
panic(err)
}
fmt.Println("Parsed Event:", event)
} else {
fmt.Println("Can't find 'event' key in JSON")
}
}
In your case look for the _embedded key and then Unmarshal its value to Events
yes of course
type Embed struct {
TM []struct {
Name string `json:"name"`
Url string `json:"url"`
Dates struct {
Start struct {
LocalDate string
LocalTime string
}
} // add tag here if you want
} `json:"_embedded"`
}

Nested structs using gorm model

I have struct called User:
type User struct {
Email string
Name string
}
and struct called UserDALModel:
type UserDALModel struct {
gorm.Model
models.User
}
gorm Model looks like this:
type Model struct {
ID uint `gorm:"primary_key"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt *time.Time `sql:"index"`
}
this is possible to make UserDALModel nested with gorm model and user model so the output will be:
{
ID
CreatedAt
UpdatedAt
DeletedAt
Email
Name
}
now the output is:
{
Model: {
ID
CreatedAt
UpdatedAt
DeletedAt
}
User: {
Name
Email
}
}
According to this test in gorm, I think you need to add an embedded tag to the struct.
type UserDALModel struct {
gorm.Model `gorm:"embedded"`
models.User `gorm:"embedded"`
}
You can also specify a prefix if you want with embedded_prefix.
I found the answer:
type UserModel struct {
Email string
Name string
}
type UserDALModel struct {
gorm.Model
*UserModal
}
------------------------------
user := UserModel{"name", "email#email.com"}
userDALModel := UserDALModel{}
userDal.UserModal = &user
be careful embedding two structs with the same column:
package tests
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql"
"testing"
)
type A struct {
X string
Y string
}
type B struct {
X string
Y string
}
type AB struct {
B B `gorm:"embedded"` // Embedded struct B before struct A
A A `gorm:"embedded"`
}
var DB *gorm.DB
func connectDB() error {
var err error
spec := "slumberuser:password#tcp(localhost:3306)/slumber"
DB, err = gorm.Open("mysql", spec+"?parseTime=true&loc=UTC&charset=utf8")
DB.LogMode(true) // Print SQL statements
//defer DB.Close()
if err != nil {
return err
}
return nil
}
// cd tests
// go test -run TestGormEmbed
func TestGormEmbed(t *testing.T) {
if err := connectDB(); err != nil {
t.Errorf("error connecting to db %v", err)
}
values := []interface{}{&A{}, &B{}}
for _, value := range values {
DB.DropTable(value)
}
if err := DB.AutoMigrate(values...).Error; err != nil {
panic(fmt.Sprintf("No error should happen when create table, but got %+v", err))
}
DB.Save(&A{X: "AX1", Y: "AY1"})
DB.Save(&A{X: "AX2", Y: "AY2"})
DB.Save(&B{X: "BX1", Y: "BY1"})
DB.Save(&B{X: "BX2", Y: "BY2"})
//select * from `as` join `bs`;
// # x,y,x,y
// # AX1,AY1,BX1,BY1
// # AX2,AY2,BX1,BY1
// # AX1,AY1,BX2,BY2
// # AX2,AY2,BX2,BY2
var abs []AB
DB.Select("*").
Table("as").
Joins("join bs").
Find(&abs)
for _, ab := range abs {
fmt.Println(ab.A, ab.B)
}
// if it worked it should print
//{AX1 AY1} {BX1 BY1}
//{AX2 AY2} {BX1 BY1}
//{AX1 AY1} {BX2 BY2}
//{AX2 AY2} {BX2 BY2}
// but actually prints
//{BX1 BY1} {AX1 AY1}
//{BX1 BY1} {AX2 AY2}
//{BX2 BY2} {AX1 AY1}
//{BX2 BY2} {AX2 AY2}
}

Marshall/Unmarshal JSONPB

I am trying to Unmarshal some json data to a proto message.
JSON
{
"id": 1,
"first_name": "name",
"phone_numbers": []
}
Proto
message Item {
uint32 id=1;
string name=2;
repeated string numbers=3;
}
Proto.go
type Item struct {
Id uint32 `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
Numbers []string `protobuf:"bytes,4,rep,name=numbers" json:"numbers,omitempty"`
}
How can I map the above JSON to my proto Message (from what I can see there is no way to specify tags in proto atm)?
Your JSON document doesn't match the proto definition; name != first_name and numbers != phone_numbers.
You can define another type that has the same fields as Item but different struct tags and then convert to Item:
var x struct {
Id uint32 `json:"id,omitempty"`
Name string `json:"first_name,omitempty"`
Numbers []string `json:"phone_numbers,omitempty"`
}
if err := json.Unmarshal(jsonDoc, &x); err != nil {
log.Fatal(err)
}
var i = Item(x)
If every JSON document you want to decode has this structure, it may be more convenient to let Item implement json.Unmarshaler:
package main
import (
"encoding/json"
"fmt"
"log"
)
var jsonDoc = []byte(`
{
"id": 1,
"first_name": "name",
"phone_numbers": [
"555"
]
}
`)
type Item struct {
Id uint32 `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
Numbers []string `protobuf:"bytes,4,rep,name=numbers" json:"numbers,omitempty"`
}
// You can define this function is item_json.go or so, then it
// isn't removed if you re-generate your types.
func (i *Item) UnmarshalJSON(b []byte) error {
type item struct {
Id uint32 `json:"id,omitempty"`
Name string `json:"first_name,omitempty"`
Numbers []string `json:"phone_numbers,omitempty"`
}
var x item
if err := json.Unmarshal(jsonDoc, &x); err != nil {
return err
}
*i = Item(x)
return nil
}
func main() {
var i Item
if err := json.Unmarshal(jsonDoc, &i); err != nil {
log.Fatal(err)
}
fmt.Printf("%#v\n", i)
}
Try it on the playground: https://play.golang.org/p/0qibavRJbwi

Resources