How to write table data into a nested struct in Go - go

Summary
I am trying to write data from several postgres tables into a nested Go struct for the purpose of returning a single json response to a GET request within my web app.
Questions
Is the way I'm declaring a nested struct reasonable from a Go best practices perspective, or is there a reason I should avoid this method and do it another way?
What am I doing wrong in Step 3 to prevent my code from working? (I fear the answer is 'everything')
What I've got so far
I've declared my struct of structs
type MainObject struct {
SubObjects []struct {
SpecificDetail string `json:"specific-detail"`
} `json:"sub-object"`
...(other []structs)...
}
I've retrieved rows from the tables
func getMainObjectHandler(w http.ResponseWriter, r *http.Request) {
...(database connnection)...
MainObjectID := r.URL.Query().Get("moid")
if MainObjectID != "null" {
NewMainObject := MainObject{}
SubObjectDetail_rows, err := db.Query("SELECT specific_detail from the_table WHERE moid= '" + MainObjectID + "'")
if err != nil {
log.Fatalf("could not execute query: %v", err)
}
...(other db.Query rows)...
I've tried (and failed) to build the row data into the struct.
for SubObjectDetail_rows.Next() {
SpecificDetail := NewMainObject.SubObject.SpecificDetail{}
SubObjectDetail_rows.Scan(&SpecificDetail)
SubObject = append(SubObject, SpecificDetail)
}
NewMainObject = append(MainObject, SubObject)
defer persona_rows.Close()
Finally, I've set up Marshal and write.
NMOListBytes, err := json.Marshal(NewMainObject)
if err != nil {
fmt.Println(fmt.Errorf("Error: %v", err))
w.WriteHeader(http.StatusInternalServerError)
return
}
w.Write(NMOListBytes)

Firstly, please use placeholders when creating your SQL query to avoid injections:
// db.Query("SELECT specific_detail from the_table WHERE moid= '" + MainObjectID + "'") // not this
db.Query("SELECT specific_detail from the_table WHERE moid=?", MainObjectID)
Unless you're using a framework like GORM you can't Scan() into a single struct value.
From the docs:
Scan copies the columns in the current row into the values pointed at
by dest. The number of values in dest must be the same as the number
of columns in Rows.
It looks like you're pulling JSON from a DB query, as you're only querying one column, so you probably want:
var bs []byte // get raw JSON bytes
err = SubObjectDetail_rows.Scan(&bs)
if err != nil { /* always check errors */ }
and then unmarshal them to your struct:
err = json.Unmarshal(bs, &SpecificDetail)
if err != nil { /* ... */ }

Related

How to design RestAPI for too many tables in Golang

I think if i keep using the method below, i'll have to write too much code.
I declared structures for all the tables. and i used the go validate package for validation.
[types.go]
type TableA struct {
Field1 string `json:"field1" validate:"required, max=10"`
Field2 int `json:"field2" validate:"number"`
}
type TableB struct {
...
}
And i initialized the router for each method and connected the handlers.
[tableA.go]
router.Get("/table-a", r.Get_tableA_Handler),
router.Post("/table-a", r.Post_tableA_Handler),
router.Patch("/table-a", r.Patch_tableA_Handler),
router.Delete("/table-a", r.Delete_tableA_Handler)
...
Each handler parses the json in the request body, validates the data and call the db function.
[tableA_router.go]
func (rt *tableARouter) Post_tableA_Handler(w http.ResponseWriter, r *http.Request) error {
//Json to Struct
req := new(types.tableA)
if err := httputils.DecodeJsonBody(r, req); err != nil {
return err
}
// Validation
if err := validCheck(req); err != nil {
return err
}
// DB function
err := rt.insert_tableA_DB(r.Context(), req)
if err != nil {
return err
}
return rt.rd.JSON(w, http.StatusCreated, "Create Success")
}
...
func validCheck(data interface{}) error {
validate := validator.New()
err := validate.Struct(data)
return err
}
This is a DB function called from the handler function above (using Gorm)
[tableA_db.go]
func (rt *tableARouter) insert_tableA_DB(ctx context.Context, data *types.TableA) error {
// DB Connect
db, err := db.Open(rt.dbcfg)
if err != nil {
return err
}
defer db.Close()
tx := db.Begin()
defer tx.Rollback()
// == INSERT ==
query := `INSERT INTO table_a
(field1, field2, ...)
VALUES (?, ?, ...)`
result := tx.WithContext(ctx).Exec(query,
data.Field1, data.Field2, ...)
//Result
if result.Error != nil {
...
}
There are too many tables now... If there are 100 tables i have to write 100 handlers and 100 DB functions.
Is there any way to use something like /tables/{tableName}?
Please give me any advice.... Thank you.
You can use an ORM package, like GORM to make easier your work.
Or you can make an universal handler and with the reflect package, analyze your defined structs and make every SQL query dinamically. But it's not the best solution if any of your struct has inner slices, other embedded structs, or if you need to use joined tables you also have to deal with it manually. I have servers where we have more than 200 endpoints with more than 3-400 methods with 200+ SQL tables and the whole server was written by hand. But I can say, it's very rare when a handler and the DB func can be reused without modifying.
Maybe you can wrap the error handling, rollback/commit, json parse and response parts in a func then use it to call the DB methods.

How can I do conditional actions in SQLX

I have database store function:
func (p *ProductsRep) FindAll(PageNumber int, PaginationSize int, Query string) []*postgresmodels.Product {
Also I have SQL query look like this:
SELECT * FROM table_name.
Then I want to concat conditional action like WHERE some_value=3 if some value (in this case Query) exists then I want to get SELECT * FROM table_name WHERE some_value=3.
I tried to use fmt.Sprintf to concat, or strings.Join, or bytes.Buffer.WriteString. But everytime I getting this error:
I replace real value for understanding:
pq: column "Some value" does not exist.
How can I do "adaptive" queries, which depends on inputed function values.
I believe you are trying to query rows in the database by using parameters.
You need to make sure you don't pass this data in as RAW values, due to the potential risk of SQL injection. You can make queries by using store procedures
You can use the function Query to pass in your query with your parameters. In the example case this is $1. If you wanted to you could add $2, $3... etc depending on how many parameters you wanted to query
Here is two examples
Postgres
using "github.com/jackc/pgx/v4" driver
ctx := context.Background()
type Bar struct {
ID int64
SomeValue string
}
rows, err := conn.Query(ctx, `SELECT * FROM main WHERE some_value=$1`, "foo")
if err != nil {
fmt.Println("ERRO")
panic(err) // handle error
}
defer rows.Close()
var items []Bar
for rows.Next() {
var someValue string
var id int64
if err := rows.Scan(&id, &someValue); err != nil {
log.Fatal(err) // handle error
}
item := Bar{
ID: id,
SomeValue: someValue,
}
items = append(items, item)
}
fmt.Println(items)
MySQL Driver
https://golang.org/pkg/database/sql/#DB.QueryRow
type Bar struct {
ID int64
SomeValue string
}
rows, err := conn.Query(`SELECT * FROM main WHERE some_value=$1`, "foo")
if err != nil {
fmt.Println("ERRO")
panic(err) // handle error
}
defer rows.Close()
var items []Bar
for rows.Next() {
var someValue string
var id int64
if err := rows.Scan(&id, &someValue); err != nil {
log.Fatal(err) // handle error
}
item := Bar{
ID: id,
SomeValue: someValue,
}
items = append(items, item)
}
fmt.Println(items)

Update method does not update zero value

Original Question
When using the Update method in GORM the new data does not get saved. i.e. I want to set a bool from true to false, but it stays true even after the Update method.
In the description of the method there is a warning: "WARNING when update with struct, GORM will not update fields that with zero value"
Since I am using a struct to update and false is the zero value of bool, this seems expected behaviour, but I don't see any reason why to do so and how to overcome this.
func UpdateData(c *fiber.Ctx) error {
db := database.DBConn
data := new([]entities.Data)
if err := c.BodyParser(&data); err != nil {
return err
}
db.Update(&data)
return c.JSON(data)
}
Solution Summary
First, as suggested I left out the new keyword when instantiating the structs. Then, I used a helper function (from here) for converting a struct to map while keeping the json alias as keys:
// StructToMap Converts a struct to a map while maintaining the json alias as keys
func StructToMap(obj interface{}) (newMap map[string]interface{}, err error) {
data, err := json.Marshal(obj)
if err != nil {
return
}
err = json.Unmarshal(data, &newMap) // Convert to a map
return
}
Then I loop over each element in the data slice in order to convert it and update it one by one:
func UpdateData(c *fiber.Ctx) error {
db := database.DBConn
data := []entities.Dard{}
if err := c.BodyParser(&data); err != nil {
return err
}
for _, record := range data {
mappedData, _ := StructToMap(record)
db.Model(&entities.Data{}).Update(mappedData)
}
return c.JSON(data)
}
*Error handling is obviously reduced in this example.
From official doc
NOTE When update with struct, GORM will only update non-zero
fields, you might want to use map to update attributes or use Select
to specify fields to update
So use map[string]interface{} to update non-zero fields also. Example:
db.Model(&user).Updates(map[string]interface{}{"name": "hello", "age": 18, "actived": false})
As you have struct already you can convert struct into map[string]interface{} (See details about conversion) then update. Another way is change the type of field as pointer.

Gorm query returning only a single row

We're trying to use Gorm with mysql 8 to much frustration.
I have the following tables (simplified for brevity here)
type StoragePool struct {
gorm.Model
PoolId string `json:"id" gorm:"column:poolid;size:40;unique;not null"`
Volumes []Volume `json:"volumes" gorm:"foreignkey:StorageId;association_foreignkey:PoolId"`
}
type Volume struct {
gorm.Model
StorageId string `json:"storageid" gorm:"column:storageid;size:40"`
}
Data insertions seem to work fine. Both tables get populated and no constraints are violated.
A query that expects a single record seems to work fine:
poolRecord := &StoragePool{}
if err := tx.Where("poolid = ?", pool.PoolId).First(&StoragePool{}).Scan(poolRecord).Error; err != nil {
return err
}
This query only returns a single row. When I perform this exact query as raw SQL outside of go, it returns all 31 records I expect.
var poolVolumes []Volume
if err := tx.Where("storageid = ?", pool.PoolId).Find(&Volume{}).Scan(&poolVolumes).Error; err != nil {
return err
}
log.Debugf("found %d volumes belonging to %q [%s]", len(poolVolumes), pool.Name, pool.PoolId)
According to the docs, that second sql statement is the equivalent of "SELECT * FROM VOLUMES WHERE STORAGEID = 'poolid'". That is not the behavior I am getting.
Anyone have any ideas what I'm doing wrong here?
I rarely use an ORM while coding with go, but following the doc from gorm, it seems like you are doing it the wrong way.
Scan is used for scanning result into another struct, like this:
type Result struct {
Name string
Age int
}
var result Result
db.Table("users").Select("name, age").Where("name = ?", 3).Scan(&result)
The correct way to get query results into a slice of structs should be:
var poolVolumes []Volume
if err := tx.Where("storageid = ?", pool.PoolId).Find(&poolVolumes).Error; err != nil {
return err
}

Golang Rest api data not showing in postman

I am facing an issue where i have made an api in Go every thing work fine but i am not getting data in postman. When i print the data in logs i am getting the data properly but it is showing blank data in postman.
authorizeModel.go
func GetSkillList() map[string]interface{} {
db := GetDB()
var (
// id int
skillName string
)
type SkillList struct {
name string
}
skillList := SkillList{}
skillArr := []SkillList{}
rows, err := db.Query("select DISTINCT(name) as name from skills where company_id IN ('2') and name != 'Skill Needed' order by name")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&skillName)
if err != nil {
log.Fatal(err)
}
skillList.name = skillName
skillArr = append(skillArr, skillList)
}
response := u.Message(true, "Skill list retrieved successfully")
response["data"] = skillArr
log.Println(skillArr)
response["authorization"] = false
return response
}
authController.go
var SkillTagList = func(w http.ResponseWriter, r *http.Request) {
resp := models.GetSkillList()
u.Respond(w, resp)
}
routes.go
router.HandleFunc("/api/v1/authorize/skillTagList", controllers.SkillTagList).Methods("POST")
If you see authorizeModel.go i have printed my data in logs i am getting that data successfully in logs. But see the postman screenshot below.
You have to rename name to Name
I'm not sure what is u.Respond(), so I will assume it's a helper function of some framework that you are using, and I will assume u.Respond() is internally using json.Marshal.
If your struct has unexported fields(fields name starting with lowercase letter, in your case name), json.Marshal cannot access those field, and the result won't have name field. That is why you are getting empty objects in JSON.

Resources