convert go-pg query into plain sql - go

Is it possible to convert go-pg query
err = db.Model(story).
Relation("Author").
Where("story.id = ?", story1.Id).
Select()
into plain SQL?
It would be helpful for debugging. So I could copy this plain SQL query and run in psql client as a string.
Probably there is some kind of package for this?

This is listed in the project's wiki:
How to view queries this library generates?
How to view queries this library generates?
You can setup query logger like this:
type dbLogger struct { }
func (d dbLogger) BeforeQuery(c context.Context, q *pg.QueryEvent) (context.Context, error) {
return c, nil
}
func (d dbLogger) AfterQuery(c context.Context, q *pg.QueryEvent) (context.Context, error) {
fmt.Println(q.FormattedQuery())
return c, nil
}
db := pg.Connect(&pg.Options{...})
db.AddQueryHook(dbLogger{})

Problem: There is no default parsing to RAW sql from ORM since v10.
Well, that's too late i guess. Maybe someone (like me) will face this problem in 2021. There is some steps how you could solve this:
[x] Read DOCs.
[x] Check all structs.
[x] Implement all methods.
Solving the problem
This solve is "forked" from this issue but i'll explain it step by step.
First of all we need to read some source code of go-pg hook.
As I said before: we need to check all structs from this doc. But we're lucky. There is only 1 struct!
// QueryEvent ...
type QueryEvent struct {
StartTime time.Time
DB orm.DB
Model interface{}
Query interface{}
Params []interface{}
fmtedQuery []byte
Result Result
Err error
Stash map[interface{}]interface{}
}
We don't really need to implement this struct completely.
But when you use db.AddQueryHook() (where db is ref on our DB connection and AddQueryHook() is method) AddQueryHook() wait's from you this interface:
type QueryHook interface {
BeforeQuery(context.Context, *QueryEvent) (context.Context, error)
AfterQuery(context.Context, *QueryEvent) error
}
So, we already read DOCs, checked structs. And what's next? Answer is pretty easy:
Implement all methods.
TBH, I thought that this is harder than it is.
To implement it you just need to create 2 methods of current (new empty) structure that implements functionality of methods above, like this:
Creating empty structure
type dbLogger struct{}
Adding methods from doc:
func (d dbLogger) BeforeQuery(c context.Context, q *pg.QueryEvent) (context.Context, error) {
return c, nil
}
func (d dbLogger) AfterQuery(c context.Context, q *pg.QueryEvent) error {
fq, _ := q.FormattedQuery()
fmt.Println(string(fq))
return nil
}
I hope this helps everyone who ever encounters this problem.

I've just been upgrading from go-pg v7 to v10 & had a problem where Query.AppendFormat() which is what I was using to get the RAW SQL had been removed.
After using the comments in this post for inspiration I managed extract it, using the code below
import (
"github.com/go-pg/pg/v10/orm"
)
func QueryToString(q *orm.Query) string {
value, _ := q.AppendQuery(orm.NewFormatter(), nil)
return string(value)
}
Hope this helps future viewers

//db your *pg.DB
// q your *orm.Query = db.Model(&yourModel).
qq := pg.QueryEvent{
DB: db,
Model: q.TableModel(),
Query: q,
}
fmt.Println(qq.FormattedQuery())
So in your case
q:= db.Model(story).
Relation("Author").
Where("story.id = ?", story1.Id)
fmt.Println("running SQL:")
qq := pg.QueryEvent{
DB: db,
Model: q.TableModel(),
Query: q,
}
fmt.Println(qq.FormattedQuery())
q.Select()

Related

How can I generate SQL code from GORM struct model?

I'm using goose to manage my database migrations but I need to write SQL sentences directly in the migrations file. There is a way to generate the SQL directly from the GORM model?
Unfortunately using the gorm.Session{DryRun: true} option doesn't make the migration SQL statement/s available to the caller as it does with normal queries.
The only way I can see right now would be to capture the SQL that is run by the migration when it's being logged by reimplementing the gorm.io/gorm/logger.Interface interface. Specifically, the Trace method.
type Interface interface {
LogMode(LogLevel) Interface
Info(context.Context, string, ...interface{})
Warn(context.Context, string, ...interface{})
Error(context.Context, string, ...interface{})
Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error)
}
Inside Trace you can call that fc function argument to get the SQL and RowsAffected, which you can do whatever you want with.
For example:
import (
"time"
"context"
"gorm.io/gorm/logger"
)
type RecorderLogger struct {
logger.Interface
Statements []string
}
func (r *RecorderLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
sql, _ := fc()
r.Statements = append(r.Statements, sql)
}
Now use it as:
recorder := RecorderLogger{logger.Default.LogMode(logger.Info)}
session := db.Session(&gorm.Session{
Logger: &recorder
})
session.AutoMigrate(&Model{}, ...)
// or
session.Migrator().CreateTable(&Model{}, ...) // or any method therein
// now recorder.Statements contains the statements run during migration
This is very hacky, and you may run into problems because AutoMigrate modifies the current state of the database and migrates it up to what your model requires (up to a point) and for that to work your current database must reflect the current state of your production database (or whatever database your hope to migrate). So, you could build that tool that helps you get the migration script started if you're careful, but to properly gain the advantages of a migration system like goose you'll need to get your hands dirty with the SQL :)
you can using this lib: https://github.com/sunary/sqlize
It's allowed you create sql from models, also support migration by differ between models and existing sql.
I personally would use the migration functionality that is available inside Gorm, but for your case we can do the following.
Firstly there is a feature in Gorm called Dry Run and you can use this to see the SQL statements that get executed when performing queries. Unfortunately I can't see that it is possible when using migrations. So what I suggest is to use github.com/DATA-DOG/go-sqlmock
I would usually use this for testing purposes, but you could use it temporarily to get the SQL needed for your separate migrations.
package main
import (
"database/sql"
"time"
"github.com/DATA-DOG/go-sqlmock"
"gorm.io/driver/mysql"
"gorm.io/gorm"
)
type Model struct {
ID uint64 `gorm:"primaryKey"`
Name string `gorm:"index"`
Description string
CreatedAt time.Time
LastLogin sql.NullTime
}
func main() {
sqlDB, _, err := sqlmock.New()
if err != nil {
panic(err)
}
gormDB, err := gorm.Open(mysql.New(mysql.Config{
Conn: sqlDB,
SkipInitializeWithVersion: true,
}), &gorm.Config{})
if err != nil {
panic(err)
}
defer sqlDB.Close()
gormDB.AutoMigrate(&Model{})
}
This will give you a result like this
all expectations were already fulfilled, call to ExecQuery 'CREATE TABLE `models` (`id` bigint unsigned AUTO_INCREMENT,`name` varchar(191),`description` longtext,`created_at` datetime(3) NULL,`last_login` datetime(3) NULL,PRIMARY KEY (`id`),INDEX idx_models_name (`name`))' with args [] was not expected
[0.003ms] [rows:0] CREATE TABLE `models` (`id` bigint unsigned AUTO_INCREMENT,`name` varchar(191),`description` longtext,`created_at` datetime(3) NULL,`last_login` datetime(3) NULL,PRIMARY KEY (`id`),INDEX idx_models_name (`name`))
which contains the SQL statement required. This feels incredibly hacky but will give you the result you need

How to pass variable ids to statement.Query() in golang?

I have this query in postgres which queries 1 or n users based on the parameters passed:
select name, phone from clients where id in ('id1','id2')
Now when I try to use this at golang I'm having problems approaching how to pass this type of variable arguments to the statement.Query() function:
ids := []string{"0aa6c0c5-e44e-4187-b128-6ae4b2258df0", "606b0182-269f-469a-bb29-26da4fa0302b"}
rows, err := stmt.Query(ids...)
This throws error: Cannot use 'ids' (type []string) as type []interface{}
When I check in source code query it can receive many variables of type interface:
func (s *Stmt) Query(args ...interface{}) (*Rows, error) {
return s.QueryContext(context.Background(), args...)
}
If I do this manually it works:
rows, err := stmt.Query("0aa6c0c5-e44e-4187-b128-6ae4b2258df0", "606b0182-269f-469a-bb29-26da4fa0302b")
But of course I need the args to be 1 or many more, and dynamically generated.
I'm using Sqlx lib.
As we can see on the Query() method scheme and also from the error message, the method requires an argument in []interface{} type.
func (s *Stmt) Query(args ...interface{}) (*Rows, error) {
return s.QueryContext(context.Background(), args...)
}
In your code, the ids variable hold []string data. Change it to []interface{} so it'll meet Query() requirements, then it'll work.
ids := []interface{}{
"0aa6c0c5-e44e-4187-b128-6ae4b2258df0",
"606b0182-269f-469a-bb29-26da4fa0302b",
}
rows, err := stmt.Query(ids...)

How to modify GORM DB pointer in function?

I have a REST API application written in Go, which uses GORM as ORM. During refactoring of some parts, I wanted to move some common operations to an external function ApplyToDBQuery(query *gorm.DB), which takes a pointer to the DB query and modifies it, so that the modified query can be used later.
Example usage:
query = shared.DB.Debug()
req.ApplytoDBQuery(query)
if query.find(&data).Error != nil {...}
func (this *MyCustomRequest) ApplyToDBQuery(query *gorm.DB) {
query.Limit(...)
query.Offset(...)
query.Where(...)
}
I thought, that since I'm passing a pointer to the function, the original query should've been modified, but nothing really happened to the original query.
I've also tried passing a pointer to pointer ApplyToDBQuery(query **gorm.DB), returning the modified pointer ApplyToDBQuery(query *gorm.DB) *gorm.DB and out of lack of ideas, even a combination of these two - ApplyToDBQuery(query **gorm.DB) *gorm.DB
Gorm object clone itself for every operation, therefore the original pointed value is never changed.
You should return the latest version of gorm.DB:
return query.Limit(...).Offset(...).Where(...)
Change the method receiver to
func (this *MyCustomRequest) ApplyToDBQuery(query *gorm.DB) *gorm.DB {
return query.Limit(...).
Offset(...).
Where(...)
}
then use it as:
query = req.ApplytoDBQuery(query)
if query.find(&data).Error != nil {...}
The reason is already pointed by #R3v4n
You should go with what the previous two answers recommend, but, if for some reason, you have to apply changes to the passed in pointer, you can still do it "manually".
query = shared.DB.Debug()
req.ApplytoDBQuery(query)
if query.find(&data).Error != nil {...}
func (r *MyCustomRequest) ApplyToDBQuery(query *gorm.DB) {
q := query.Limit(...).Offset(...).Where(...)
*query = *q
}
As a side note, it is generally discouraged to use receiver names like this and self, instead the preferred way is to use a short, let's say 1 to 3 letters, abbreviation of the type's name.
For example:
func (r *Request) AddCookie(c *Cookie)
func (c *Client) Get(url string) (resp *Response, err error)
func (srv *Server) ListenAndServe() error
Complementing #R3v4n answer:
Use chaining
Use db.Scopes for code reusage instead of usual func calls.
It can be like this:
query = shared.DB.Debug()
if query.Scopes(req.ApplyToDBQuery).find(&data).Error != nil {
// handle error
}
func (this *MyCustomRequest) ApplyToDBQuery(query *gorm.DB) *gorm.DB {
return query.Where(...).Limit(...).Offset(...)
}

Cannot Range Over List Type Interface {} In Function Using Go

Cannot Range Over List Type Interface {} In Function Using Go.
for me is important then i execute for in a function.
How can fix?
package main
import (
"fmt"
)
type MyBoxItem struct {
Name string
}
type MyBox struct {
Items []MyBoxItem
}
func (box *MyBox) AddItem(item MyBoxItem) []MyBoxItem {
box.Items = append(box.Items, item)
return box.Items
}
func PrintCustomArray(list interface{}) interface{} {
//items := reflect.ValueOf(list)
for _, v := range list {
fmt.Println(v.Key,v.Value)
}
return 0
}
func main() {
items := []MyBoxItem{}
item := MyBoxItem{Name: "Test Item 1"}
box := MyBox{items}
box.AddItem(item)
fmt.Println((box.Items))
PrintCustomArray(box.Items)
}
https://play.golang.org/p/ZcIBLMliq3
Error : cannot range over list (type interface {})
How can fix?
Note
The answer below describes, in broad strokes, 2 possible approaches: using interfaces, and using specific types. The approach focusing on interfaces is mentioned for completeness sake. IMHO, the case you've presented is not a viable use-case for interfaces.
Below, you'll find a link to a playground example that uses both techniques. It should be apparent to anyone that the interface approach is too cumbersome if for this specific case.
Quite apart from the fact that you don't really seem to be too familiar with how loops work in go (v.Key and v.Value are non-existent fields for example), I'll attempt to answer your question.
You are passing a list to your function, sure enough, but it's being handled as an interface{} type. That means your function accepts, essentially, any value as an argument. You can't simply iterate over them.
What you can do is use type assertions to convert the argument to a slice, then another assertion to use it as another, specific interface:
type Item interface{
key() string
val() string
}
func (i MyBoxItem) key() string {
return i.Key
}
func (i MyBoxItem) val() string {
return i.Value
}
func PrintCustomArray(list interface{}) error {
listSlice, ok := list.([]interface{})
if !ok {
return fmt.Errorf("Argument is not a slice")
}
for _, v := range listSlice {
item, ok := v.(Item)
if !ok {
return fmt.Errorf("element in slice does not implement the Item interface")
}
fmt.Println(item.key(), item.val())
}
return nil
}
But let's be honest, a function like this only works if a slice is passed as an argument. So having that first type assertion in there makes no sense whatsoever. At the very least, changing the function to something like this makes a lot more sense:
func PrintCustomArray(list []interface{})
Then, because we're not expecting an array as such, but rather a slice, the name should be changed to PrintCustomSlice.
Lastly, because we're using the same type assertion for every value in the slice, we might as well change the function even more:
// at this point, we'll always return 0, which is pointless
// just don't return anything
func PrintCustomSlice(list []Item) {
for _, v := range list {
fmt.Println(v.key(), v.val())
}
}
The advantages of a function like this is that it can still handle multiple types (all you have to do is implement the interface). You don't need any kind of expensive operations (like reflection), or type assertions.
Type assertions are very useful, but in a case like this, they merely serve to hide problems that would otherwise have resulted in a compile-time error. Go's interface{} type is a very useful thing, but you seem to be using it to get around the type system. If that's what you want to achieve, why use a typed language in the first place?
Some closing thoughts/remarks: If your function is only going to be used to iterate over specific "thing", you don't need the interfaces at all, simply specify the type you're expecting to be passed to the function in the first place. In this case that would be:
func PrintCustomSlice(list []MyBoxItem) {
for _, v := range list {
fmt.Println(v.Key, v.Value)
}
}
Another thing that I've noticed is that you seem to be exporting everything (all functions, types, and fields start with a capital letter). This, in go, is considered bad form. Only export what needs to be public. In the main package, that usually means you're hardly export anything.
Lastly, as I mentioned at the start: you don't seem to have a firm grasp on the basics just yet. I'd strongly recommend you go through the interactive tour. It covers the basics nicely, but shows you the features of the language at a decent pace. It doesn't take long, and is well worth taking a couple of hours to complete
Playground demo
It's possible to implement PrintCustomArray using the reflect package, but most experienced Go programmers will write a simple for loop:
for _, i := range box.Items {
fmt.Println("Name:", i.Name)
}
https://play.golang.org/p/RhubiCpry0
You can also encapsulate it in a function:
func PrintCustomArray(items []MyBoxItem) {
for _, i := range items {
fmt.Println("Name:", i.Name)
}
}
https://play.golang.org/p/c4EPQIx1AH
Here since you are returning box.Items from AddItem(), Items is of the type []MyBoxItem , so list should be of type []MyBoxItem .Moreover you are returning 0 in PrintCustomArray and the return type you have set is {}interface.
func PrintCustomArray(list []MyBoxItem) {
//items := reflect.ValueOf(list)
for i, v := range list {
fmt.Println(i, v)
}
//return 0
}
Again, MyBoxItem struct has only one variable named Name so v.key v.value won't make any sense.
This is what the proper code should look like https://play.golang.org/p/ILoUwEWv6Y .
You need to clear your understanding about interfaces in go. This might help https://golang.org/doc/effective_go.html#interfaces_and_types .

Using empty interfaces in go

I am trying to understand the code that is used at my company. I am new to go lang, and I have already gone through the tutorial on their official website. However, I am having a hard time wrapping my head around empty interfaces, i.e. interface{}. From various sources online, I figured out that the empty interface can hold any type. But, I am having a hard time figuring out the codebase, especially some of the functions. I will not be posting the entire thing here, but just the minimal functions in which it has been used. Please bear with me!
Function (I am trying to understand):
func (this *RequestHandler) CreateAppHandler(rw http.ResponseWriter, r *http.Request) *foo.ResponseError {
var data *views.Data = &views.Data{Attributes: &domain.Application{}}
var request *views.Request = &views.Request{Data: data}
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
logrus.Error(err)
return foo.NewResponsePropogateError(foo.STATUS_400, err)
}
requestApp := request.Data.Attributes.(*domain.Application)
requestApp.CreatedBy = user
Setting some context, RequestHandler is a struct defined in the same package as this code. domain and views are seperate packages. Application is a struct in the package domain. The following two structs are part of the package views:
type Data struct {
Id string `json:"id"`
Type string `json:"type"`
Attributes interface{} `json:"attributes"`
}
type Request struct {
Data *Data `json:"data"`
}
The following are part of the package json:
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
if err := dec.tokenPrepareForDecode(); err != nil {
return err
}
if !dec.tokenValueAllowed() {
return &SyntaxError{msg: "not at beginning of value"}
}
// Read whole value into buffer.
n, err := dec.readValue()
if err != nil {
return err
}
dec.d.init(dec.buf[dec.scanp : dec.scanp+n])
dec.scanp += n
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
err = dec.d.unmarshal(v)
// fixup token streaming state
dec.tokenValueEnd()
return err
}
type Decoder struct {
r io.Reader
buf []byte
d decodeState
scanp int // start of unread data in buf
scan scanner
err error
tokenState int
tokenStack []int
}
Now, I understood that, in the struct Data in package views, Application is being set as a type for the empty interface. After that, a pointer to Request in the same package is created which points to the variable data.
I have the following doubts:
What exactly does this keyword mean in Go? What is the purpose of writing this * RequestHandler?
Initialization of a structure in Go can be done while assigning it to a variable by specifying the values of all it's members. However, here, for the struct Data, only the empty interface value is assigned and the values for the other two fields are not assigned?
What is the advantage of assigning the Application struct to an empty interface? Does it mean I can use the struct members using the interface variable directly?
Can someone help me figure out the meaning of this statement? json.NewDecoder(r.Body).Decode(request)?
While I know this is too much, but I am having a hard time figuring out the meaning of interfaces in Go. Please help!
this is not a keyword in go; any variable name can be used there. That is called the receiver. A function declared in that way must be called like thing.func(params), where "thing" is an expression of the type of the receiver. Within the function, the receiver is set to the value of thing.
A struct literal does not have to contain values for all the fields (or any of them). Any fields not explicitly set will have the zero value for their types.
As you said, an empty interface can take on a value of any type. To use a value of type interface{}, you would use type assertion or a type switch to determine the type of the value, or you could use reflection to use the value without having to have code for the specific type.
What specifically about that statement do you not understand? json is the name of a package in which the function NewDecoder is declared. That function is called, and then the Decode function (which is implemented by the type of the return value of NewDecoder) is called on that return value.
You may want to take a look at Effective Go and/or The Go Programming Language Specification for more information.

Resources