How to use rows.Scan of Go's database/sql - go

I use database/sql and define a struct mapping to DB table columns(tag field):
// Users ...
type Users struct {
ID int64 `field:"id"`
Username string `field:"username"`
Password string `field:"password"`
Tel string `field:"tel"`
}
then I query:
rows, err := db.Query(sql) // select * from users
if err != nil {
fmt.Println(err)
}
defer rows.Close()
for rows.Next() {
user := new(Users)
// works but I don't think it is good code for too many columns
err = rows.Scan(&user.ID, &user.Username, &user.Password, &user.Tel)
// TODO: How to scan in a simple way
if err != nil {
fmt.Println(err)
}
fmt.Println("user: ", user)
list = append(list, *user)
}
if err := rows.Err(); err != nil {
fmt.Println(err)
}
As you can see for rows.Scan() , I have to write all columns , and I don't think it's a good way for 20 or more columns .
How to scan in a clear way.

It's a good practice for using reflect:
for rows.Next() {
user := Users{}
s := reflect.ValueOf(&user).Elem()
numCols := s.NumField()
columns := make([]interface{}, numCols)
for i := 0; i < numCols; i++ {
field := s.Field(i)
columns[i] = field.Addr().Interface()
}
err := rows.Scan(columns...)
if err != nil {
log.Fatal(err)
}
log.Println(user)
}

You may consider using jmoiron's sqlx package. It has support for assigning to a struct.
Excerpt from the readme:
type Place struct {
Country string
City sql.NullString
TelCode int
}
places := []Place{}
err = db.Select(&places, "SELECT * FROM place ORDER BY telcode ASC")
if err != nil {
fmt.Println(err)
return
}

Related

How to pass in variable length variable to statement.Exec?

For example
func Query(myvarlist []string) {
stmt, err := tx.Prepare("SELECT * FROM mytable WHERE
myvar = $1 AND myvar2 = $2".......)
defer stmt.Close()
if _, err := stmt.Exec(myvarlist....); err != nil {
}
Can we pass in Exec a variable length variable?
You can do something like the this:
func Query() {
db, err := sql.Open("pgx", `postgresql://CONNSTRING`)
if err != nil {
panic(err)
}
queryParamMap := map[string]string{
"id": "6",
"name": "1",
}
// Build up statement and params
cols := make([]string, len(queryParamMap))
args := make([]any, len(queryParamMap))
i := 0
for k, v := range queryParamMap {
cols[i] = fmt.Sprintf("%s = $%d", k, i+1) // WARNING - SQL Injection possible here if col names are not sanitised
args[i] = v
i++
}
// Using Prepare because the question used it but this is only worthwhile if you will run stmt multiple times
stmt, err := db.Prepare(`SELECT id FROM devices WHERE ` + strings.Join(cols, " and "))
if err != nil {
panic(err)
}
defer stmt.Close()
rows, err := stmt.Query(args...)
if err != nil {
panic(err)
}
defer rows.Close()
for rows.Next() {
var id int
if err = rows.Scan(&id); err != nil {
panic(err)
}
fmt.Println("value:", id)
}
// Should check rows.Error() etc but this is just an example...
}
I've put the column names and values into a map because it was not clear where any extra column names would come from in your question but hopefully this provides the info you need.
This example is also using Query rather than Exec (because it's easier to test) but the same approach will work with Exec.
Note: Take a look at squirrel for an example of how to take this a lot further....

Update mongo db collection to create new field with unique values without impacting existing data using mongo go driver

I am new to mongo and mongo go driver. Need to add new field "uri" to my collection with existing data - using mongo go driver. New field needs to be populated with unique values so that unique index can be created on it. the collection uses _id as well, if there is a way we can populate new field based on _id field that will work as well.
I am trying below code, not sure how to populate unique values.
//Step1: update all documents to add new field with unique values
_, err := myColl.UpdateMany(
ctx,
bson.D{},// select all docs in collection
bson.D{
{"$set", bson.D{{"uri", GenerateRandomUniqueString()}}},
},
)
if err != nil {
return err
}
// then next step is to create index on this field:
key := bson.D{{"uri", 1}}
opt := options.Index().SetName("uri-index").SetUnique(true)
model := mongo.IndexModel{Keys: key, Options: opt}
_, err = myColl.Indexes().CreateOne(ctx, model)
if err != nil {
return err
}
Once the index is set up, old records will marked read only, but we can not delete those. New data will have unique 'uri' string value.
Any help is much appreciated.
Using above code fails while unique index creation, as the same value is used for backfill.
I tried this as well:
func BackFillUri(db *mongo.Database) error {
myColl := db.Collection("myColl")
ctx := context.Background()
cursor, err := myColl.Find(ctx, bson.M{})
if err != nil {
return err
}
defer cursor.Close(ctx)
for cursor.Next(ctx) {
var ds bson.M
if err = cursor.Decode(&ds); err != nil {
return err
}
_, err1 := myColl.UpdateOne(
ctx,
bson.D{"_id": ds.ObjectId},
bson.D{
{"$set", bson.D{{"uri", rand.Float64()}}},
},
)
if err1 != nil {
return err1
}
}
return nil
}
But i am getting quite a few errors and not sure if any of the above logic is correct
I finally used below code, hope it helps someone who's new like me :-)
const charset = "abcdefghijklmnopqrstuvwxyz" +
"ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
func RandomUniqueString(length int) string {
return StringWithCharset(length, charset)
}
var seededRand *rand.Rand = rand.New(
rand.NewSource(time.Now().UnixNano()))
func StringWithCharset(length int, charset string) string {
b := make([]byte, length)
for i := range b {
b[i] = charset[seededRand.Intn(len(charset))]
}
return string(b)
}
// Adds index on uri
func AddUriIndex(db *mongo.Database) error {
mycoll := db.Collection("mycoll")
ctx := context.Background()
//backfill code starts
type attribute struct {
Key string `bson:"key"`
Value interface{} `bson:"value"`
}
type item struct {
ID primitive.ObjectID `bson:"_id"`
ResourceAttributes []attribute `bson:"resourceAttributes,omitempty"`
}
cursor, err := mycoll.Find(ctx, primitive.M{})
if err != nil {
return err
}
defer cursor.Close(ctx)
for cursor.Next(ctx) {
var result item
if err = cursor.Decode(&result); err != nil {
return err
}
//fmt.Println("Found() result:", result)
filter := primitive.M{"_id": result.ID}
update := primitive.M{"$set": primitive.M{"uri": RandomUniqueString(32)}}
if _, err := mycoll.UpdateOne(ctx, filter, update); err != nil {
return err
}
}
//add uri-index starts
key := bson.D{{"uri", 1}}
opt := options.Index().
SetName("uri-index").
SetUnique(true)
model := mongo.IndexModel{Keys: key, Options: opt}
_, err = mycoll.Indexes().CreateOne(ctx, model)
if err != nil {
return err
}
return nil
}

Reading BigQuery in Golang. Not all expected results are given. What to do?

Given that the SQL is running perfectly in Query Editor. Still after assigning it to a struct, the data seems to have different values. Why is it like that?
var RunQuery = func(req *http.Request, query string)(*bigquery.RowIterator, error){
ctx := appengine.NewContext(req)
ctxWithDeadline, _ := context.WithTimeout(ctx, 30*time.Minute)
bqClient, bqErr := bigquery.NewClient(ctxWithDeadline, project, option.WithCredentialsFile(serviceAccount))
if bqErr != nil {
log.Errorf(ctx, "%v", bqErr)
return nil, bqErr
}
q := bqClient.Query(query)
job, err := q.Run(ctx)
if err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
status, err := job.Wait(ctx)
if err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
if err := status.Err(); err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
it, err := job.Read(ctx)
if err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
log.Infof(ctx, "Total Rows: %v", it.TotalRows)
return it, nil
}
type Customers struct {
CustomerName string `bigquery:"customer_name"`
CustomerAge int `bigquery:"customer_age"`
}
var rowsRead int
func main() {
query := `SELECT
name as customer_name,
age as customer_age
FROM customer_table
WHERE customerStatus = '0'`
customerInformation, customerInfoErr := RunQuery(req, query, false)
if customerInfoErr != nil {
log.Errorf(ctx, "Fetching customer information error :: %v", customerInfoErr)
return
}
for {
var row Customers
err := customerInformation.Next(&row)
log.Infof(ctx, "row %v", row)
if err == iterator.Done {
log.Infof(ctx, "ITERATION COMPLETE. Rows read %v", rowsRead)
break
}
rowsRead++
}
}
Let's say i have Query Results of
customer_name|customer_age
cat | 2
dog | 3
horse | 10
But after assigning it to a struct the results was
customer_name|customer_age
"" | 2
dog | ""
"" | ""
Why is it like this? i even tested it on chunk where i set the limit to 1000, still the same results. But the query results in Query Editor is what i expect
Solved it using Value Loader bigquery.Value. Instead of using expected struct in mapping the query results. used map[string]bigquery.Value. Still don't know why mapping query results with expected struct is not working perfectly. Here is my solution.
for {
row := make(map[string]bigquery.Value)
err := customerInformation.Next(&row)
log.Infof(ctx, "row %v", row)
if err == iterator.Done {
log.Infof(ctx, "ITERATION COMPLETE. Rows read %v", rowsRead)
break
}
rowsRead++
}
From the documentation:
If dst is a pointer to a struct, each column in the schema will be matched with an exported field of the struct that has the same name, ignoring the case. Unmatched schema columns and struct fields will be ignored.
cloud.google.com/go/bigquery
Here you try to resolve customer_age to a struct property named CustomerAge. If you update it to Customer_Age or customer_age it should work.

Fill a slice with field pointers of a struct

I'm creating a function which receives an interface{}, which will always be a struct, but it could be any struct.
I need to fill one slice with pointers of all the fields of the struct received. The place in the code below where I need to pick the field pointer is flagged with **FIELD POINTER**
My final goal is create a function to receive a struct equivalent to the return of the query sent in the parameter sqlQuery. I want to create a dynamic function to perform any type of query to select, always using the struct received for .Scan.
It may be that I'm thinking the wrong way, I'm still starting Golang.
func QuerySelect(entity interface{}, sqlQuery string) {
val := reflect.Indirect(reflect.ValueOf(entity))
psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+
"password=%s dbname=%s sslmode=disable",
host, port, user, password, dbname)
// Validate database params
db, err := sql.Open(driver, psqlInfo)
// If returned any error
if err != nil {
panic(err)
}
// Validate connection with database
err = db.Ping()
if err != nil {
panic(err)
}
// Execute query
rows, err := db.Query(sqlQuery)
countColumns := val.Type().NumField()
var allRows []interface{}
for rows.Next() {
columnsPointers := make([]interface{}, countColumns)
for i := 0; i < countColumns; i++ {
columnsPointers[i] = **FIELD POINTER (entity struct)**
}
if err := rows.Scan(columnsPointers...); err != nil {
log.Fatal(err)
}
allRows = append(allRows, entity)
}
}
Per mkopriva's commentary on the question, use the following to get the address of the field:
for i := 0; i < countColumns; i++ {
columnsPointers[i] = val.Field(i).Addr().Interface()
}

How to append objects to a slice?

I am new to golang and I'd like to aggregaet query results into a results slice to be pushed to the browser. Here is the code:
type Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
Tasks []Task
}
type Cats struct {
category Category
}
func CategoriesCtrl(w http.ResponseWriter, req *http.Request) {
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("taskdb").C("categories")
iter := c.Find(nil).Iter()
result := Category{}
results := []Cats //Here is the problem
for iter.Next(&result) {
results = append(results, result)
fmt.Printf("Category:%s, Description:%s\n", result.Name, result.Description)
tasks := result.Tasks
for _, v := range tasks {
fmt.Printf("Task:%s Due:%v\n", v.Description, v.Due)
}
}
if err = iter.Close(); err != nil {
log.Fatal(err)
}
fmt.Fprint(w, results)
}
But instead I get
type []Cats is not an expression
How can I fix this?
You can say
results := make([]Cats, 0)
or
var results []Cats
or
results := []Cats{}
instead.
You can use results := make([]Cats, len) instead, where len is the initial length of slice.
results := []Cats{} will also work.
If you use var results []Cats, its initial value is nil so you'd need to initialize it before using append.

Resources