I have a small Heroku app in which i print out name and age from each rows after query execution.
I want to avoid looping rows.Next(),Scan().. and just want to show what database returned after query execution which may be some data or error.
Can we directly dump data to a string for printing?
rows, err := db.Query("SELECT name FROM users WHERE age = $1", age)
if err != nil {
log.Fatal(err)
}
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
Pretty much: No.
The Query method is going to return a pointer to a Rows struct:
func (db *DB) Query(query string, args ...interface{}) (*Rows, error)
If you print that (fmt.Printf("%#v\n", rows)) you'll see something such as:
&sql.Rows{dc:(*sql.driverConn)(0xc8201225a0), releaseConn:(func(error)(0x4802c0), rowsi:(*pq.rows)(0xc820166700), closed:false, lastcols:[]driver.Value(nil), lasterr:error(nil), closeStmt:driver.Stmt(nil)}
...probably not what you want.
Those correspond to the Rows struct from the sql package (you'll notice the fields are not exported):
type Rows struct {
dc *driverConn // owned; must call releaseConn when closed to release
releaseConn func(error)
rowsi driver.Rows
closed bool
lastcols []driver.Value
lasterr error // non-nil only if closed is true
closeStmt driver.Stmt // if non-nil, statement to Close on close
}
You'll see []driver.Value (an interface from the driver package), that looks like where we can expect to find some useful, maybe even human readable data. But when directly printed it doesn't appear useful, it's even empty... So you have to somehow get at the underlying information. The sql package gives us the Next method to start with:
Next prepares the next result row for reading with the Scan method.
It returns true on success, or false if there is no next
result row or an error happened while preparing it. Err
should be consulted to distinguish between the two cases.
Every call to Scan, even the first one, must be preceded by a call to Next.
Next is going to make a []driver.Value the same size as the number of columns I have, which is accessible (within the sql package) through driver.Rows (the rowsi field) and populate it with values from the query.
After calling rows.Next() if you did the same fmt.Printf("%#v\n", rows) you should now see that []diver.Value is no longer empty but it's still not going to be anything that you can read, more likely something resembling:[]diver.Value{[]uint8{0x47, 0x65...
And since the field isn't exported you can't even try and convert it to something more meaningful. But the sql package gives us a means to do something with the data, which is Scan.
The Scan method is pretty concise, with lengthy comments that I won't paste here, but the really important bit is that it ranges over the columns in the current row you get from the Next method and calls convertAssign(dest[i], sv), which you can see here:
https://golang.org/src/database/sql/convert.go
It's pretty long but actually relatively simple, it essentially switches on the type of the source and destination and converts where it can, and copies from source to destination; the function comments tell us:
convertAssign copies to dest the value in src, converting it if possible. An error is returned if the copy would result in loss of information. dest should be a pointer type.
So now you have a method (Scan) which you can call directly and which hands you back converted values. Your code sample above is fine (except maybe the call to Fatal() on a Scan error).
It's important to realize that the sql package has to work with a specific driver, which is in turn implemented for specific database software, so there is quite some work going on behind the scenes.
I think your best bet if you want to hide/generalize the whole Query() ---> Next() ---> Scan() idiom is to drop it into another function which does it behind the scenes... write a package in which you abstract away that higher level implementation, as the sql package abstracts away some of the driver-specific details, the converting and copying, populating the Rows, etc.
Related
I have been learning about databases and wanted to implement one as well for learning purposes and not for production. I have a defined schema:
type Row struct {
ID int32
Username string
Email string
}
Now, currently, i am able to encode structs of this type to a file in an append only manner.
//Just to show i use a file for the encoding, it has missing details.
func NewEncoder(db *DB) *gob.Encoder{
return gob.NewEncoder(db.File)
}
func SerializeRow(r Row, encoder *gob.Encoder, db *DB) {
err := encoder.Encode(r)
if err != nil {
log.Println("encode error:", err)
}
}
Now, it's relatively easy to mimick a "select" statement by simply decoding the entire file using gob.decode
func DeserializeRow(decoder *gob.Decoder, db *DB){
var rows Row
db.File.Seek(0, 0)
err := decoder.Decode(&rows)
for err == nil {
if err != nil {
log.Println("decode error:", err)
}
fmt.Printf("%d %s %s\n", rows.ID, rows.Username, rows.Email)
err = decoder.Decode(&rows)
}
}
My current issue is, I want to be able to retrieve specific rows based on ID. I know sqlite uses 4kb paging, in the sense that serialized rows occupy a "page" ie. 4KB till a page can't hold them anymore, then another is created. How do I mimick such a behaviour using gob in the most simplistic and idiomatic way?
Seen: I have seen this and this
A Gob stream may contain type definitions and decoding instructions, so you can't seek a Gob stream. You can only read it from the beginning up to the point you find what you need.
A Gob stream is completely unsuitable for a database storage format in which you need to skip elements.
You could create a new encoder and serialize each records separately, in which case you could skip elements (by maintaining a file index storing which record starts at which position), but it would be terribly inefficient and redundant (as described in the linked answer, the speed and storage cost amortizes as you write more values of the same type, and always creating new encoders loses this gain).
A much better approach would be to not use encoding/gob for this, but rather define your own format. To efficiently support searches (select), you have to build some kind of index on the searchable columns / fields, else you still need to perform a full table scan.
i want to send the 2 parameters a lambda needs in order to work and it basically needs the value i want to search and as a second parameter the field where to find that value.
Now with no problem i've been able to access some other lambdas with that only need one parameter with a code like this.
func (s *resourceService) GetProject(ctx context.Context, name string) projectStruct {
payload, err := json.Marshal(name)
util.Logger.Debugf("Payload",payload)
invokeOutput, err := s.lambdaSvc.Invoke(ctx, &lambda.InvokeInput{
FunctionName: &s.getProject,
InvocationType: "RequestResponse",
Payload: payload,
})
if err != nil {
panic(err.Error())
}
var project projectStruct
err = json.Unmarshal(invokeOutput.Payload, &project)
if err != nil {
panic(err.Error())
}
util.Logger.Debugf("Invocation output [%v]", invokeOutput)
return project
}
now with 2 parameters i've had a lot of problems and tried a LOT of different approaches starting for adding another Payload value, creating a string with the 2 values and marshal it, marshaling both parameters and try and add them as the payload, even append both marshaled bytes array but i've been incapable of sending 2 parameters as the payload
Do you know the right way to do so? Please Help
Lambda functions only take one Payload. In V1 of the AWS SDK, InvokeInput takes one []byte parameter expressing JSON, as you already know.
You can structure your one Json Payload to carry a list. Looking at your example, Payload could look something like
["name","name"]
You could change your signature like so:
func (s *resourceService) GetProject(ctx context.Context, names []string) projectStruct
json.Marshal can handle marshaling a slice just as well as the elements within the slice, so the remaining code doesn't need to change.
Of course the receiving function must agree about the schema of the data it's being passed. If you wish to change from a string to a list of strings, that will be a breaking change. For that reason, Json schemas typically use named values instead of scalars.
[{ "Name": "Joan"},{"Name":"Susan"}]
You can add Age and Address without breaking the receiving function (though of course, it will ignore the new fields until you program it to ignore them).
Take time to Get to know JSON - it's a simple and expressive encoding standard that is reliably supported everywhere. JSON is a natural choice for encoding structured data in Go because JSON integrates well with Go's with structures, maps, and slices.
I'm using the Golang validate library to do some input error checking as part of an API (a silly demo API for learning purposes).
When one performs the validation a slice of errors is returned. In reality, the slice is made up of the validate library's struct BadField, which looks like this:
type BadField struct {
Field string
Err error
}
func (b BadField) Error() string {
return fmt.Sprintf("field %s is invalid: %v", b.Field, b.Err)
}
I'd like to pass around a more-specific slice, so rather than []error I would like have []BadField so that I can access the Field value.
So far I can't find a way of casting/converting from one to the other. Maybe there isn't one (due to the nature of go and slices). Maybe there's a package that will do this for me.
My initial implementation
The way I've come up with is to loop through the slice and cast each element individually.
errors := valueWithBadStuff.Validate()
validationErrors := make([]validate.BadField, len(errors))
for _, err := range errors {
validationError, ok := err.(validate.BadField)
if !ok {
panic("badarghfiremyeyes") // S/O purposes only
}
validationErrors = append(validationErrors, validationError)
}
Which feels kinda long for something "simple" but perhaps there's a more go idiomatic way? Or a nicer way?
For background, my intention (at the moment) is to take the slice of validation errors and pipe it back to the client as an array of JSON objects with the Field name and the error message (i.e. for a fictional age field: ["field_name": "age", "Cannot be less than 0"])
Just after the loop above I do more conversion to generate a slice of structs that are tagged with json that will actually go the client. The extra conversion may be duplication and pointless but, right now, it's purely a learning piece and I'll probably refactor it in an hour or two.
There's not really a "nicer" way of doing this. To convert a slice you have to basically do what you've already discovered.
If you are simply returning these errors to a client, you could probably avoid the need to typecast this at all.
Implement the JSON Marshaler interface and you can make your type will automatically output the JSON in the format you desire. For example, for the format you gave above this would be:
func (e BadField) MarshalJSON() ([]byte, error) {
return json.Marshal([]string{"field_name",e.Field,e.Err.Error()})
}
I suspect however that you would probably rather have a response something like:
[
{
"field":"age",
"error":"msg1"
},
{
"field":"name",
"error":"msg2"
}
]
To do this, you could simply add the JSON tags to the struct definition, e.g.
type BadField struct {
Field string `json:"field"`
Err error `json:"error"`
}
This would mean calling json.Marshal on a slice of []error which contains BadField instances would result in the JSON above.
It might be helpful to read more about JSON & Go
PS Consider if you want your methods to be value or pointer receivers
I have a function that basically looks like this:
func (db *datastoreDB) GetAllUsers(ctx context.Context) (*[]User, error) {
query := datastore.NewQuery("User")
var users []User
_, err := db.client.GetAll(ctx, query, &users)
return &users, nil
}
with the struct:
type User struct {
username string
password []byte
}
Now, if I try to call
users, err := models.DB.GetAllUsers(ctx)
log.Println(users)
then it will only print 1 user, even though there are many..
I tried to Print using users[0], users[1] but that returned errors, also tried with *users[1], &users[1], and for i, _ range users { log.Println(users[i]) }
Haven't quite been able to understand when/how to use * and & even though I read many online tutorials, so often just do trail and error.. I doubt there is anything wrong with my datastore GetAll function, so I assume I just fail to properly access/return the struct slice but feel like I tried everything..
Slices include pointers; in fact they are structs with a pointer to an array and some information as to where the slice begins and ends.
A slice is a descriptor of an array segment. It consists of a pointer to the array, the length of the segment, and its capacity (the maximum length of the segment)
Golang blog
The asterix before a type designates that type as a pointer (unless it already is a pointer, in that case it dereferences it). I think you probably meant to write []*User, which will expect a slice of pointers to Users. You can think of [] and User as distinct types.
To create a slice, the simplest way is probably with make();
You can try, instead of var users []User,
users := make([]*User, 0) // replace 0 with the amount of nil values in the slice you're looking to fill
Finally, you'll have to remove the & signs you place before users, as these pass the pointer to a value (but as I pointed out above, slices are already pointers)
To better understand pointers, Dave Cheney recently wrote a blog post titled Understand Go pointers in less than 800 words or your money back, you might find it useful.
I'm new to the Go language.
I'm making a small web application with Go, the Gorilla toolkit, and the Mustache template engine.
Everything works great so far.
I use hoisie/mustache and gorilla/sessions, but I'm struggling with passing variables from one to the other. I have a map[string]interface{} that I pass to the template engine. When a user is logged in, I want to take the user's session data and merge it with my map[string]interface{} so that the data becomes available for rendering.
The problem is that gorilla/sessions returns a map[interface{}]interface{} so the merge cannot be done (with the skills I have in this language).
I thought about extracting the string inside the interface{} variable (reflection?).
I also thought about making my session data a map[interface{}]interface{} just like what gorilla/sessions provides. But I'm new to Go and I don't know if that can be considered best practice. As a Java guy, I feel like working with variables of type Object.
I would like to know the best approach for this problem in your opinion.
Thanks in advance.
You'll need to perform type assertions: specifically this section of Effective Go.
str, ok := value.(string)
if ok {
fmt.Printf("string value is: %q\n", str)
} else {
fmt.Printf("value is not a string\n")
}
A more precise example given what you're trying to do:
if userID, ok := session.Values["userID"].(string); ok {
// User ID is set
} else {
// User ID is not set/wrong type; raise an error/HTTP 500/re-direct
}
type M map[string]interface{}
err := t.ExecuteTemplate(w, "user_form.tmpl", M{"current_user": userID})
if err != nil {
// handle it
}
What you're doing is ensuring that the userID you pull out of the interface{} container is actually a string. If it's not, you handle it (if you don't, you'll program will panic as per the docs).
If it is, you pass it to your template where you can access it as {{ .current_user }}. M is a quick shortcut that I use to avoid having to type out map[string]interface{} every time I call my template rendering function.