When should you initialize a new variable and when you should not? - performance

I'm looking at the code examples sql.query and i'm a bit confused by the way the variables are initialized. As far as I understand the var keyword initialize the variable but if you already have a such variable it's better to 'reuse' it instead to reinitialize it. I'm aware that I might have misunderstood the golang specs so I hope this question would help me (and perhaps other folks) get it right.
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
Why is the "name" variable initialized within the loop and not outside the loop ? (see below). Isn't it less performant to reinitialize it on each loop ?
//how I would do this
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
var name string //outside the loop
for rows.Next() {
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
or even better use a pointer
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
name := new(string) //pointer outside the loop
for rows.Next() {
if err := rows.Scan(name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}

Unless you have determined that the allocation is a performance bottleneck, I wouldn't consider such a premature optimisation. After all, it might not even make a difference, so it is best to err on the side of readability/maintainability.
In general, I'd suggest using the smallest scope for your variables that makes sense. If they are stack allocated, then they will be quite cheap -- assuming space is available, it probably just involves initialising the variable to zero or its initial value. Stack allocated variables scoped within a loop will probably end up with the same memory location each time through the loop too, so there isn't much to be gained from moving them out.
With that said, it isn't always obvious when a variable will be allocated on the stack. If the compiler decides that the pointer passed to row.Scan could possibly be retained past the function call (that is, it escapes), then name will be allocated on the heap even though it has been defined with var.
Similarly if the escape analysis determines that the variable doesn't escape, the version that creates the string variable with new may decide to place it on the stack.

Related

G110: Potential DoS vulnerability via decompression bomb (gosec)

I'm getting the following golintci message:
testdrive/utils.go:92:16: G110: Potential DoS vulnerability via decompression bomb (gosec)
if _, err := io.Copy(targetFile, fileReader); err != nil {
^
Read the corresponding CWE and I'm not clear on how this is expected to be corrected.
Please offer pointers.
func unzip(archive, target string) error {
reader, err := zip.OpenReader(archive)
if err != nil {
return err
}
for _, file := range reader.File {
path := filepath.Join(target, file.Name) // nolint: gosec
if file.FileInfo().IsDir() {
if err := os.MkdirAll(path, file.Mode()); err != nil {
return err
}
continue
}
fileReader, err := file.Open()
if err != nil {
return err
}
defer fileReader.Close() // nolint: errcheck
targetFile, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.Mode())
if err != nil {
return err
}
defer targetFile.Close() // nolint: errcheck
if _, err := io.Copy(targetFile, fileReader); err != nil {
return err
}
}
return nil
}
The warning you get comes from a rule provided in gosec.
The rule specifically detects usage of io.Copy on file decompression.
This is a potential issue because io.Copy:
copies from src to dst until either EOF is reached on src or an error occurs.
So, a malicious payload might cause your program to decompress an unexpectedly big amount of data and go out of memory, causing denial of service as mentioned in the warning message.
In particular, gosec will check (source) the AST of your program and warn you about usage of io.Copy or io.CopyBuffer together with any one of the following:
"compress/gzip".NewReader
"compress/zlib".NewReader or NewReaderDict
"compress/bzip2".NewReader
"compress/flate".NewReader or NewReaderDict
"compress/lzw".NewReader
"archive/tar".NewReader
"archive/zip".NewReader
"*archive/zip".File.Open
Using io.CopyN removes the warning because (quote) it "copies n bytes (or until an error) from src to dst", thus giving you (the program writer) control of how many bytes to copy. So you could pass an arbitrarily large n that you set based on the available resources of your application, or copy in chunks.
Based on various pointers provided, replaced
if _, err := io.Copy(targetFile, fileReader); err != nil {
return err
}
with
for {
_, err := io.CopyN(targetFile, fileReader, 1024)
if err != nil {
if err == io.EOF {
break
}
return err
}
}
PS while this helps memory footprint, this wouldn't help a DDOS attack copying very long and/or infinite stream ...
Assuming that you're working on compressed data, you need to use io.CopyN.
You can try a workaround with --nocompress flag. But this will cause the data to be included uncompressed.
See the following PR and related issue : https://github.com/go-bindata/go-bindata/pull/50

CheckErr(err) function crashing in golang

If I run this code everything works fine and nothing is apparently wrong in the end result (the right id is being printed).
stmt, err := db.Conn.Prepare("INSERT INTO tablename a VALUES (?)")
CheckErr(err)
defer stmt.Close()
res, err := stmt.Exec(&t.Id)
CheckErr(err)
id, err := res.LastInsertId()
fmt.Println(id)
But when I add another CheckErr(err) at the end and I run, I get this:
runtime error: invalid memory address or nil pointer dereference
This is the first thing in the stack trace after the panics:
id, err := res.LastInsertId()
And no id is printed, which is weird since the println comes before the new CheckErr(err)
Any idea why this is happening? I'm clueless
func CheckErr(err error) {
if err != nil {
raven.CaptureErrorAndWait(err, nil)
}
}
CheckErr(err) does not exit or return if err != nil. The program will continue to execute to id, err := res.LastInsertId() and panic if res is nil.

Is there a way to compose potentially failing operations in Go?

Most go code I read contains frequent occurrences of the following pattern:
result1, err := failingOp1()
if err != nil {
return err
}
dependingResult, err := failingOp2(result1)
if err != nil {
return err
}
// do stuff with dependingResult
In functional programming we have the Either monad and its cousins (e.g. Scala's Try) that allow us to compose failing operations without constantly repeating ourselves.
Is there a go equivalent that helps decluttering the code?
Reading up a bit further, in particular this SO answer, it seems idiomatic go prefers handling errors at the call-site rather than propagating the potential error upwards (which the monadic approach favours).
Following this line of thinking:
func wrapFailingOp1() ResultType {
result1, err := failingOp1()
if err != nil {
return defaultRTOrPanic()
}
return result1
}
func wrapFailingOp2(result1 ResultType) DependingResultType {
dependingResult, err := failingOp2(result1)
if err != nil {
return defaultDRTOrPanic()
}
return dependingResult
}
x := wrapFailingOp1()
y := wrapFailingOp2(x)

Golang - why is string slice element not included in exec cat unless I sort it

I have a slightly funky issue in golang. Essentially I have a slice of strings which represent file paths. I then run a cat against those filepaths to combine the files before sorting, deduping, etc.
here is the section of code (where 'applicableReductions' is the string slice):
applicableReductions := []string{}
for _, fqFromListName := range fqFromListNames {
filePath := GetFilePath()
//BROKE CODE GOES HERE
}
applicableReductions = append(applicableReductions, filePath)
fileOut, err := os.Create(toListWriteTmpFilePath)
if err != nil {
return err
}
cat := exec.Command("cat", applicableReductions...)
catStdOut, err := cat.StdoutPipe()
if err != nil {
return err
}
go func(cat *exec.Cmd) error {
if err := cat.Start(); err != nil {
return fmt.Errorf("File reduction error (cat) : %s", err)
}
return nil
}(cat)
// Init Writer & write file
writer := bufio.NewWriter(fileOut)
defer writer.Flush()
_, err = io.Copy(writer, catStdOut)
if err != nil {
return err
}
if err = cat.Wait(); err != nil {
return err
}
fDiff.StandardiseData(fileOut, toListUpdateFolderPath, list.Name)
The above works fine. The problem comes when I try to append a new ele to the array. I have a seperate function which creates a new file from db content which is then added to the applicableReductions slice.
func RetrieveDomainsFromDB(collection *Collection, listName, outputPath string) error {
domains, err := domainReviews.GetDomainsForList(listName)
if err != nil {
return err
}
if len(domains) < 1 {
return ErrNoDomainReviewsForList
}
fh, err := os.OpenFile(outputPath, os.O_RDWR, 0774)
if err != nil {
fh, err = os.Create(outputPath)
if err != nil {
return err
}
}
defer fh.Close()
_, err = fh.WriteString(strings.Join(domains, "\n"))
if err != nil {
return err
}
return nil
}
If I call the above function and append the filePath to the applicableReduction slice, it is in there but doesnt get called by cat.
To clarify, when I put the following where it says BROKE CODE GOES HERE:
if dbSource {
err = r.RetrieveDomainsFromDB(collection, ToListName, filePath)
if err != nil {
return err
continue
}
}
The filepath can be seen when doing fmt.Println(applicableReductions) but the content of the files contents are not seen in the cat output file.
I thought perhaps a delay in the file being written so i tried adding a time.wait, tis didnt help. However the solution I found was to sort the slice, e.g this code above the call to exec cat solves the problem but I dont know why:
sort.Strings(applicableReductions)
I have confirmed all files present on both successful and unsucessful runs the only difference is without the sort, the content of the final appended file is missing
An explanation from a go-pro out there would be very much appreciated, let me know if you need more info, debug - happy to oblige to understand
UPDATE
It has been suggested that this is the same issue as here: Golang append an item to a slice, I think I understand the issue there and I'm not saying this isnt the same but I cannot see the same thing happenning - the slice in question is not touched from outside the main function (e.g. no editing of the slice in RetrieveDomainsFromDB function), I create the slice before a loop, append to it within a loop and then use it after the loop - Ive added an example at the top to show how the slice is built - please could someone clarify where this slice is being copied if this is the case
UPDATE AND CLOSE
Please close question - the issue was unrelated to the use of a string slice. Turns out that I was reading from the final output file before bufio-writer had been flushed (at end of function before defer flush kicked in on function return)
I think the sorting was just re-arranging the problem so I didnt notice it persisted or possibly giving some time for the buffer to flush. Either way sorted now with a manual call to flush.
Thanks for all help provided

Go Using db.Query to return more than one column

In the Go SQL docs they give an example here of a query that only returns 1 column (poor example in my opinion, at least return 2...)
age := 27
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
The docs state here that
Scan copies the columns in the current row into the values pointed at by dest.
How does this work with a struct, lets say I have a struct
type User struct{
Name string
Age int
}
and I modify my query to SELECT name, age from users where age=?
How do I unpack *Rows into my struct? I did find this example, but it didn't deal with structs. I will be following Active Record pattern conventions so my structs will map to my database via snake case conversion.
Looking at the source, it seems the copy is done with ... syntax on destination pointers:
func (rs *Rows) Scan(dest ...interface{}) error
So in your example, you can do for instance:
for rows.Next() {
u := User{} // An empty user
...
if err := rows.Scan(&u.Name, &u.Age); err != nil {
...
}
}
As long as you pass the exact number of pointers, this should work, whether they are from a struct or not.

Resources