What is the "idiomatic" version of this function? - go

Trying to understand the mentality of Go. I wrote the following function which looks for *.txt files of a folder that have a date in the filename, get the latest date and return that date.
func getLatestDate(path string) (time.Time, error) {
if fns, e := filepath.Glob(filepath.Join(path, "*.txt")); e == nil {
re, _ := regexp.Compile(`_([0-9]{8}).txt$`)
max := ""
for _, fn := range fns {
if ms := re.FindStringSubmatch(fn); ms != nil {
if ms[1] > max {
max = ms[1]
}
}
}
date, _ := time.Parse("20060102", max)
return date, nil
} else {
return time.Time{}, e
}
}
What would be the more idiomatic version of this function, if there is one?

Here is my take
Use MustCompile to compile a static regexp. This will panic if it doesn't compile and saves an error check
Hoist compiling the regexp out of the function - you only need it compiled once. Note that I've called it with a lowercase initial letter so it won't be visible outside the package.
Use an early return when checking errors - this saves indentation and is idiomatic go
Use named return parameters for those early returns - saves defining nil values for types and typing in general (not to everyone's taste though)
return time.Parse directly which checks the errors (you weren't before)
The code
var dateRe = regexp.MustCompile(`_([0-9]{8}).txt$`)
func getLatestDate(path string) (date time.Time, err error) {
fns, err := filepath.Glob(filepath.Join(path, "*.txt"))
if err != nil {
return
}
max := ""
for _, fn := range fns {
if ms := dateRe.FindStringSubmatch(fn); ms != nil {
if ms[1] > max {
max = ms[1]
}
}
}
return time.Parse("20060102", max)
}

Here's how I would have written it. Don't ignore errors, use guard clauses for error handling, and don't recompile regexps inside a loop.
var datePat = regexp.MustCompile(`_([0-9]{8}).txt$`)
func getLatestDate(path string) (time.Time, error) {
fns, err := filepath.Glob(filepath.Join(path, "*.txt"))
if err != nil {
return time.Time{}, err
}
max := time.Time{}
for _, fn := range fns {
if ms := re.FindStringSubmatch(fn); ms != nil {
if t, err := time.Parse("20060102", ms[1]); err == nil && t.After(max) {
max = t
}
}
}
return max, nil
}

Related

Testing with Golang, redis and time

I was trying to test a bit with Redis for the first time and I bumped into some confusion with HGET/HSET/HGETALL. My main problem was that I needed to store time, and I wanted to use a hash as I'll continuously update the time.
At first I read about how a MarshalBinary function such as this would save me:
func (f Foo) MarshalBinary() ([]byte, error) {
return json.Marshal(f)
}
What that did was that it saved the struct as a json string, but only as a string and not as an actual Redis hash. What I ended up doing in the end was a fairly large boilerplate code that makes my struct I want to save into a map, and that one is properly stored as a hash in Redis.
type Foo struct {
Number int `json:"number"`
ATime time.Time `json:"atime"`
String string `json:"astring"`
}
func (f Foo) toRedis() map[string]interface{} {
res := make(map[string]interface{})
rt := reflect.TypeOf(f)
rv := reflect.ValueOf(f)
if rt.Kind() == reflect.Ptr {
rt = rt.Elem()
rv = rv.Elem()
}
for i := 0; i < rt.NumField(); i++ {
f := rt.Field(i)
v := rv.Field(i)
switch t := v.Interface().(type) {
case time.Time:
res[f.Tag.Get("json")] = t.Format(time.RFC3339)
default:
res[f.Tag.Get("json")] = t
}
}
return res
}
Then to parse back into my Foo struct when calling HGetAll(..).Result(), I'm getting the result as a map[string]string and create a new Foo with these functions:
func setRequestParam(arg *Foo, i int, value interface{}) {
v := reflect.ValueOf(arg).Elem()
f := v.Field(i)
if f.IsValid() {
if f.CanSet() {
if f.Kind() == reflect.String {
f.SetString(value.(string))
return
} else if f.Kind() == reflect.Int {
f.Set(reflect.ValueOf(value))
return
} else if f.Kind() == reflect.Struct {
f.Set(reflect.ValueOf(value))
}
}
}
}
func fromRedis(data map[string]string) (f Foo) {
rt := reflect.TypeOf(f)
rv := reflect.ValueOf(f)
for i := 0; i < rt.NumField(); i++ {
field := rt.Field(i)
v := rv.Field(i)
switch v.Interface().(type) {
case time.Time:
if val, ok := data[field.Tag.Get("json")]; ok {
if ti, err := time.Parse(time.RFC3339, val); err == nil {
setRequestParam(&f, i, ti)
}
}
case int:
if val, ok := data[field.Tag.Get("json")]; ok {
in, _ := strconv.ParseInt(val, 10, 32)
setRequestParam(&f, i, int(in))
}
default:
if val, ok := data[field.Tag.Get("json")]; ok {
setRequestParam(&f, i, val)
}
}
}
return
}
The whole code in its ungloryness is here
I'm thinking that there must be a saner way to solve this problem? Or am I forced to do something like this? The struct I need to store only contains ints, strings and time.Times.
*edit
The comment field is a bit short so doing an edit instead:
I did originally solve it like 'The Fool' suggested in comments and as an answer. The reason I changed to the above part, while more complex a solution, I think it's more robust for changes. If I go with a hard coded map solution, I'd "have to" have:
Constants with hash keys for the fields, since they'll be used at least in two places (from and to Redis), it'll be a place for silly mistakes not picked up by the compiler. Can of course skip that but knowing my own spelling it's likely to happen
If someone just wants to add a new field and doesn't know the code well, it will compile just fine but the new field won't be added in Redis. An easy mistake to do, especially for junior developers being a bit naive, or seniors with too much confidence.
I can put these helper functions in a library, and things will just magically work for all our code when a time or complex type is needed.
My intended question/hope though was: Do I really have to jump through hoops like this to store time in Redis hashes with go? Fair, time.Time isn't a primitive and Redis isn't a (no)sql database, but I would consider timestamps in cache a very common use case (in my case a heartbeat to keep track of timed out sessions together with metadata enough to permanently store it, thus the need to update them). But maybe I'm misusing Redis, and I should rather have two entries, one for the data and one for the timestamp, which would then leave me with two simple get/set functions taking in time.Time and returning time.Time.
You can use redigo/redis#Args.AddFlat to convert struct to redis hash we can map the value using redis tag.
package main
import (
"fmt"
"time"
"github.com/gomodule/redigo/redis"
)
type Foo struct {
Number int64 `json:"number" redis:"number"`
ATime time.Time `json:"atime" redis:"atime"`
AString string `json:"astring" redis:"astring"`
}
func main() {
c, err := redis.Dial("tcp", ":6379")
if err != nil {
fmt.Println(err)
return
}
defer c.Close()
t1 := time.Now().UTC()
var foo Foo
foo.Number = 10000000000
foo.ATime = t1
foo.AString = "Hello"
tmp := redis.Args{}.Add("id1").AddFlat(&foo)
if _, err := c.Do("HMSET", tmp...); err != nil {
fmt.Println(err)
return
}
v, err := redis.StringMap(c.Do("HGETALL", "id1"))
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%#v\n", v)
}
Then to update ATime you can use redis HSET
if _, err := c.Do("HMSET", "id1", "atime", t1.Add(-time.Hour * (60 * 60 * 24))); err != nil {
fmt.Println(err)
return
}
And to retrieve it back to struct we have to do some reflect magic
func structFromMap(src map[string]string, dst interface{}) error {
dt := reflect.TypeOf(dst).Elem()
dv := reflect.ValueOf(dst).Elem()
for i := 0; i < dt.NumField(); i++ {
sf := dt.Field(i)
sv := dv.Field(i)
if v, ok := src[strings.ToLower(sf.Name)]; ok {
switch sv.Interface().(type) {
case time.Time:
format := "2006-01-02 15:04:05 -0700 MST"
ti, err := time.Parse(format, v)
if err != nil {
return err
}
sv.Set(reflect.ValueOf(ti))
case int, int64:
x, err := strconv.ParseInt(v, 10, sv.Type().Bits())
if err != nil {
return err
}
sv.SetInt(x)
default:
sv.SetString(v)
}
}
}
return nil
}
Final Code
package main
import (
"fmt"
"time"
"reflect"
"strings"
"strconv"
"github.com/gomodule/redigo/redis"
)
type Foo struct {
Number int64 `json:"number" redis:"number"`
ATime time.Time `json:"atime" redis:"atime"`
AString string `json:"astring" redis:"astring"`
}
func main() {
c, err := redis.Dial("tcp", ":6379")
if err != nil {
fmt.Println(err)
return
}
defer c.Close()
t1 := time.Now().UTC()
var foo Foo
foo.Number = 10000000000
foo.ATime = t1
foo.AString = "Hello"
tmp := redis.Args{}.Add("id1").AddFlat(&foo)
if _, err := c.Do("HMSET", tmp...); err != nil {
fmt.Println(err)
return
}
v, err := redis.StringMap(c.Do("HGETALL", "id1"))
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%#v\n", v)
if _, err := c.Do("HMSET", "id1", "atime", t1.Add(-time.Hour * (60 * 60 * 24))); err != nil {
fmt.Println(err)
return
}
var foo2 Foo
structFromMap(v, &foo2)
fmt.Printf("%#v\n", foo2)
}
func structFromMap(src map[string]string, dst interface{}) error {
dt := reflect.TypeOf(dst).Elem()
dv := reflect.ValueOf(dst).Elem()
for i := 0; i < dt.NumField(); i++ {
sf := dt.Field(i)
sv := dv.Field(i)
if v, ok := src[strings.ToLower(sf.Name)]; ok {
switch sv.Interface().(type) {
case time.Time:
format := "2006-01-02 15:04:05 -0700 MST"
ti, err := time.Parse(format, v)
if err != nil {
return err
}
sv.Set(reflect.ValueOf(ti))
case int, int64:
x, err := strconv.ParseInt(v, 10, sv.Type().Bits())
if err != nil {
return err
}
sv.SetInt(x)
default:
sv.SetString(v)
}
}
}
return nil
}
Note: The struct field name is matched with the redis tag

Read and merge two Yaml files in go language

Assuming we have two yaml files
master.yaml
someProperty: "someVaue"
anotherProperty: "anotherValue"
override.yaml
someProperty: "overriddenVaue"
Is it possible to unmarshall, merge, and then write those changes to a file without having to define a struct for every property in the yaml file?
The master file has over 500 properties in it that are not at all important to the service at this point of execution, so ideally I'd be able to just unmarshal into a map, do a merge and write out in yaml again but I'm relatively new to go so wanted some opinions.
I've got some code to read the yaml into an interface but i'm unsure on the best approach to then merge the two.
var masterYaml interface{}
yamlBytes, _ := ioutil.ReadFile("master.yaml")
yaml.Unmarshal(yamlBytes, &masterYaml)
var overrideYaml interface{}
yamlBytes, _ = ioutil.ReadFile("override.yaml")
yaml.Unmarshal(yamlBytes, &overrideYaml)
I've looked into libraries like mergo but i'm not sure if that's the right approach.
I'm hoping that after the master I would be able to write out to file with properties
someProperty: "overriddenVaue"
anotherProperty: "anotherValue"
Assuming that you just want to merge at the top level, you can unmarshal into maps of type map[string]interface{}, as follows:
package main
import (
"io/ioutil"
"gopkg.in/yaml.v2"
)
func main() {
var master map[string]interface{}
bs, err := ioutil.ReadFile("master.yaml")
if err != nil {
panic(err)
}
if err := yaml.Unmarshal(bs, &master); err != nil {
panic(err)
}
var override map[string]interface{}
bs, err = ioutil.ReadFile("override.yaml")
if err != nil {
panic(err)
}
if err := yaml.Unmarshal(bs, &override); err != nil {
panic(err)
}
for k, v := range override {
master[k] = v
}
bs, err = yaml.Marshal(master)
if err != nil {
panic(err)
}
if err := ioutil.WriteFile("merged.yaml", bs, 0644); err != nil {
panic(err)
}
}
For a broader solution (with n input files), you can use this function. I have used #robox answer to do my solution:
func ReadValues(filenames ...string) (string, error) {
if len(filenames) <= 0 {
return "", errors.New("You must provide at least one filename for reading Values")
}
var resultValues map[string]interface{}
for _, filename := range filenames {
var override map[string]interface{}
bs, err := ioutil.ReadFile(filename)
if err != nil {
log.Info(err)
continue
}
if err := yaml.Unmarshal(bs, &override); err != nil {
log.Info(err)
continue
}
//check if is nil. This will only happen for the first filename
if resultValues == nil {
resultValues = override
} else {
for k, v := range override {
resultValues[k] = v
}
}
}
bs, err := yaml.Marshal(resultValues)
if err != nil {
log.Info(err)
return "", err
}
return string(bs), nil
}
So for this example you should call it with this order:
result, _ := ReadValues("master.yaml", "overwrite.yaml")
In the case you have an extra file newFile.yaml, you could also use this function:
result, _ := ReadValues("master.yaml", "overwrite.yaml", "newFile.yaml")
DEEP MERGE TWO YAML FILES
package main
import (
"fmt"
"io/ioutil"
"sigs.k8s.io/yaml"
)
func main() {
// declare two map to hold the yaml content
base := map[string]interface{}{}
currentMap := map[string]interface{}{}
// read one yaml file
data, _ := ioutil.ReadFile("conf.yaml")
if err := yaml.Unmarshal(data, &base); err != nil {
}
// read another yaml file
data1, _ := ioutil.ReadFile("conf1.yaml")
if err := yaml.Unmarshal(data1, &currentMap); err != nil {
}
// merge both yaml data recursively
base = mergeMaps(base, currentMap)
// print merged map
fmt.Println(base)
}
func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
out := make(map[string]interface{}, len(a))
for k, v := range a {
out[k] = v
}
for k, v := range b {
if v, ok := v.(map[string]interface{}); ok {
if bv, ok := out[k]; ok {
if bv, ok := bv.(map[string]interface{}); ok {
out[k] = mergeMaps(bv, v)
continue
}
}
}
out[k] = v
}
return out
}

http.ResponseWriter.Write with interface{}

I'm running a SQL query using the sample from denisenkom but coupled with a http.ResponseWriterand I'm struggling with interface{} type conversion. There are a few posts that are close to what I'm doing but the solution seems kind of heavy handed and always use fmt (which I'm not using).
Note that my query works and returns a result. I'm just trying to display that result.
Here is my code that I think is relatively close but doesn't work. I've tried a couple other things but none even compile.
vals := make([]interface{}, len(cols))
for i := 0; i < len(cols); i++ {
vals[i] = new(interface{})
if i != 0 {
w.Write([]byte("\t"))
}
w.Write([]byte(cols[i]))
}
for rows.Next() {
err = rows.Scan(vals...)
if err != nil {
w.Write([]byte("Row problem: " + err.Error()))
continue
}
for i := 0; i < len(vals); i++ {
if i != 0 {
w.Write([]byte("\t"))
}
//THIS IS THE PART I'M STUCK ON
switch v := vals[i].(type) {
case int:
w.Write([]byte("int!\n" + string(v)))
case int8:
w.Write([]byte("int8!\n" + string(v)))
//etc, etc
//more types
//etc, etc
case float64:
w.Write([]byte("float64!\n" + string(v))) //This fails, can't convert, will need something else
case string:
w.Write([]byte("string!\n" + v))
default:
w.Write([]byte("something else!\n"))
}
}
}
But isn't there a better way to dynamically check the underlying type and convert it to something readable? All I want to do is spit out the results of a query, this seems like I'm doing something wrong.
Note that it always hits the default case, even when it's explicitly the same type.
The example at denisekom prints to stdout using the fmt.Print family of functions. Change the example to print to a response writer by using the fmt.Fprint family of functions.
The fmt.Fprint functions write to the io.Writer specified as the first argument. The response writer satisfies the io.Writer interface. Here's the modified code from the original example where w is the http.ResponseWriter.
vals := make([]interface{}, len(cols))
for i := 0; i < len(cols); i++ {
vals[i] = new(interface{})
if i != 0 {
fmt.Fprint(w, "\t")
}
fmt.Fprint(w, cols[i])
}
fmt.Fprintln(w)
for rows.Next() {
err = rows.Scan(vals...)
if err != nil {
fmt.Fprintln(w, err)
continue
}
for i := 0; i < len(vals); i++ {
if i != 0 {
fmt.Fprint(w, "\t")
}
printValue(w, vals[i].(*interface{}))
}
fmt.Fprintln(w)
}
...
func printValue(w io.Writer, pval *interface{}) {
switch v := (*pval).(type) {
case nil:
fmt.Fprint(w, "NULL")
case bool:
if v {
fmt.Fprint(w, "1")
} else {
fmt.Fprint(w, "0")
}
case []byte:
fmt.Fprint(w, string(v))
case time.Time:
fmt.Fprint(w, v.Format("2006-01-02 15:04:05.999"))
default:
fmt.Fprint(w, v)
}
}
Regarding the code in the question: the expression string(v) is a string conversion. A string conversion does not convert numbers to a decimal representation as the code is assuming. See the spec for details on string conversions. Use either the fmt package as above or the strconv package to convert numbers to decimal strings. The type switch should be switch v := (*(vals[i].(*interface{})).(type) {. That's a bit of mess, and that leads to my next point.
The original example uses more indirection than needed. Here's a simplified version ready to be called with a response writer:
func exec(w io.Writer, db *sql.DB, cmd string) error {
rows, err := db.Query(cmd)
if err != nil {
return err
}
defer rows.Close()
cols, err := rows.Columns()
if err != nil {
return err
}
if cols == nil {
return nil
}
vals := make([]interface{}, len(cols))
args := make([]interface{}, len(cols))
for i := 0; i < len(cols); i++ {
args[i] = &vals[i]
if i != 0 {
fmt.Fprint(w, "\t")
}
fmt.Fprint(w, cols[i])
}
for rows.Next() {
err = rows.Scan(args...)
if err != nil {
fmt.Fprintln(w, err)
continue
}
for i := 0; i < len(vals); i++ {
if i != 0 {
fmt.Print("\t")
}
printValue(w, vals[i])
}
fmt.Fprintln(w)
}
if rows.Err() != nil {
return rows.Err()
}
return nil
}
func printValue(w io.Writer, v interface{}) {
switch v := v.(type) {
case nil:
fmt.Fprint(w, "NULL")
case bool:
if v {
fmt.Fprint(w, "1")
} else {
fmt.Fprint(w, "0")
}
case []byte:
fmt.Fprint(w, string(v))
case time.Time:
fmt.Fprint(w, v.Format("2006-01-02 15:04:05.999"))
default:
fmt.Fprint(w, v)
}
}

Use ast to get all function calls in a function

I'm trying to list all function call in a function using ast. But having trouble understanding how it is suppose to be used. I have been able to get this far.
set := token.NewFileSet()
packs, err := parser.ParseFile(set, serviceFile, nil, 0)
if err != nil {
fmt.Println("Failed to parse package:", err)
os.Exit(1)
}
funcs := []*ast.FuncDecl{}
for _, d := range packs.Decls {
if fn, isFn := d.(*ast.FuncDecl); isFn {
funcs = append(funcs, fn)
}
}
I have inspected funcs. I get to funcs[n1].Body.List[n2].
But after this i don't understand how i'm suppose to read the underlaying data.X.Fun.data.Sel.name (got it from evaluation in gogland) to get name of the function being called.
you can use ast.Inspect for that and use a switch on the type of the node.
for _, fun := range funcs {
ast.Inspect(fun, func(node ast.Node) bool {
switch n := node.(type) {
case *ast.CallExpr:
fmt.Println(n) // prints every func call expression
}
return true
})
}
Ok so what i found is that you have to a lot of casting to actually extract the data.
Here is an example on how to do extract the func call in a func.
for _, function := range funcs {
extractFuncCallInFunc(function.Body.List)
}
func extractFuncCallInFunc(stmts []ast.Stmt) {
for _, stmt := range funcs {
if exprStmt, ok := stmt.(*ast.ExprStmt); ok {
if call, ok := exprStmt.X.(*ast.CallExpr); ok {
if fun, ok := call.Fun.(*ast.SelectorExpr); ok {
funcName := fun.Sel.Name
}
}
}
}
}
I also found this with kind of help with finding out what you need to cast it to.
http://goast.yuroyoro.net/

Need faster way to list all datasets/tables in project

I am creating a utility that needs to be aware of all the datasets/tables that exist in my BigQuery project. My current code for getting this information is as follows (using Go API):
func populateExistingTableMap(service *bigquery.Service, cloudCtx context.Context, projectId string) (map[string]map[string]bool, error) {
tableMap := map[string]map[string]bool{}
call := service.Datasets.List(projectId)
//call.Fields("datasets/datasetReference")
if err := call.Pages(cloudCtx, func(page *bigquery.DatasetList) error {
for _, v := range page.Datasets {
if tableMap[v.DatasetReference.DatasetId] == nil {
tableMap[v.DatasetReference.DatasetId] = map[string]bool{}
}
table_call := service.Tables.List(projectId, v.DatasetReference.DatasetId)
//table_call.Fields("tables/tableReference")
if err := table_call.Pages(cloudCtx, func(page *bigquery.TableList) error {
for _, t := range page.Tables {
tableMap[v.DatasetReference.DatasetId][t.TableReference.TableId] = true
}
return nil
}); err != nil {
return errors.New("Error Parsing Table")
}
}
return nil
}); err != nil {
return tableMap, err
}
return tableMap, nil
}
For a project with about 5000 datasets, each with up to 10 tables, this code takes almost 15 minutes to return. Is there a faster way to iterate through the names of all existing datasets/tables? I have tried using the Fields method to return only the fields I need (you can see those lines commented out above), but that results in only 50 (exactly 50) of my datasets being returned.
Any ideas?
Here is an updated version of my code, with concurrency, that reduced the processing time from about 15 minutes to 3 minutes.
func populateExistingTableMap(service *bigquery.Service, cloudCtx context.Context, projectId string) (map[string]map[string]bool, error) {
tableMap = map[string]map[string]bool{}
call := service.Datasets.List(projectId)
//call.Fields("datasets/datasetReference")
if err := call.Pages(cloudCtx, func(page *bigquery.DatasetList) error {
var wg sync.WaitGroup
wg.Add(len(page.Datasets))
for _, v := range page.Datasets {
if tableMap[v.DatasetReference.DatasetId] == nil {
tableMap[v.DatasetReference.DatasetId] = map[string]bool{}
}
go func(service *bigquery.Service, datasetID string, projectId string) {
defer wg.Done()
table_call := service.Tables.List(projectId, datasetID)
//table_call.Fields("tables/tableReference")
if err := table_call.Pages(cloudCtx, func(page *bigquery.TableList) error {
for _, t := range page.Tables {
tableMap[datasetID][t.TableReference.TableId] = true
}
return nil // NOTE: returning a non-nil error stops pagination.
}); err != nil {
// TODO: Handle error.
fmt.Println(err)
}
}(service, v.DatasetReference.DatasetId, projectId)
}
wg.Wait()
return nil // NOTE: returning a non-nil error stops pagination.
}); err != nil {
return tableMap, err
// TODO: Handle error.
}
return tableMap, nil
}

Resources