Search for a value in HashiCorp vault - go

Is there a way to search for a value in Hashicorp Vault? I am trying to write Golang code to search and list all locations a value appears in vault. It would be similar to golang's walk function on directories. Does anyone have a good approach for this? I was thinking of using concurrency to search vault for a value. Thank you
Below is a sample of the code I came up with. I am looking on how to make this faster by using concurrency. Is there a way to traverse a directory concurrently?
func walkDir(client *api.Client, path string) {
var value *api.Secret
var err error
if path != "" {
value, err = client.Logical().List(path)
} else {
path = vault_path
value, err = client.Logical().List(path)
}
if err != nil {
fmt.Println(err)
}
var datamap map[string]interface{}
datamap = value.Data
data := datamap["keys"].([]interface{})
for _, item := range data {
itemString := item.(string)
if strings.HasSuffix(itemString, "/") {
walkDir(client, path+itemString)
} else {
//its a secret
data := read(client, path+itemString)
if *searchKey!="" && searchForKey(data,*searchKey){
fmt.Println(path + itemString)
}
if *searchValue!="" && searchForValue(data,*searchValue){
fmt.Println(path + itemString)
}
}
}
}
func read(client *api.Client, path string) map[string]interface{} {
value, err := client.Logical().Read(path)
if err != nil {
fmt.Println(err)
}
values := value.Data
return values
}
func searchForValue(mapp map[string]interface{}, searchValue string) bool {
for _, value := range mapp {
if searchValue == value {
return true
}
}
return false
}
func searchForKey(mapp map[string]interface{}, searchKey string) bool {
for key := range mapp {
if searchKey == key {
return true
}
}
return false
}

You can LIST "directories" in Vault (I'm assuming you're just looking at the kv engine). So treat it somewhat like a regular file-system: start at the root, list the entries, check the contents of each of them for that value, then iterate through each entry, listing its contents, and so forth.
https://www.vaultproject.io/api-docs/secret/kv/kv-v1#list-secrets

Related

Golang for each filtering into a new var

Im working with for each loop and var of information and filtering it by A) regex.matchString B)Timecomparrison. The filtering works well and I have the data I need but currently I'm outputting it to screen via fmt.Println in part of the loop. My goal is to take that data and build another var with the now filtered list. I guess I need to make a new variable and add to it? But how do I return that and something I can use later?
Any assistance is appreciated.
for _, thing := range things {
if thing.element1 != nil {
matched, err := regexp.MatchString(z, element1)
if err != nil {
fmt.Println(err)
}
if matched {
if timecomparrison(element2, a) {
// this is a section that needs to be added new var and returned as a var
fmt.Println("****")
fmt.Println("element1:", element1)
fmt.Println("element2:", element2)
}
}
}
}
}
I think you need something like this.
type Thing struct {
element1 string
element2 string
}
func filter() []Thing {
things := []Thing{
{element1: "element1", element2: "element2"},
}
var result []Thing
regex := "/{}d/"
date := time.Now
for _, thing := range things {
if thing.element1 != nil {
matched, err := regexp.MatchString(regex, thing.element1)
if err != nil {
fmt.Println(err)
}
if matched {
if timeComparison(thing.element2, date) {
// this is a section that needs to be added new var and returned as a var
fmt.Println("****")
fmt.Println("element1:", thing.element1)
fmt.Println("element2:", thing.element2)
result = append(result, thing)
}
}
}
}
return result
}
I cleaned the code, added a type and some data, fixed some issues and renamed some things, but you should get the idea :)

Unmarshal hcl to struct using viper

Trying to Unmarshal a hcl config file to a struct, using viper, this error is returned: 1 error(s) decoding:\n\n* 'NATS' expected a map, got 'slice'. What is missing?
The code:
func lab() {
var c conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Error(err)
return
}
log.Info(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Error(err)
return
}
log.Infow("got conf", "conf", c)
}
type conf struct {
NATS struct {
HTTPPort int
Port int
Username string
Password string
}
}
And the config file (draft.hcl inside current directory):
NATS {
HTTPPort = 10044
Port = 10041
Username = "cl1"
Password = "__Psw__4433__"
}
Edit
Have checked this struct with hcl package and it gets marshaled/unmarshalled correctly. Also this works correctly with yaml and viper.
There is a difference between these two where log.Info(viper.Get("NATS")) is called. While the hcl version returns a slice of maps, the yaml version returns a map: map[password:__psw__4433__ httpport:10044 port:10041 username:cl1].
Your conf struct is not matching the HCL. When converted to json the HCL looks like below
{
"NATS": [
{
"HTTPPort": 10044,
"Password": "__Psw__4433__",
"Port": 10041,
"Username": "cl1"
}
]
}
So the Conf Struct should look like this
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
Modified code
package main
import (
"log"
"github.com/spf13/viper"
"fmt"
)
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
func main() {
var c Conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Fatal(err)
}
fmt.Println(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Fatal(err)
}
fmt.Println(c.NATS[0].Username)
}
I know this question is more than two years old now, but I came across the same issue recently.
I'm using viper to be able to load different configuration files into a Go struct, allowing configuration in JSON, YAML, TOML, HCL, just pick your favourite :)
HCL file format does wrap a map into a slice because it allows redefining a section like:
section = {
key1 = "value"
}
section = {
key2 = "value"
}
which is something that is not supported by the other formats.
And here's how I fixed it:
My solution implies each new block will override any previous definition of the same key, and keep all the others. You can do some merging magic but I didn't need to.
You need to make a hook to convert a slice of maps into a map:
// sliceOfMapsToMapHookFunc merges a slice of maps to a map
func sliceOfMapsToMapHookFunc() mapstructure.DecodeHookFunc {
return func(from reflect.Type, to reflect.Type, data interface{}) (interface{}, error) {
if from.Kind() == reflect.Slice && from.Elem().Kind() == reflect.Map && (to.Kind() == reflect.Struct || to.Kind() == reflect.Map) {
source, ok := data.([]map[string]interface{})
if !ok {
return data, nil
}
if len(source) == 0 {
return data, nil
}
if len(source) == 1 {
return source[0], nil
}
// flatten the slice into one map
convert := make(map[string]interface{})
for _, mapItem := range source {
for key, value := range mapItem {
convert[key] = value
}
}
return convert, nil
}
return data, nil
}
}
then you need to create a DecodeHook:
configOption := viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
sliceOfMapsToMapHookFunc(),
mapstructure.StringToTimeDurationHookFunc(),
mapstructure.StringToSliceHookFunc(","),
))
the two other hooks are the default ones so you might want to keep them
then you pass the option to the Unmarshal method
viper.Unmarshal(&c, configOption)
With this method you don't need a slice around your structs or your maps. Also that makes it compatible with the other configuration file formats

What's the idiomatic way of parsing dynamic YAML in Go?

I have some code for handling a YAML config file that's getting a little out-of-control w/ type assertions and I feel like there must be a better way to do this.
Here's the relevant snippet from my config file:
plugins:
taxii20:
default: default
api_roots:
default:
auth:
- ldap
- mutualtls
collections:
all:
selector: g.V().Save("<type>").Save("<created>").All()
selector_query_lang: gizmo
And here's my parsing code:
func parseTaxiiConfig() {
plg.ConfigMutex.Lock()
taxiiConfig := plg.ConfigData.Plugins["taxii20"].(map[interface{}]interface{})
ConfigData = &Config{}
if taxiiConfig["default"] != nil {
ConfigData.DefaultRoot = taxiiConfig["default"].(string)
}
if taxiiConfig["api_roots"] != nil {
ConfigData.APIRoots = make([]model.APIRoot, 0)
iroots := taxiiConfig["api_roots"].(map[interface{}]interface{})
for iname, iroot := range iroots {
root := model.APIRoot{Name: iname.(string)}
authMethods := iroot.(map[interface{}]interface{})["auth"].([]interface{})
root.AuthMethods = make([]string, 0)
for _, method := range authMethods {
root.AuthMethods = append(root.AuthMethods, method.(string))
}
collections := iroot.(map[interface{}]interface{})["collections"].(map[interface{}]interface{})
root.Collections = make([]model.Collection, 0)
for icolName, icollection := range collections {
collection := model.Collection{Name: icolName.(string)}
collection.Selector = icollection.(map[interface{}]interface{})["selector"].(string)
collection.SelectorQueryLang = icollection.(map[interface{}]interface{})["selector_query_lang"].(string)
root.Collections = append(root.Collections, collection)
}
ConfigData.APIRoots = append(ConfigData.APIRoots, root)
}
}
plg.ConfigMutex.Unlock()
// debug
fmt.Println(ConfigData)
}
The code works as intended, but there's just so many type assertions here and I can't shake the feeling that I'm missing a better way.
One possible critical item of note, as the config implies, this is configuration for a Caddy-style plugin system, so the main config parser cannot know ahead of time what the shape of the plugin config will look like. It has to delegate processing of the plugin's portion of the config file to the plugin itself.
Here's what I came up with instead. Much more readable.
// Config represents TAXII 2.0 plugin structure
type Config struct {
DefaultRoot string
APIRoots []model.APIRoot
}
// Intermediate config for mapstructure
type configRaw struct {
DefaultRoot string `mapstructure:"default"`
APIRoots map[string]apiRootRaw `mapstructure:"api_roots"`
}
type apiRootRaw struct {
AuthMethods []string `mapstructure:"auth"`
Collections map[string]collectionRaw `mapstructure:"collections"`
}
type collectionRaw struct {
Selector string `mapstructure:"selector"`
SelectorQueryLang string `mapstructure:"selector_query_lang"`
}
func parseTaxiiConfig() error {
plg.ConfigMutex.Lock()
defer plg.ConfigMutex.Unlock()
taxiiConfig := plg.ConfigData.Plugins["taxii20"].(map[interface{}]interface{})
fmt.Println(taxiiConfig)
ConfigData = &Config{}
raw := &configRaw{}
err := mapstructure.Decode(taxiiConfig, raw)
if err != nil {
return err
}
ConfigData.DefaultRoot = raw.DefaultRoot
ConfigData.APIRoots = make([]model.APIRoot, 0)
for name, root := range raw.APIRoots {
apiRoot := model.APIRoot{Name: name}
apiRoot.AuthMethods = root.AuthMethods
apiRoot.Collections = make([]model.Collection, 0)
for colName, col := range root.Collections {
collection := model.Collection{Name: colName}
collection.Selector = col.Selector
collection.SelectorQueryLang = col.SelectorQueryLang
apiRoot.Collections = append(apiRoot.Collections, collection)
}
ConfigData.APIRoots = append(ConfigData.APIRoots, apiRoot)
}
return nil
}

Parsing plist xml

How to parse xml in such silly format:
<key>KEY1</key><string>VALUE OF KEY1</string>
<key>KEY2</key><string>VALUE OF KEY2</string>
<key>KEY3</key><integer>42</integer>
<key>KEY3</key><array>
<integer>1</integer>
<integer>2</integer>
</array>
Parsing would be very simple if all values would have same type - for example strings. But in my case each value could be string, data, integer, boolean, array or dict.
This xml looks nearly like json, but unfortunately format is fixed, and I cannot change it. And I would prefer solution without any external packages.
Use a lower-level parsing interface provided by encoding/xml which allows you to iterate over individual tokens in the XML stream (such as "start element", "end element" etc).
See the Token() method of the encoding/xml's Decoder type.
Since the data is not well structured, and you can't modify the format, you can't use xml.Unmarshal, so you can process the XML elements by creating a new Decoder, then iterate over the tokens and use DecodeElement to process them one by one. In my sample code below, it puts everything in a map. The code is also on github here...
package main
import (
"encoding/xml"
"strings"
"fmt"
)
type PlistArray struct {
Integer []int `xml:"integer"`
}
const in = "<key>KEY1</key><string>VALUE OF KEY1</string><key>KEY2</key><string>VALUE OF KEY2</string><key>KEY3</key><integer>42</integer><key>KEY3</key><array><integer>1</integer><integer>2</integer></array>"
func main() {
result := map[string]interface{}{}
dec := xml.NewDecoder(strings.NewReader(in))
dec.Strict = false
var workingKey string
for {
token, _ := dec.Token()
if token == nil {
break
}
switch start := token.(type) {
case xml.StartElement:
fmt.Printf("startElement = %+v\n", start)
switch start.Name.Local {
case "key":
var k string
err := dec.DecodeElement(&k, &start)
if err != nil {
fmt.Println(err.Error())
}
workingKey = k
case "string":
var s string
err := dec.DecodeElement(&s, &start)
if err != nil {
fmt.Println(err.Error())
}
result[workingKey] = s
workingKey = ""
case "integer":
var i int
err := dec.DecodeElement(&i, &start)
if err != nil {
fmt.Println(err.Error())
}
result[workingKey] = i
workingKey = ""
case "array":
var ai PlistArray
err := dec.DecodeElement(&ai, &start)
if err != nil {
fmt.Println(err.Error())
}
result[workingKey] = ai
workingKey = ""
default:
fmt.Errorf("Unrecognized token")
}
}
}
fmt.Printf("%+v", result)
}

How can I make this object mapping more dry and reusable in Go?

I have created an object mapping in Go that is not relational, it is very simple.
I have several structs that looks like this:
type Message struct {
Id int64
Message string
ReplyTo sql.NullInt64 `db:"reply_to"`
FromId int64 `db:"from_id"`
ToId int64 `db:"to_id"`
IsActive bool `db:"is_active"`
SentTime int64 `db:"sent_time"`
IsViewed bool `db:"is_viewed"`
Method string `db:"-"`
AppendTo int64 `db:"-"`
}
To create a new message I just run this function:
func New() *Message {
return &Message{
IsActive: true,
SentTime: time.Now().Unix(),
Method: "new",
}
}
And then I have a message_crud.go file for this struct that looks like this:
To find a message by a unique column (for example by id) I run this function:
func ByUnique(column string, value interface{}) (*Message, error) {
query := fmt.Sprintf(`
SELECT *
FROM message
WHERE %s = ?
LIMIT 1;
`, column)
message := &Message{}
err := sql.DB.QueryRowx(query, value).StructScan(message)
if err != nil {
return nil, err
}
return message, nil
}
And to save a message (insert or update in the database) I run this method:
func (this *Message) save() error {
s := ""
if this.Id == 0 {
s = "INSERT INTO message SET %s;"
} else {
s = "UPDATE message SET %s WHERE id=:id;"
}
query := fmt.Sprintf(s, sql.PlaceholderPairs(this))
nstmt, err := sql.DB.PrepareNamed(query)
if err != nil {
return err
}
res, err := nstmt.Exec(*this)
if err != nil {
return err
}
if this.Id == 0 {
lastId, err := res.LastInsertId()
if err != nil {
return err
}
this.Id = lastId
}
return nil
}
The sql.PlaceholderPairs() function looks like this:
func PlaceholderPairs(i interface{}) string {
s := ""
val := reflect.ValueOf(i).Elem()
count := val.NumField()
for i := 0; i < count; i++ {
typeField := val.Type().Field(i)
tag := typeField.Tag
fname := strings.ToLower(typeField.Name)
if fname == "id" {
continue
}
if t := tag.Get("db"); t == "-" {
continue
} else if t != "" {
s += t + "=:" + t
} else {
s += fname + "=:" + fname
}
s += ", "
}
s = s[:len(s)-2]
return s
}
But every time I create a new struct, for example a User struct I have to copy paste the "crud section" above and create a user_crud.go file and replace the words "Message" with "User", and the words "message" with "user". I repeat alot of code and it is not very dry. Is there something I could do to not repeat this code for things I would reuse? I always have a save() method, and always have a function ByUnique() where I can return a struct and search by a unique column.
In PHP this was easy because PHP is not statically typed.
Is this possible to do in Go?
Your ByUnique is almost generic already. Just pull out the piece that varies (the table and destination):
func ByUnique(table string, column string, value interface{}, dest interface{}) error {
query := fmt.Sprintf(`
SELECT *
FROM %s
WHERE %s = ?
LIMIT 1;
`, table, column)
return sql.DB.QueryRowx(query, value).StructScan(dest)
}
func ByUniqueMessage(column string, value interface{}) (*Message, error) {
message := &Message{}
if err := ByUnique("message", column, value, &message); err != nil {
return nil, err
}
return message, error
}
Your save is very similar. You just need to make a generic save function along the lines of:
func Save(table string, identifier int64, source interface{}) { ... }
Then inside of (*Message)save, you'd just call the general Save() function. Looks pretty straightforward.
Side notes: do not use this as the name of the object inside a method. See the link from #OneOfOne for more on that. And do not get obsessed about DRY. It is not a goal in itself. Go focuses on code being simple, clear, and reliable. Do not create something complicated and fragile just to avoid typing a simple line of error handling. This doesn't mean that you shouldn't extract duplicated code. It just means that in Go it is usually better to repeat simple code a little bit rather than create complicated code to avoid it.
EDIT: If you want to implement Save using an interface, that's no problem. Just create an Identifier interface.
type Ider interface {
Id() int64
SetId(newId int64)
}
func (msg *Message) Id() int64 {
return msg.Id
}
func (msg *Message) SetId(newId int64) {
msg.Id = newId
}
func Save(table string, source Ider) error {
s := ""
if source.Id() == 0 {
s = fmt.Sprintf("INSERT INTO %s SET %%s;", table)
} else {
s = fmt.Sprintf("UPDATE %s SET %%s WHERE id=:id;", table)
}
query := fmt.Sprintf(s, sql.PlaceholderPairs(source))
nstmt, err := sql.DB.PrepareNamed(query)
if err != nil {
return err
}
res, err := nstmt.Exec(source)
if err != nil {
return err
}
if source.Id() == 0 {
lastId, err := res.LastInsertId()
if err != nil {
return err
}
source.SetId(lastId)
}
return nil
}
func (msg *Message) save() error {
return Save("message", msg)
}
The one piece that might blow up with this is the call to Exec. I don't know what package you're using, and it's possible that Exec won't work correctly if you pass it an interface rather than the actual struct, but it probably will work. That said, I'd probably just pass the identifier rather than adding this overhead.
You probably want to use an ORM.
They eliminate a lot of the boilerplate code you're describing.
See this question for "What is an ORM?"
Here is a list of ORMs for go: https://github.com/avelino/awesome-go#orm
I have never used one myself, so I can't recommend any. The main reason is that an ORM takes a lot of control from the developer and introduces a non-negligible performance overhead. You need to see for yourself if they fit your use-case and/or if you are comfortable with the "magic" that's going on in those libraries.
I don't recommend doing this, i personally would prefer being explicit about scanning into structs and creating queries.
But if you really want to stick to reflection you could do:
func ByUnique(obj interface{}, column string, value interface{}) error {
// ...
return sql.DB.QueryRowx(query, value).StructScan(obj)
}
// Call with
message := &Message{}
ByUnique(message, ...)
And for your save:
type Identifiable interface {
Id() int64
}
// Implement Identifiable for message, etc.
func Save(obj Identifiable) error {
// ...
}
// Call with
Save(message)
The approach i use and would recommend to you:
type Redirect struct {
ID string
URL string
CreatedAt time.Time
}
func FindByID(db *sql.DB, id string) (*Redirect, error) {
var redirect Redirect
err := db.QueryRow(
`SELECT "id", "url", "created_at" FROM "redirect" WHERE "id" = $1`, id).
Scan(&redirect.ID, &redirect.URL, &redirect.CreatedAt)
switch {
case err == sql.ErrNoRows:
return nil, nil
case err != nil:
return nil, err
}
return &redirect, nil
}
func Save(db *sql.DB, redirect *Redirect) error {
redirect.CreatedAt = time.Now()
_, err := db.Exec(
`INSERT INTO "redirect" ("id", "url", "created_at") VALUES ($1, $2, $3)`,
redirect.ID, redirect.URL, redirect.CreatedAt)
return err
}
This has the advantage of using the type system and mapping only things it should actually map.

Resources