I have two 16-byte hex values that relate to each other that I want to keep in memory in Go (so they only need to exist for the lifetime of the running process) that can be represented as a simple map like so:
{"aabbcc": "112233"}
Obviously I could represent these as a struct of two strings, but I'm just wondering if there's a faster (i.e. performance) or more memory-efficient way to store the strings in Go? I've only delved into Go lightly so far, so don't know the standard library well enough to know a good answer.
Edit: For an idea of what I'm getting at (in pseudo-code):
I've got two UUIDs from different sources, that are generated/received as strings:
uuid_a_1 = "aabb-1122-3344"
uuid_a_2 = "ddee-5566-7788"
I want to store these in relation to each other:
uuid_map[] = {uuid_a_1: uuid_a_2}
So that I can return one when I lookup the other:
return uuid_map[uuid_a_1]
I'm just curious if there's a more efficient way to store these in memory than a simple map of strings, as I may for example want to store several thousand during the lifetime of the process, and want to be able to key/value store these as quickly as possible (the idea being that because I know the exact size and type of the keys and values that I can do it fast).
As FUZxxl is suggesting you can encode the string to a byte array.
http://play.golang.org/p/7MYWTWSu2-
package main
import (
"encoding/hex"
"fmt"
"strings"
)
func main() {
b, err := hex.DecodeString(strings.Replace("df31a780-f640-11e3-a3ac-0800200c9a66", "-", "", -1))
if err != nil {
fmt.Println(err)
}
fmt.Println("Decoded:", b)
fmt.Println("Encoded:", hex.EncodeToString(b))
}
Related
Coming from Nodejs, I could do something like:
// given an array `list` of objects with a field `fruit`:
fruits = list.map(el => el.fruit) # which will return an array of fruit strings
Any way to do that in an elegant one liner in golang?
I know I can do it with a range loop, but I am looking for the possibility of a one liner solution
In Go, arrays are inflexible (because their length is encoded in their type) and costly to pass to functions (because a function operates on copies of its array arguments). I'm assuming you'd like to operate on slices rather than on arrays.
Because methods cannot take additional type arguments, you cannot simply declare a generic Map method in Go. However, you can define Map as a generic top-level function:
func Map[T, U any](ts []T, f func(T) U) []U {
us := make([]U, len(ts))
for i := range ts {
us[i] = f(ts[i])
}
return us
}
Then you can write the following code,
names := []string{"Alice", "Bob", "Carol"}
fmt.Println(Map(names, utf8.RuneCountInString))
which prints [5 3 5] to stdout (try it out in this Playground).
Go 1.18 saw the addition of a golang.org/x/exp/slices package, which provides many convenient operations on slices, but a Map function is noticeably absent from it. The omission of that function was the result of a long discussion in the GitHub issue dedicated to the golang.org/x/exp/slices proposal; concerns included the following:
hidden cost (O(n)) of operations behind a one-liner
uncertainty about error handling inside Map
risk of encouraging a style that strays too far from Go's traditional style
Russ Cox ultimately elected to drop Map from the proposal because it's
probably better as part of a more comprehensive streams API somewhere else.
Can I delete the first element in map? It is possible with slices slice = append(slice, slice[1:]...), but can I do something like this with maps?
Maps being hashtables don't have a specified order, so there's no way to delete keys in a defined order, unless you track keys in a separate slice, in the order you're adding them, something like:
type orderedMap struct {
data map[string]int
keys []string
mu *sync.RWMutex
}
func (o *orderedMap) Shift() (int, error) {
o.mu.Lock()
defer o.mu.Unlock()
if len(o.keys) == 0 {
return 0, ErrMapEmpty
}
i := o.data[o.keys[0]]
delete(o.data, o.keys[0])
o.keys = o.keys[1:]
return i, nil
}
Just to be unequivocal about why you can't really delete the "first" element from a map, let me reference the spec:
A map is an unordered group of elements of one type, called the element type, indexed by a set of unique keys of another type, called the key type. The value of an uninitialized map is nil.
Added the emphasis on the fact that map items are unordered
Using a slice to preserve some notion of the order of keys is, fundamentally, flawed, though. Given operations like this:
foo := map[string]int{
"foo": 1,
"bar": 2,
}
// a bit later:
foo["foo"] = 3
Is the index/key foo now updated, or reassigned? Should it be treated as a new entry, appended to the slice if keys, or is it an in-place update? Things get muddled really quickly. The simple fact of the matter is that the map type doesn't contain an "order" of things, trying to make it have an order quickly devolves in a labour intensive task where you'll end up writing your own type.
As I said earlier: it's a hashtable. Elements within get reshuffled behind the scenes if the hashing algorithm used for the keys produces collisions, for example. This question has the feel of an X-Y problem: why do you need the values in the map to be ordered? Maybe a map simply isn't the right approach for your particular problem.
This question already has answers here:
Why are map values not addressable?
(2 answers)
Closed 4 years ago.
type S struct {
e int
}
func main() {
a := []S{{1}}
a[0].e = 2
b := map[int]S{0: {1}}
b[0].e = 2 // error
}
a[0] is addressable but b[0] is not.
I know first 0 is an index and second 0 is a key.
Why golang implement like this? Any further consideration?
I've read source code of map in github.com/golang/go/src/runtime and map structure already supported indirectkey and indirectvalue if maxKeySize and maxValueSize are little enough.
type maptype struct {
...
keysize uint8 // size of key slot
indirectkey bool // store ptr to key instead of key itself
valuesize uint8 // size of value slot
indirectvalue bool // store ptr to value instead of value itself
...
}
I think if golang designers want this syntax, it works easy now.
Of course indirectkey indirectvalue may cost more resource and GC also need do more work.
So performance is the only reason for supporting this?
Or any other consideration?
In my opinion, supporting syntax like this is valuable.
As far as I known,
That's because a[0] can be replaced with address of array.
Similarly, a[1] can be replace with a[0]+(keySize*1).
But, In case of map one cannot do like that, hash algorithm changes from time to time based on your key, value pairs and number of them.
They are also rearranged from time to time.
specific computation is needed in-order to get the address of value.
Arrays or slices are easily addressable, but in case of maps it's like multiple function calls or structure look-ups ...
If one is thinking to replace it with what ever computation is needed, then binary size is going to be increased in orders of magnitude, and more over hash algorithm can keep changing from time to time.
Given a hashmap in Golang which has a key and a value, what is the simplest way of retrieving the key given the value?
For example Ruby equivalent would be
key = hashMap.key(value)
There is no built-in function to do this; you will have to make your own. Below is an example function that will work for map[string]int, which you can adapt for other map types:
func mapkey(m map[string]int, value int) (key string, ok bool) {
for k, v := range m {
if v == value {
key = k
ok = true
return
}
}
return
}
Usage:
key, ok := mapkey(hashMap, value)
if !ok {
panic("value does not exist in map")
}
The important question is: How many times will you have to look up the value?
If you only need to do it once, then you can iterate over the key, value pairs and keep the key (or keys) that match the value.
If you have to do the look up often, then I would suggest you make another map that has key, values reversed (assuming all keys map to unique values), and use that for look up.
I am in the midst of working on a server based on bitcoin and there is a list of constants and byte codes for the payment scripts. In the C++ version it has both identifiers with the codes and then another function that returns the string version. So it's really not much extra work to just take the original, with opcodes as string keys and the byte as value, and then reverse the order. The only thing that niggles me is duplicate keys on values. But since those are just true and false, overlapping zero and one, all of the first index of the string slice are the numbers and opcodes, and the truth values are the second index.
To iterate the list every time to identify the script command to execute would cost on average 50% of the map elements being tested. It's much simpler to just have a reverse lookup table. Executing the scripts has to be done maybe up to as much as 10,000 times on a full block so it makes no sense to save memory and pay instead in processing.
I am following the go tour and something bothered me.
Maps must be created with make (not new) before use
Fair enough:
map = make(map[int]Cats)
However the very next slide shows something different:
var m = map[string]Vertex{
"Bell Labs": Vertex{
40.68433, -74.39967,
},
"Google": Vertex{
37.42202, -122.08408,
},
}
This slide shows how you can ignore make when creating maps
Why did the tour say maps have to be created with make before they can be used? Am I missing something here?
Actually the only reason to use make to create a map is to preallocate a specific number of values, just like with slices (except you can't set a cap on a map)
m := map[int]Cats{}
s := []Cats{}
//is the same as
m := make(map[int]Cats)
s := make([]Cats, 0, 0)
However if you know you will have a minimum of X amount of items in a map you can do something like:
m := make(map[int]Cats, 100)// this will speed things up initially
Also check http://dave.cheney.net/2014/08/17/go-has-both-make-and-new-functions-what-gives
So they're actually right that you always need to use make before using a map. The reason it looks like they aren't in the example you gave is that the make call happens implicitly. So, for example, the following two are equivalent:
m := make(map[int]string)
m[0] = "zero"
m[1] = "one"
// Equivalent to:
m := map[int]string{
0: "zero",
1: "one",
}
Make vs New
Now, the reason to use make vs new is slightly more subtle. The reason is that new only allocates space for a variable of the given type, whereas make actually initializes it.
To give you a sense of this distinction, imagine we had a binary tree type like this:
type Tree struct {
root *node
}
type node struct {
val int
left, right *node
}
Now you can imagine that if we had a Tree which was allocated and initialized and had some values in it, and we made a copy of that Tree value, the two values would point to the same underlying data since they'd both have the same value for root.
So what would happen if we just created a new Tree without initializing it? Something like t := new(Tree) or var t Tree? Well, t.root would be nil, so if we made a copy of t, both variables would not point to the same underlying data, and so if we added some elements to the Tree, we'd end up with two totally separate Trees.
The same is true of maps and slices (and some others) in Go. When you make a copy of a slice variable or a map variable, both the old and the new variables refer to the same underlying data, just like an array in Java or C. Thus, if you just use new, and then make a copy and initialize the underlying data later, you'll have two totally separate data structures, which is usually not what you want.