I've encountered some strange behavior with reflect.DeepEqual. I have an object of type map[string][]string, with one key whose value is an empty slice. When I use gob to encode this object, and then decode it into another map, these two maps are not equal according to reflect.DeepEqual (even though the content is identical).
package main
import (
"fmt"
"bytes"
"encoding/gob"
"reflect"
)
func main() {
m0 := make(map[string][]string)
m0["apple"] = []string{}
// Encode m0 to bytes
var network bytes.Buffer
enc := gob.NewEncoder(&network)
enc.Encode(m0)
// Decode bytes into a new map m2
dec := gob.NewDecoder(&network)
m2 := make(map[string][]string)
dec.Decode(&m2)
fmt.Printf("%t\n", reflect.DeepEqual(m0, m2)) // false
fmt.Printf("m0: %+v != m2: %+v\n", m0, m2) // they look equal to me!
}
Output:
false
m0: map[apple:[]] != m2: map[apple:[]]
A couple notes from follow-up experiments:
If I make the value of m0["apple"] a nonempty slice, for example m0["apple"] = []string{"pear"}, then DeepEqual returns true.
If I keep the value as an empty slice but I construct the identical map from scratch rather than with gob, then DeepEqual returns true:
m1 := make(map[string][]string)
m1["apple"] = []string{}
fmt.Printf("%t\n", reflect.DeepEqual(m0, m1)) // true!
So it's not strictly an issue with how DeepEqual handles empty slices; it's some strange interaction between that and gob's serialization.
This is because you encode an empty slice, and during decoding the encoding/gob package only allocates a slice if the one provided (the target to decode into) is not big enough to accomodate the encoded values. This is documented at: gob: Types and Values:
In general, if allocation is required, the decoder will allocate memory. If not, it will update the destination variables with values read from the stream.
Since there are 0 elements encoded, and a nil slice is perfectly capable of accomodating 0 elements, no slice will be allocated. We can verify this if we print the result of comparing the slices to nil:
fmt.Println(m0["apple"] == nil, m2["apple"] == nil)
Output of the above is (try it on the Go Playground):
true false
Note that the fmt package prints nil slice values and empty slices the same way: as [], you cannot rely on its output to judge if a slices is nil or not.
And reflect.DeepEqual() treats a nil slice and an empty but non-nil slice different (non-deep equal):
Note that a non-nil empty slice and a nil slice (for example, []byte{} and []byte(nil)) are not deeply equal.
Related
I'm trying to do direct i/o on linux, so I need to create memory aligned buffers. I copied some code to do it, but I don't understand how it works:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
"yottaStore/yottaStore-go/src/yfs/test/utils"
)
const (
AlignSize = 4096
BlockSize = 4096
)
// Looks like dark magic
func Alignment(block []byte, AlignSize int) int {
return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096*2)
a := Alignment(file, AlignSize)
offset := 0
if a != 0 {
offset = AlignSize - a
}
file = file[offset : offset+BlockSize]
n, readErr := unix.Pread(fd, file, 0)
if readErr != nil {
panic(readErr)
}
fmt.Println(a, offset, offset+utils.BlockSize, len(file))
fmt.Println("Content is: ", string(file))
}
I understand that I'm generating a slice twice as big than what I need, and then extracting a memory aligned block from it, but the Alignment function doesn't make sense to me.
How does the Alignment function works?
If I try to fmt.Println the intermediate steps of that function I get different results, why? I guess because observing it changes its memory alignment (like in quantum physics :D)
Edit:
Example with fmt.println, where I don't need any more alignment:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
)
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096)
fmt.Println("Pointer: ", &file[0])
n, readErr := unix.Pread(fd, file, 0)
fmt.Println("Return is: ", n)
if readErr != nil {
panic(readErr)
}
fmt.Println("Content is: ", string(file))
}
Your AlignSize has a value of a power of 2. In binary representation it contains a 1 bit followed by full of zeros:
fmt.Printf("%b", AlignSize) // 1000000000000
A slice allocated by make() may have a memory address that is more or less random, consisting of ones and zeros following randomly in binary; or more precisely the starting address of its backing array.
Since you allocate twice the required size, that's a guarantee that the backing array will cover an address space that has an address in the middle somewhere that ends with as many zeros as the AlignSize's binary representation, and has BlockSize room in the array starting at this. We want to find this address.
This is what the Alignment() function does. It gets the starting address of the backing array with &block[0]. In Go there's no pointer arithmetic, so in order to do something like that, we have to convert the pointer to an integer (there is integer arithmetic of course). In order to do that, we have to convert the pointer to unsafe.Pointer: all pointers are convertible to this type, and unsafe.Pointer can be converted to uintptr (which is an unsigned integer large enough to store the uninterpreted bits of a pointer value), on which–being an integer–we can perform integer arithmetic.
We use bitwise AND with the value uintptr(AlignSize-1). Since AlignSize is a power of 2 (contains a single 1 bit followed by zeros), the number one less is a number whose binary representation is full of ones, as many as trailing zeros AlignSize has. See this example:
x := 0b1010101110101010101
fmt.Printf("AlignSize : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))
Output:
AlignSize : 1000000000000
AlignSize-1 : 111111111111
x : 1010101110101010101
result of & : 110101010101
So the result of & is the offset which if you subtract from AlignSize, you get an address that has as many trailing zeros as AlignSize itself: the result is "aligned" to the multiple of AlignSize.
So we will use the part of the file slice starting at offset, and we only need BlockSize:
file = file[offset : offset+BlockSize]
Edit:
Looking at your modified code trying to print the steps: I get an output like:
Pointer: 0xc0000b6000
Unsafe pointer: 0xc0000b6000
Unsafe pointer, uintptr: 824634466304
Unpersand: 0
Cast to int: 0
Return is: 0
Content is:
Note nothing is changed here. Simply the fmt package prints pointer values using hexadecimal representation, prefixed by 0x. uintptr values are printed as integers, using decimal representation. Those values are equal:
fmt.Println(0xc0000b6000, 824634466304) // output: 824634466304 824634466304
Also note the rest is 0 because in my case 0xc0000b6000 is already a multiple of 4096, in binary it is 1100000000000000000100001110000000000000.
Edit #2:
When you use fmt.Println() to debug parts of the calculation, that may change escape analysis and may change the allocation of the slice (from stack to heap). This depends on the used Go version too. Do not rely on your slice being allocated at an address that is (already) aligned to AlignSize.
See related questions for more details:
Mix print and fmt.Println and stack growing
why struct arrays comparing has different result
Addresses of slices of empty structs
This question already has answers here:
How to produce JSON with sorted keys in Go?
(2 answers)
Closed 3 years ago.
I am writing code that will check if data changed based on a comparison of json.Marshaled hashes of maps. I've created small code to produce what I am doing in abstracted way (available also in playground)
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
)
func main() {
fmt.Println("Hello, playground")
a := make(map[string]string)
a["a"] = "a1"
a["b"] = "b2"
sa, _ := json.Marshal(a)
ha := GenerateSHA256Hash(string(sa))
b := make(map[string]string)
b["a"] = "a1"
b["b"] = "b2"
sb, _ := json.Marshal(b)
hb := GenerateSHA256Hash(string(sb))
fmt.Println(ha)
fmt.Println(hb)
fmt.Println(ha == hb)
}
func GenerateSHA256Hash(s string) string {
hasher := sha256.New()
hasher.Write([]byte(s))
return hex.EncodeToString(hasher.Sum(nil))
}
But I recall that order of maps are unordered and in Golang spec it's written that
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next. If a map entry that has not yet been reached is removed during iteration, the corresponding iteration value will not be produced. If a map entry is created during iteration, that entry may be produced during the iteration or may be skipped. The choice may vary for each entry created and from one iteration to the next. If the map is nil, the number of iterations is 0.
So, in the code above I am building map, in the same way, each time and not accessing it concurrently during json.Marshalling.
Question: Will the hashes, produced in such manner, be always equal? Or will this approach be stable?
Go spec in this case is irrelevant since it's a details of the Go standard library (the encoding/json module)
As of this very moment it's implemented as
// Extract and sort the keys.
keys := v.MapKeys()
sv := make([]reflectWithString, len(keys))
for i, v := range keys {
sv[i].v = v
if err := sv[i].resolve(); err != nil {
e.error(fmt.Errorf("json: encoding error for type %q: %q", v.Type().String(), err.Error()))
}
}
sort.Slice(sv, func(i, j int) bool { return sv[i].s < sv[j].s })
Additionally, given the encoding/json documentation says
The map keys are sorted and used as JSON object keys by applying the following rules, subject to the UTF-8 coercion described for string values above:
it's safe to expect the same hash until at least Go 2.
I'm going through a Go tutorial and I reached the lesson about nil slices where it says:
A nil slice has a length and capacity of 0 and has no underlying array.
In order to show this they present this code which works
package main
import "fmt"
func main() {
var s []int
fmt.Println(s, len(s), cap(s))
if s == nil {
fmt.Println("nil!")
}
}
However, I tried to experiment and I replaced var s []int with s := []int{}. The console still prints [] 0 0 as in the first case but no longer the nil! string. So why is the first one nil and the other one not?
For s := []int{}:
Because it is initialized to a new type (e.g. struct) with an underlying array, a length, and a capacity
A slice, once initialized, is always associated with an underlying array that holds its elements.
For var s []int see Slice types:
The value of an uninitialized slice is nil.
The zero value:
When storage is allocated for a variable, either through a declaration or a call of new, or when a new value is created, either through a composite literal or a call of make, and no explicit initialization is provided, the variable or value is given a default value. Each element of such a variable or value is set to the zero value for its type: false for booleans, 0 for numeric types, "" for strings, and nil for pointers, functions, interfaces, slices, channels, and maps. This initialization is done recursively, so for instance each element of an array of structs will have its fields zeroed if no value is specified.
These two simple declarations are equivalent:
var i int
var i int = 0
After
type T struct { i int; f float64; next *T }
t := new(T)
the following holds:
t.i == 0
t.f == 0.0
t.next == nil
The same would also be true after
var t T
I hope this helps.
Code below will raise a runtime error when append reflect.Value of nil:
package main
import (
"fmt"
"reflect"
)
func main() {
var list []interface{}
v := reflect.ValueOf(list)
v = reflect.Append(v, reflect.ValueOf(1)) // [1]
v = reflect.Append(v, reflect.ValueOf("1")) // [1, 1]
v = reflect.Append(v, reflect.ValueOf(nil)) // runtime error
fmt.Println(v)
}
So
why there is a runtime error?
how can I use reflect.Append to add a nil to interface{} slice?
interface{} is an interface type, and they are "tricky". They are wrappers around a concrete value and the concrete type, schematically a (value, type) pair.
So when you pass a concrete value to a function that expects an interface{} value, the concrete value will be wrapped in an interface{} value automatically, implicitly. If you pass a nil to such a function, the interface value itself will be nil. If you pass a nil pointer to it, such as (*int)(nil), the interface value will not be nil but an interface value holding "(nil, *int)".
If you pass nil to reflect.ValueOf(), it results in a "zero" reflect.Value which represents no value at all. If you pass this to reflect.Append(), it will not have the type information, it will not know what you want to append to the slice.
It is possible to create a value that represents the nil interface value.
To do that, we may start from the type descriptor of a value of an interface pointer (pointers to interface rarely makes sense, but this is one of them). We navigate to the type descriptor of the pointed type, which is interface{}. We obtain a zero value of this type (using reflect.Zero()), which is nil (zero value of interface types is nil).
Zero returns a Value representing the zero value for the specified type. The result is different from the zero value of the Value struct, which represents no value at all.
So this is how it looks like:
typeOfEmptyIface := reflect.TypeOf((*interface{})(nil)).Elem()
valueOfZeroEmptyIface := reflect.Zero(typeOfEmptyIface)
v = reflect.Append(v, valueOfZeroEmptyIface)
Or as a single line:
v = reflect.Append(v, reflect.Zero(reflect.TypeOf((*interface{})(nil)).Elem()))
To check the results, let's use:
fmt.Printf("%#v\n", v)
And also let's type-assert back the slice, and add a nil value using the builtin append() function:
list = v.Interface().([]interface{})
list = append(list, nil)
fmt.Printf("%#v\n", list)
Let's do an explicit, extra check if the elements are nil (compare them to nil). Although using %#v verb this is redundant, %v likes to print non-nil interfaces holding nil concrete values just as nil (the same as if the interface value itself would be nil).
fmt.Println(list[2] == nil, list[3] == nil)
Ouptut will be (try it on the Go Playground):
[]interface {}{1, "1", interface {}(nil)}
[]interface {}{1, "1", interface {}(nil), interface {}(nil)}
true true
See related question: Hiding nil values, understanding why golang fails here
Also: The Go Blog: The Laws of Reflection
I would like to append to a slice using only reflection. But I can't figure out how to "replace" the value of a with the new slice.
func main() {
fmt.Println("Hello, playground")
a := []string {"a","b","c"}
values := []string {"d","e"}
v := reflect.ValueOf(a)
fmt.Printf("%t\n\n", v.Type())
fmt.Printf("%t\n\n", v.Type().Elem().Kind())
for _, val := range values {
v.Set(reflect.Append(v, reflect.ValueOf(val)))
}
fmt.Printf("%t - %v", a, a)
}
This code is available for fiddling at https://play.golang.org/p/cDlyH3jBDS.
You can't modify the value wrapped in reflect.Value if it originates from a non-pointer. If it would be allowed, you could only modify a copy and would cause more confusion. A slice value is a header containing a pointer to a backing array, a length and a capacity. When you pass a to reflect.ValueOf(), a copy of this header is made and passed, and any modification you could do on it could only modify this header-copy. Adding elements (and thus changing its length and potentially the pointer and capacity) would not be observed by the original slice header, the original would still point to the same array, and would still contain the same length and capacity values. For details see Are Golang function parameter passed as copy-on-write?; and Golang passing arrays to the function and modifying it.
You have to start from a pointer, and you may use Value.Elem() to obtain the reflect.Value descriptor of the pointed, dereferenced value. But you must start from a pointer.
Changing this single line in your code makes it work:
v := reflect.ValueOf(&a).Elem()
And also to print the type of a value, use the %T verb (%t is for bool values):
fmt.Printf("%T\n\n", v.Type())
fmt.Printf("%T\n\n", v.Type().Elem().Kind())
// ...
fmt.Printf("%T - %v", a, a)
Output (try it on the Go Playground):
Hello, playground
*reflect.rtype
reflect.Kind
[]string - [a b c d e]
For a deeper understanding of Go's reflection, read the blog post: The Laws of Reflection
And read related questions+answers:
Assigning a value to struct member through reflection in Go
Changing pointer type and value under interface with reflection
Using reflection SetString