This question already has answers here:
How to produce JSON with sorted keys in Go?
(2 answers)
Closed 3 years ago.
I am writing code that will check if data changed based on a comparison of json.Marshaled hashes of maps. I've created small code to produce what I am doing in abstracted way (available also in playground)
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
)
func main() {
fmt.Println("Hello, playground")
a := make(map[string]string)
a["a"] = "a1"
a["b"] = "b2"
sa, _ := json.Marshal(a)
ha := GenerateSHA256Hash(string(sa))
b := make(map[string]string)
b["a"] = "a1"
b["b"] = "b2"
sb, _ := json.Marshal(b)
hb := GenerateSHA256Hash(string(sb))
fmt.Println(ha)
fmt.Println(hb)
fmt.Println(ha == hb)
}
func GenerateSHA256Hash(s string) string {
hasher := sha256.New()
hasher.Write([]byte(s))
return hex.EncodeToString(hasher.Sum(nil))
}
But I recall that order of maps are unordered and in Golang spec it's written that
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next. If a map entry that has not yet been reached is removed during iteration, the corresponding iteration value will not be produced. If a map entry is created during iteration, that entry may be produced during the iteration or may be skipped. The choice may vary for each entry created and from one iteration to the next. If the map is nil, the number of iterations is 0.
So, in the code above I am building map, in the same way, each time and not accessing it concurrently during json.Marshalling.
Question: Will the hashes, produced in such manner, be always equal? Or will this approach be stable?
Go spec in this case is irrelevant since it's a details of the Go standard library (the encoding/json module)
As of this very moment it's implemented as
// Extract and sort the keys.
keys := v.MapKeys()
sv := make([]reflectWithString, len(keys))
for i, v := range keys {
sv[i].v = v
if err := sv[i].resolve(); err != nil {
e.error(fmt.Errorf("json: encoding error for type %q: %q", v.Type().String(), err.Error()))
}
}
sort.Slice(sv, func(i, j int) bool { return sv[i].s < sv[j].s })
Additionally, given the encoding/json documentation says
The map keys are sorted and used as JSON object keys by applying the following rules, subject to the UTF-8 coercion described for string values above:
it's safe to expect the same hash until at least Go 2.
Related
This question already has answers here:
Assign a new value to a struct field
(2 answers)
Closed 10 months ago.
The output of the following code surprises me:
package main
import (
"fmt"
)
type Thing struct {
mappings map[string]int
orderings []string
}
func NewThing() Thing {
t := Thing{}
t.mappings = make(map[string]int)
return t
}
func (t Thing) Add(s string) {
t.mappings[s] = 1
t.orderings = append(t.orderings, s)
}
func main() {
t := NewThing()
t.Add("foo")
if len(t.mappings) == len(t.orderings) {
fmt.Printf("Equal lengths: %v versus %v", t.mappings, t.orderings)
} else {
fmt.Printf("Unequal lengths: %v versus %v", t.mappings, t.orderings)
}
}
When run on the playground (https://play.golang.org/p/Ph67tHOt2Z_I) the output is this:
Unequal lengths: map[foo:1] versus []
I believe I'm treating the slice correctly; from my understanding it is initialized to nil in NewThing(), and is appended to in Add() (ensuring that the value returned from append is only assigned to its first argument).
Am I missing something incredibly obvious?
I looked at the following resources for an explanation:
https://gobyexample.com/slices - only uses either slice literals (i.e. not a struct field) or slices with set capacities, and I will not know the final size of t.orderings. It's my understanding that append should perform the extension and allocation automatically.
https://go.dev/blog/slices-intro - again, all demonstrations use slice literals. If the fields are moved out of the struct things work as expected. It's only once in the struct that this behavior occurs.
https://yourbasic.org/golang/gotcha-append/ - while it does describe behavior where append does not work as expected, the explanation involves append reusing memory when the slice has enough capacity for a new element, causing unexpected behavior when attempts to append the same array to two different copies. In my case, there is no reassignment of slice operations such as the one in this article, which is discouraged (some_var = append(some_other_var, elem)).
And I looked at the following questions for inspiration:
Go - append to slice in struct: the solution to this question was to assign the result of append back to the field, which I have done.
Correct way to initialize empty slice: the explanation is that slices don't have to be initialized, and can be left as nil and "appended to with allocation", so I believe I'm fine not initializing Thing.orderings.
Incase you don't want to use a pointer ,you can declare a global variable for Thing struct and assign it with the value of t from add function.Here is the code for the same logic :
package main
import (
"fmt"
)
var thing Thing
type Thing struct {
mappings map[string]int
orderings []string
}
func NewThing() Thing {
t := Thing{}
t.mappings = make(map[string]int)
return t
}
func (t Thing) Add(s string) {
t.mappings[s] = 1
t.orderings = append(t.orderings, s)
thing = t
}
func main() {
t := NewThing()
t.Add("foo")
if len(thing.mappings) == len(thing.orderings) {
fmt.Printf("Equal lengths: %v versus %v", thing.mappings, thing.orderings)
} else {
fmt.Printf("Unequal lengths: %v versus %v", thing.mappings, thing.orderings)
}
}
Output:
Equal lengths: map[foo:1] versus [foo]
package main
import (
"fmt"
"bufio"
"os"
"strconv"
)
func main() {
mp := make(map[int]string)//make a mapping
in := bufio.NewScanner(os.Stdin)
fmt.Println("Limit and Enter Strings")
in.Scan()
n := in.Text()
num, err := strconv.Atoi(n)
fmt.Println(err)
for i:=0; i<=num;i++ {
in.Scan()
mp[i] = in.Text()
}
fmt.Println(mp)
}
/* Output Limit and Enter Strings
5
<nil>
one
two
three
four
five
six
map[3:four 4:five 5:six 0:one 1:two 2:three]*/
The program is for making mapping from int to string. When I enter numbers in sequential format it prints a mapping in incorrect order.
Go provides a built-in map type that implements a hash table.
When iterating over a map with a range loop, the iteration order is
not specified and is not guaranteed to be the same from one iteration
to the next. Since Go 1 the runtime randomizes map iteration order, as
programmers relied on the stable iteration order of the previous
implementation. If you require a stable iteration order you must
maintain a separate data structure that specifies that order. This
example uses a separate sorted slice of keys to print a map[int]string
in key order:
import "sort"
var m map[int]string
var keys []int
for k := range m {
keys = append(keys, k)
}
sort.Ints(keys)
for _, k := range keys {
fmt.Println("Key:", k, "Value:", m[k])
}
Link: https://blog.golang.org/go-maps-in-action
Maps in Go language are not iterated in order (Check out https://blog.golang.org/go-maps-in-action for more on this). It's like a hash. If you need to order, I suggest you consider storing the keys in a slice (in addition to storing the key-value pair in the map) and then when you need to print in the order you entered, iterate over the slice to get the keys, and then retrieve the value for each of the keys from the hash.
So I would like to preserve the order of the post fields.
But now using the http ParseForm function it will put the fields into a map which will have a different order each time.
The original query : a=1&b=2&c=3 can become b=2&c=3&a=1 or any random order.
Since I hash the query and compare it with the hash of the user his query the hash on my side changes all the time since the order of the fields are random.
Code:
func parsePostQuery(r *http.Request, hashQuery string) bool {
urlquery := url.Values{}
r.ParseForm()
for k, p := range r.Form {
urlquery.Set(k, p[0])
}
//some psuedo code
if hashQuery == hash(urlquery.Encode()){
return true
}
return false
}
How can I parse the fields that are submitted by the user and keep the field order of the user?
Sidenote: I do not know the field names in advance.
In the same area as other answers, you will need the clients to calculate their hash by alphabetizing all parameters before hashing. The code you've supplied should work fine; values.Encode() will sort the values by key on it's own:
Encode encodes the values into “URL encoded” form ("bar=baz&foo=quux") sorted by key.
There's no way to preserve the ordering of the client; in fact, what you receive may not even be how it was ordered on the client end. However unlikely, there's no guarantee that intermediate processes won't change things.
tl;dr: You can't implicitly. The underlying data structure is a map, for which the order is not guaranteed. You need to take additional steps.
However Go maps in action shows an easy way to access the map in a sorted way. You create a slice of the keys, sort that slice and access the map value by iterating over the keys in the sorted slice.
For your example, it would look something like this
package main
import (
"crypto/md5"
"fmt"
"io"
"sort"
)
func main() {
// Which is the same structure as url.Values()
var m map[string][]string = make(map[string][]string)
m["c"] = []string{"19.95"}
m["b"] = []string{"foo", "bar", "baz"}
m["a"] = []string{"1"}
// Note that playground is deterministic, so the order should be preserved there
// However, you can not rely on that in the real world
fmt.Println("Unsorted")
for k, v := range m {
fmt.Println("Key:", k, "Value:", v)
}
var keys []string
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
hash := md5.New()
fmt.Println("\nSorted")
for _, k := range keys {
fmt.Println("Key:", k, "Value:", m[k])
// Add Key/Value pair to hash
fmt.Printf("\tAdding KEY '%s' to hash\n", k)
io.WriteString(hash, k)
for _, v := range m[k] {
fmt.Printf("\tAdding VALUE '%s' to hash\n", v)
io.WriteString(hash, v)
}
}
fmt.Printf("\nHash: %x", hash.Sum(nil))
}
Run above code on Playground
You can read the request body and check for the form parameters. They will appear in the same order as in the request(hope your client application is also aware of this order preserving)
You can create a reader for reading the request body. A sample code looks like
body, err := ioutil.ReadAll(r.Body)
if err != nil {
fmt.Println("Reading Error ")
return
}
fmt.Println("Req Body : ", string(body))
Note : Be aware of the content type
Is there a way to write a generic array/slice deduplication in go, for []int we can have something like (from http://rosettacode.org/wiki/Remove_duplicate_elements#Go ):
func uniq(list []int) []int {
unique_set := make(map[int] bool, len(list))
for _, x := range list {
unique_set[x] = true
}
result := make([]int, len(unique_set))
i := 0
for x := range unique_set {
result[i] = x
i++
}
return result
}
But is there a way to extend it to support any array? with a signature like:
func deduplicate(a []interface{}) []interface{}
I know that you can write that function with that signature, but then you can't actually use it on []int, you need to create a []interface{} put everything from the []int into it, pass it to the function then get it back and put it into a []interface{} and go through this new array and put everything in a new []int.
My question is, is there a better way to do this?
While VonC's answer probably does the closest to what you really want, the only real way to do it in native Go without gen is to define an interface
type IDList interface {
// Returns the id of the element at i
ID(i int) int
// Returns the element
// with the given id
GetByID(id int) interface{}
Len() int
// Adds the element to the list
Insert(interface{})
}
// Puts the deduplicated list in dst
func Deduplicate(dst, list IDList) {
intList := make([]int, list.Len())
for i := range intList {
intList[i] = list.ID(i)
}
uniques := uniq(intList)
for _,el := range uniques {
dst.Insert(list.GetByID(el))
}
}
Where uniq is the function from your OP.
This is just one possible example, and there are probably much better ones, but in general mapping each element to a unique "==able" ID and either constructing a new list or culling based on the deduplication of the IDs is probably the most intuitive way.
An alternate solution is to take in an []IDer where the IDer interface is just ID() int. However, that means that user code has to create the []IDer list and copy all the elements into that list, which is a bit ugly. It's cleaner for the user to wrap the list as an ID list rather than copy, but it's a similar amount of work either way.
The only way I have seen that implemented in Go is with the clipperhouse/gen project,
gen is an attempt to bring some generics-like functionality to Go, with some inspiration from C#’s Linq and JavaScript’s underscore libraries
See this test:
// Distinct returns a new Thing1s slice whose elements are unique. See: http://clipperhouse.github.io/gen/#Distinct
func (rcv Thing1s) Distinct() (result Thing1s) {
appended := make(map[Thing1]bool)
for _, v := range rcv {
if !appended[v] {
result = append(result, v)
appended[v] = true
}
}
return result
}
But, as explained in clipperhouse.github.io/gen/:
gen generates code for your types, at development time, using the command line.
gen is not an import; the generated source becomes part of your project and takes no external dependencies.
You could do something close to this via an interface. Define an interface, say "DeDupable" requiring a func, say, UniqId() []byte, which you could then use to do the removing of dups. and your uniq func would take a []DeDupable and work on it
This question already has answers here:
Copying all elements of a map into another
(4 answers)
Closed 4 months ago.
Is there a built in function in go for making copies of arbitrary maps?
I would be able to write one by hand but I found out earlier I was looking a similar question when I wanted to make a deep comparison of maps and there seemed to be a function already built in for that! So similarly, maybe I was wondering if there was an built in or some library or package for making deep copies of maps in golang. I am sure I am not the first person to want to make copies of maps in go.
By copy I mean you can create two different variables that reference a different map in memory even though they are the same content wise.
For a more general answer, you can encode your map and decode it in a new variable with encoding/gob.
The advantages of this way is that it'll even work on more complex data structure, like a slice of struct containing a slice of maps.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
func main() {
ori := map[string]int{
"key": 3,
"clef": 5,
}
var mod bytes.Buffer
enc := gob.NewEncoder(&mod)
dec := gob.NewDecoder(&mod)
fmt.Println("ori:", ori) // key:3 clef:5
err := enc.Encode(ori)
if err != nil {
log.Fatal("encode error:", err)
}
var cpy map[string]int
err = dec.Decode(&cpy)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Println("cpy:", cpy) // key:3 clef:5
cpy["key"] = 2
fmt.Println("cpy:", cpy) // key:2 clef:5
fmt.Println("ori:", ori) // key:3 clef:5
}
If you want to know more about gobs, there is a go blog post about it.
No, there is no built-in one-liner for map deep copy.
However, there is an iterative solution as well as a generic package for deep copy.
Copying a map is a few, trivial lines of code. Just write the code and enjoy letting go of the idea that every simple piece of code has to be in a library!
original := map[string]int{
"Hello": 4,
"World": 123,
}
copy := map[string]int{}
for k, v := range original {
copy[k] = v
}