So I would like to preserve the order of the post fields.
But now using the http ParseForm function it will put the fields into a map which will have a different order each time.
The original query : a=1&b=2&c=3 can become b=2&c=3&a=1 or any random order.
Since I hash the query and compare it with the hash of the user his query the hash on my side changes all the time since the order of the fields are random.
Code:
func parsePostQuery(r *http.Request, hashQuery string) bool {
urlquery := url.Values{}
r.ParseForm()
for k, p := range r.Form {
urlquery.Set(k, p[0])
}
//some psuedo code
if hashQuery == hash(urlquery.Encode()){
return true
}
return false
}
How can I parse the fields that are submitted by the user and keep the field order of the user?
Sidenote: I do not know the field names in advance.
In the same area as other answers, you will need the clients to calculate their hash by alphabetizing all parameters before hashing. The code you've supplied should work fine; values.Encode() will sort the values by key on it's own:
Encode encodes the values into “URL encoded” form ("bar=baz&foo=quux") sorted by key.
There's no way to preserve the ordering of the client; in fact, what you receive may not even be how it was ordered on the client end. However unlikely, there's no guarantee that intermediate processes won't change things.
tl;dr: You can't implicitly. The underlying data structure is a map, for which the order is not guaranteed. You need to take additional steps.
However Go maps in action shows an easy way to access the map in a sorted way. You create a slice of the keys, sort that slice and access the map value by iterating over the keys in the sorted slice.
For your example, it would look something like this
package main
import (
"crypto/md5"
"fmt"
"io"
"sort"
)
func main() {
// Which is the same structure as url.Values()
var m map[string][]string = make(map[string][]string)
m["c"] = []string{"19.95"}
m["b"] = []string{"foo", "bar", "baz"}
m["a"] = []string{"1"}
// Note that playground is deterministic, so the order should be preserved there
// However, you can not rely on that in the real world
fmt.Println("Unsorted")
for k, v := range m {
fmt.Println("Key:", k, "Value:", v)
}
var keys []string
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
hash := md5.New()
fmt.Println("\nSorted")
for _, k := range keys {
fmt.Println("Key:", k, "Value:", m[k])
// Add Key/Value pair to hash
fmt.Printf("\tAdding KEY '%s' to hash\n", k)
io.WriteString(hash, k)
for _, v := range m[k] {
fmt.Printf("\tAdding VALUE '%s' to hash\n", v)
io.WriteString(hash, v)
}
}
fmt.Printf("\nHash: %x", hash.Sum(nil))
}
Run above code on Playground
You can read the request body and check for the form parameters. They will appear in the same order as in the request(hope your client application is also aware of this order preserving)
You can create a reader for reading the request body. A sample code looks like
body, err := ioutil.ReadAll(r.Body)
if err != nil {
fmt.Println("Reading Error ")
return
}
fmt.Println("Req Body : ", string(body))
Note : Be aware of the content type
Related
I am using apache-arrow/go to read parquet data.
I can parse the data to table by using apach-arrow.
reader, err := ipc.NewReader(buf, ipc.WithAllocator(alloc))
if err != nil {
log.Println(err.Error())
return nil
}
defer reader.Release()
records := make([]array.Record, 0)
for reader.Next() {
rec := reader.Record()
rec.Retain()
defer rec.Release()
records = append(records, rec)
}
table := array.NewTableFromRecords(reader.Schema(), records)
Here, i can get the column info from table.Colunmn(index), such as:
for i, _ := range table.Schema().Fields() {
a := table.Column(i)
log.Println(a)
}
But the Column struct is defined as
type Column struct {
field arrow.Field
data *Chunked
}
and the println result is like
["WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN"]
However, this is not a string or slice. Is there anyway that i can get the data of each column with string type or []interface{} ?
Update:
I find that i can use reflect to get the element from col.
log.Println(col.(*array.Int64).Value(0))
But i am not sure if this is the recommended way to use it.
When working with Arrow data, there's a couple concepts to understand:
Array: Metadata + contiguous buffers of data
Record Batch: A schema + a collection of Arrays that are all the same length.
Chunked Array: A group of Arrays of varying lengths but all the same data type. This allows you to treat multiple Arrays as one single column of data without having to copy them all into a contiguous buffer.
Column: Is just a Field + a Chunked Array
Table: A collection of Columns allowing you to treat multiple non-contiguous arrays as a single large table without having to copy them all into contiguous buffers.
In your case, you're reading multiple record batches (groups of contiguous Arrays) and treating them as a single large table. There's a few different ways you can work with the data:
One way is to use a TableReader:
tr := array.NewTableReader(tbl, 5)
defer tr.Release()
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
// do something with the Array
}
}
Another way would be to interact with the columns directly as you were in your example:
for i := 0; i < table.NumCols(); i++ {
col := table.Column(i)
for _, chunk := range col.Data().Chunks() {
// do something with chunk (an arrow.Array)
}
}
Either way, you eventually have an arrow.Array to deal with, which is an interface containing one of the typed Array types. At this point you are going to have to switch on something, you could type switch on the type of the Array itself:
switch arr := col.(type) {
case *array.Int64:
// do stuff with arr
case *array.Int32:
// do stuff with arr
case *array.String:
// do stuff with arr
...
}
Alternately, you could type switch on the data type:
switch col.DataType().ID() {
case arrow.INT64:
// type assertion needed col.(*array.Int64)
case arrow.INT32:
// type assertion needed col.(*array.Int32)
...
}
For getting the data out of the array, primitive types which are stored contiguously tend to have a *Values method which will return a slice of the type. For example array.Int64 has Int64Values() which returns []int64. Otherwise, all of the types have .Value(int) methods which return the value at a particular index as you showed in your example.
Hope this helps!
Make sure you use v9
(import "github.com/apache/arrow/go/v9/arrow") because it have implemented json.Marshaller (from go-json)
Use "github.com/goccy/go-json" for Marshaler (because of this)
Then you can use TableReader to Marshal it then Unmarshal with type []any
In your example maybe look like this:
import (
"github.com/apache/arrow/go/v9/arrow"
"github.com/apache/arrow/go/v9/arrow/array"
"github.com/apache/arrow/go/v9/arrow/memory"
"github.com/goccy/go-json"
)
...
tr := array.NewTableReader(tabel, 6)
defer tr.Release()
// fmt.Printf("tbl.NumRows() = %+v\n", tbl.NumRows())
// fmt.Printf("tbl.NumColumn = %+v\n", tbl.NumCols())
// keySlice is for sorting same as data source
keySlice := make([]string, 0, tabel.NumCols())
res := make(map[string][]any, 0)
var key string
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
key = rec.ColumnName(i)
if res[key] == nil {
res[key] = make([]any, 0)
keySlice = append(keySlice, key)
}
var tmp []any
b2, err := json.Marshal(col)
if err != nil {
panic(err)
}
err = json.Unmarshal(b2, &tmp)
if err != nil {
panic(err)
}
// fmt.Printf("key = %s\n", key)
// fmt.Printf("tmp = %+v\n", tmp)
res[key] = append(res[key], tmp...)
}
}
fmt.Println("res", res)
This question already has answers here:
How to produce JSON with sorted keys in Go?
(2 answers)
Closed 3 years ago.
I am writing code that will check if data changed based on a comparison of json.Marshaled hashes of maps. I've created small code to produce what I am doing in abstracted way (available also in playground)
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
)
func main() {
fmt.Println("Hello, playground")
a := make(map[string]string)
a["a"] = "a1"
a["b"] = "b2"
sa, _ := json.Marshal(a)
ha := GenerateSHA256Hash(string(sa))
b := make(map[string]string)
b["a"] = "a1"
b["b"] = "b2"
sb, _ := json.Marshal(b)
hb := GenerateSHA256Hash(string(sb))
fmt.Println(ha)
fmt.Println(hb)
fmt.Println(ha == hb)
}
func GenerateSHA256Hash(s string) string {
hasher := sha256.New()
hasher.Write([]byte(s))
return hex.EncodeToString(hasher.Sum(nil))
}
But I recall that order of maps are unordered and in Golang spec it's written that
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next. If a map entry that has not yet been reached is removed during iteration, the corresponding iteration value will not be produced. If a map entry is created during iteration, that entry may be produced during the iteration or may be skipped. The choice may vary for each entry created and from one iteration to the next. If the map is nil, the number of iterations is 0.
So, in the code above I am building map, in the same way, each time and not accessing it concurrently during json.Marshalling.
Question: Will the hashes, produced in such manner, be always equal? Or will this approach be stable?
Go spec in this case is irrelevant since it's a details of the Go standard library (the encoding/json module)
As of this very moment it's implemented as
// Extract and sort the keys.
keys := v.MapKeys()
sv := make([]reflectWithString, len(keys))
for i, v := range keys {
sv[i].v = v
if err := sv[i].resolve(); err != nil {
e.error(fmt.Errorf("json: encoding error for type %q: %q", v.Type().String(), err.Error()))
}
}
sort.Slice(sv, func(i, j int) bool { return sv[i].s < sv[j].s })
Additionally, given the encoding/json documentation says
The map keys are sorted and used as JSON object keys by applying the following rules, subject to the UTF-8 coercion described for string values above:
it's safe to expect the same hash until at least Go 2.
How do I append output from a twitter search to the field Data in the SearchTwitterOutput{} struct.
Thanks!
I am using a twitter library to search twitter base on a query input. The search returns an array of strings(I believe), I am able to fmt.println the data but I need the data as a struct.
type SearchTwitterOutput struct {
Data string
}
func (SearchTwitter) execute(input SearchTwitterInput) (*SearchTwitterOutput, error) {
credentials := Credentials{
AccessToken: input.AccessToken,
AccessTokenSecret: input.AccessTokenSecret,
ConsumerKey: input.ConsumerKey,
ConsumerSecret: input.ConsumerSecret,
}
client, err := GetUserClient(&credentials)
if err != nil {
return nil, err
}
// search through the tweet and returns a
search, _ , err := client.Search.Tweets(&twitter.SearchTweetParams{
Query: input.Text,
})
if err != nil {
println("PANIC")
panic(err.Error())
return &SearchTwitterOutput{}, err
}
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
}
return &SearchTwitterOutput{
Data: "test", //data is a string for now it can be anything
}, nil
}
//Data field is a string type for now it can be anything
//I use "test" as a placeholder, bc IDK...
Result from fmt.Printf("Tweet %d - %s\n", k, v.Text):
Tweet 0 - You know I had to do it to them! #JennaJulien #Jenna_Marbles #juliensolomita #notjulen Got my first hydroflask ever…
Tweet 1 - RT #brenna_hinshaw: I was in J2 today and watched someone fill their hydroflask with vanilla soft serve... what starts here changes the wor…
Tweet 2 - I miss my hydroflask :(
This is my second week working with go and new to development. Any help would be great.
It doesn't look like the client is just returning you a slice of strings. The range syntax you're using (for k, v := range search.Statuses) returns two values for each iteration, the index in the slice (in this case k), and the object from the slice (in this case v). I don't know the type of search.Statuses - but I know that strings don't have a .Text field or method, which is how you're printing v currently.
To your question:
Is there any particular reason to return just a single struct with a Data field rather than directly returning the output of the twitter client?
Your function signature could look like this instead:
func (SearchTwitter) execute(input SearchTwitterInput) ([]<client response struct>, error)
And then you could operate on the text in those objects in wherever this function was called.
If you're dead-set on placing the data in your own struct, you could return a slice of them ([]*SearchTwitterOutput), in which case you could build a single SearchTwitterOutput in the for loop you're currently printing the tweets in and append it to the output list. That might look like this:
var output []*SearchTwitterOutput
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
output = append(output, &SearchTwitterOutput{
Data: v.Text,
})
}
return output, nil
But if your goal really is to return all of the results concatenated together and placed inside a single struct, I would suggest building a slice of strings (containing the text you want), and then joining them with the delimiter of your choosing. Then you could place the single output string in your return object, which might look something like this:
var outputStrings []string
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
outputStrings = append(outputStrings, v.Text)
}
output = strings.Join(outputStrings, ",")
return &SearchTwitterOutput{
Data: output,
}, nil
Though I would caution, it might be tricky to find a delimiter that will never show up in a tweet..
I have a map structured like map[string][]string. Now I have to find all keys which have the required values in the value slice. I can do something like this:
// allsvc is map[string][]string
var newsl []string
for k, v := range allsvc {
for _, val := range v {
if v == "type1" || v == "type2" {
newsl.append(k)
}
}
}
The map allsvc has atleast a half million entries at any given time, and the lookup is quite frequent. I get the allsvc map as an ouput of a 3rd party library and then I have to search in it using values in my api and provide a response. Given the high frequency of lookup not using keys but with values, the way i have done it makes my api response time to be in the seconds. Is there a way to better performance (speed of lookup)?
If you will query that map multiple times, it might be worth spending some time re-arranging it when you get it so that you can then query it faster.
It seems you need to invert the relationships, making the values in allsvc the keys in the new map, and having the keys as values so that you can then just make lookups in the new map.
This can be a way to re-arrange the map:
func arrangeMap(oldMap map[string][]string) map[string][]string {
newMap := make(map[string][]string)
for k, v := range oldMap {
for _, val := range v {
newMap[val] = append(newMap[val], k)
}
}
return newMap
}
See here a playground showing the idea:
https://play.golang.org/p/0ThZlX9xUn
I've been trying to read the values contained in an array sent by a Post Submit in a website.
Printing the r.PostForm data returns this map:
map[myArray[0].val1:[foo1] myArray[0].val2:[foo2] myArray[1].val1:[foo1] myArray[1].val2:[foo2]]
How can we manipulate the data? I already tried something like this:
func request(r http.ResponseWriter, w *http.Request){
r.ParseForm()
for key, array := range r.PostForm["myArray"] {
// Do something with the val1 and val2 values.
}
}
But this didn't work and I hadn't found any solution in the web.
Is possible to read an array contained in Post data using a basic solution?
It looks like the keys of your form are "myArray[0].val1", "myArray[0].val2", and so on.
Try:
r.ParseForm()
for k, v := range r.PostForm {
for i, value := range v {
// Do something with the i-th value for key k.
}
}