Golang slow when looping over Collection - go

Im feeding my app a json from Redis which I then unmarshal and loop through.
Here is what the json Im feeding from Redis looks like:
[
{
"titel": "test 1",
"event": "some value",
"pair": "some value",
"condition": [
"or",
[
"contains",
"url",
"/"
],[
"notcontains",
"url",
"hello"
]
],
"actions": [
[
"option1",
"12",
"1"
],
[
"option2",
"3",
"1"
]
]
}, {
"titel": "test 2",
"event": "some value",
"pair": "some value",
"condition": [
"or",
[
"contains",
"url",
"/"
]
],
"actions": [
[
"option1",
"12",
"1"
],
[
"option2",
"3",
"1"
]
]
}
]
My struct to store the json looks like this:
type Trigger struct {
Event string `json:"event"`
Pair string `json:"pair"`
Actions [][]string `json:"actions"`
Condition Condition `json:"condition"`
}
type Condition []interface{}
func (c *Condition) Typ() string {
return (*c)[0].(string)
}
func (c *Condition) Val() []string {
xs := (*c)[1].([]interface{})
ys := make([]string, len(xs))
for i, x := range xs {
ys[i] = x.(string)
}
return ys
}
func (c *Condition) String() (string, string, string) {
return c.Val()[0], c.Val()[1], c.Val()[2]
}
type Triggers struct {
Collection []Trigger
}
And Im unmarshaling it into the struct like so:
var triggers Triggers
err := json.Unmarshal([]byte(triggersJson), &triggers.Collection)
if err != nil {
fmt.Println("Error while unmarshaling triggers json: ", err)
}
This works and performs perfectly fine while testing it with siege.
However, as soon as I want to start loop over the struct Im starting to see long response times and timeouts.
This is how I loop over it:
for _,element := range triggers.Collection {
if element.Event == "some value" {
fmt.Println("We got: some value")
}
}
Performance without the loop is 2ms on 500 concurrent connections.
With the loop its 600ihs ms on 500 concurrent with a bunch of timeouts
Ideally Id like to change structure of the json to not include:
"or",
But Im not in control over the json so this is unfortunately impossible.
Any ideas whats causing this and how it could be resolved?
Edit
I found whats slowing the whole thing down.
Editing my struct like:
type Trigger struct {
Event string `json:"event"`
Pair string `json:"pair"`
Actions [][]string `json:"actions"`
//Condition Condition `json:"condition"`
}
Takes it from 600ms to 100ms. However now Im not able to parse the Condition.
Now sure how to parse the Condition in another way than Im currently doing due it having two different formats

Sounds weird, but try to avoid copying Collection elements. Something like this:
for i := range triggers.Collection {
if triggers.Collection[i].Event == "some value" {
fmt.Println("We got: some value")
}

Related

How to parse JSON-RPC table with different type

I want get informations in JSON-RPC file with this structure :
{
"id": "foo1",
"error": null,
"result": [
{
"key": [
"hello 1",
1,
"world 1"
],
"val": {
"type": "static"
}
},
{
"key": [
"hello 2",
1,
"world 2"
],
"val": {
"type": "static"
}
}
]
}
This is my parsing function, Key is string table (can't accept int type) :
type JsonRpcRsp struct {
Id string `json:"id"`
Error *string `json:"error"`
Result json.RawMessage `json:"result"`
}
type JsonRpcEntry_Val struct {
Type string `json:"type"`
}
type JsonRpcEntry struct {
Key [3]string `json:"key"`
Val JsonRpcEntry_Val `json:"val"`
}
jsonResult := JsonRpcRsp{}
json.Unmarshal(data, &jsonResult)
entries := []JsonRpcEntry{}
for _, val := range jsonResult {
json.Unmarshal(val.Result, &entries)
}
How to parse "key" table ?... problem is there are different types
key table structure is :
[ <string>, <int>, <string>]
To unmarshal arrays of different types in Go you'll need to use interfaces and consequently type assertions if you need access to the types.
This will work for you:
type Result struct {
Key [3]interface{} `json:"key"`
Val struct {
Type string `json:"type"`
} `json:"val"`
}
msg := JsonRpcRsp{}
json.Unmarshal(data, &msg)
var result []Result
json.Unmarshal(msg.Result, &result)
for _, v := range result {
key1 := v.Key[0].(string)
key2 := v.Key[1].(float64)
key3 := v.Key[2].(string)
fmt.Println(key1, key2, key3)
}
After asserting the three interfaces to their types, you can then work with them further, depending on your use case.

How to get the number of items from a structure sub field using reflect in go

I have some issues when getting the number of items from a sub field in a slice struct through reflect package.
This is how I'm trying to get the number of items from Items
func main() {
type Items struct {
Name string `json:"name"`
Present bool `json:"present"`
}
type someStuff struct {
Fields string `json:"fields"`
Items []Items `json:"items"`
}
type Stuff struct {
Stuff []someStuff `json:"stuff"`
}
some_stuff := `{
"stuff": [
{
"fields": "example",
"items": [
{ "name": "book01", "present": true },
{ "name": "book02", "present": true },
{ "name": "book03", "present": true }
]
}
]
}`
var variable Stuff
err := json.Unmarshal([]byte(some_stuff), &variable)
if err != nil {
panic(err)
}
//I want to get the number of items in my case 3
NumItems := reflect.ValueOf(variable.Stuff.Items)
}
This is the error:
variable.Items undefined (type []Stuff has no field or method Items)
I'm unsure if I can retrieve the number of items like that.
I have already fixed the issue.
In order to get the number of sub fields we can make use of Len() from reflect.ValueOf.
The code now is getting the number of Items:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
type Items struct {
Name string `json:"name"`
Present bool `json:"present"`
}
type someStuff struct {
Fields string `json:"fields"`
Items []Items `json:"items"`
}
type Stuff struct {
Stuff []someStuff `json:"stuff"`
}
some_stuff := `{
"stuff": [
{
"fields": "example",
"items": [
{ "name": "book01", "present": true },
{ "name": "book02", "present": true },
{ "name": "book03", "present": true }
]
}
]
}`
var variable Stuff
err := json.Unmarshal([]byte(some_stuff), &variable)
if err != nil {
panic(err)
}
//I want to get the number of items in my case 3
t := reflect.ValueOf(variable.Stuff[0].Items)
fmt.Println(t.Len())
}
Output: 3

REST API with gojsonschema: first path segment in URL cannot contain colon

I'm having issues making github.com/xeipuuv/gojsonschema work for my REST API that I'm currently building.
The procedure would look like this
User sends request to /api/books/create (in this case I'm sending a PUT request)
User inputs body parameters name and content
The server converts these body parameters into readable JSON
The server tries to validate the JSON using a json schema
The server performs the request
or that is how it should work.
I get this error when trying to validate the JSON and I have no clue how to fix it.
http: panic serving [::1]:58611: parse {"name":"1","content":"2"}: first path segment in URL cannot contain colon
type CreateParams struct {
Name string
Content string
}
func Create(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
data := &CreateParams{
Name: r.Form.Get("name"),
Content: r.Form.Get("Content"),
}
jsonData, err := json.Marshal(data)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(jsonData))
schema := `{
"required": [
"Name",
"Content"
],
"properties": {
"Name": {
"$id": "#/properties/Name",
"type": "string",
"title": "The Name Schema",
"default": "",
"examples": [
"1"
],
"minLength": 3,
"pattern": "^(.*)$"
},
"Content": {
"$id": "#/properties/Content",
"type": "string",
"title": "The Content Schema",
"default": "",
"examples": [
"2"
],
"pattern": "^(.*)$"
}
}
}`
schemaLoader := gojsonschema.NewStringLoader(schema)
documentLoader := gojsonschema.NewReferenceLoader(string(jsonData))
result, err := gojsonschema.Validate(schemaLoader, documentLoader)
if err != nil {
panic(err.Error())
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, desc := range result.Errors() {
fmt.Printf("- %s\n", desc)
}
}
}
My first thought was that it breaks because r.ParseForm() outputs things in a weird way, but I'm not sure anymore.
Note that I would like to have a "universal" method as I'm dealing with all kinds of requests: GET, POST, PUT, etc. But if you have a better solution I could work with that.
Any help is appreciated!

How to iterate over an []interface{} in Go

I'm struggling to get the keys and values of the following interface, which is the result of JSON marshaling the result returned by Execute as demonstrated in this example:
[
[
{
"id": 36,
"label": "TestThing",
"properties": {
"schema__testBoolean": [
{
"id": 40,
"value": true
}
],
"schema__testInt": [
{
"id": 39,
"value": 1
}
],
"schema__testNumber": [
{
"id": 38,
"value": 1.0879834
}
],
"schema__testString": [
{
"id": 37,
"value": "foobar"
}
],
"uuid": [
{
"id": 41,
"value": "7f14bf92-341f-408b-be00-5a0a430852ee"
}
]
},
"type": "vertex"
}
]
]
A reflect.TypeOf(result) results in: []interface{}.
I've used this to loop over the array:
s := reflect.ValueOf(result)
for i := 0; i < s.Len(); i++ {
singleVertex := s.Index(i).Elem() // What to do here?
}
But I'm getting stuck with errors like:
reflect.Value.Interface: cannot return value obtained from unexported
field or method
If you know that's your data structure, there's no reason to use reflection at all. Just use a type assertion:
for key, value := range result.([]interface{})[0].([]interface{})[0].(map[string]interface{}) {
// key == id, label, properties, etc
}
For getting the underlying value of an interface use type assertion. Read more about Type assertion and how it works.
package main
import (
"fmt"
)
func main() {
res, err := g.Execute( // Sends a query to Gremlin Server with bindings
"g.V(x)",
map[string]string{"x": "1234"},
map[string]string{},
)
if err != nil {
fmt.Println(err)
return
}
fetchValue(res)
}
func fetchValue(value interface{}) {
switch value.(type) {
case string:
fmt.Printf("%v is an interface \n ", value)
case bool:
fmt.Printf("%v is bool \n ", value)
case float64:
fmt.Printf("%v is float64 \n ", value)
case []interface{}:
fmt.Printf("%v is a slice of interface \n ", value)
for _, v := range value.([]interface{}) { // use type assertion to loop over []interface{}
fetchValue(v)
}
case map[string]interface{}:
fmt.Printf("%v is a map \n ", value)
for _, v := range value.(map[string]interface{}) { // use type assertion to loop over map[string]interface{}
fetchValue(v)
}
default:
fmt.Printf("%v is unknown \n ", value)
}
}

Parsing JSONSchema to struct type in golang

So, my use case consists of parsing varying JSON schemas into new struct types, which will be further used with an ORM to fetch data from a SQL database. Being compiled in nature, I believe there will not be an out-of-the-box solution in go, but is there any hack available to do this, without creating a separate go process. I tried with reflection, but could not find a satisfactory approach.
Currently, I am using a-h generate library which does generate the structs, but I am stuck at how to load these new struct types in go runtime.
EDIT
Example JSON Schema:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Address",
"id": "Address",
"type": "object",
"description": "address",
"properties": {
"houseName": {
"type": "string",
"description": "House Name",
"maxLength": 30
},
"houseNumber": {
"type": "string",
"description": "House Number",
"maxLength": 4
},
"flatNumber": {
"type": "string",
"description": "Flat",
"maxLength": 15
},
"street": {
"type": "string",
"description": "Address 1",
"maxLength": 40
},
"district": {
"type": "string",
"description": "Address 2",
"maxLength": 30
},
"town": {
"type": "string",
"description": "City",
"maxLength": 20
},
"county": {
"type": "string",
"description": "County",
"maxLength": 20
},
"postcode": {
"type": "string",
"description": "Postcode",
"maxLength": 8
}
}
}
Now, in the above-mentioned library, there is a command line tool, which generates the text for struct type for above json as below:
// Code generated by schema-generate. DO NOT EDIT.
package main
// Address address
type Address struct {
County string `json:"county,omitempty"`
District string `json:"district,omitempty"`
FlatNumber string `json:"flatNumber,omitempty"`
HouseName string `json:"houseName,omitempty"`
HouseNumber string `json:"houseNumber,omitempty"`
Postcode string `json:"postcode,omitempty"`
Street string `json:"street,omitempty"`
Town string `json:"town,omitempty"`
}
Now, the issue is that how to use this struct type without re-compilation in the program. There is a hack, where I can start a new go process, but that doesn't seem a good way to do it. One other way is to write my own parser for unmarshalling JSON schema, something like:
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"]}`)
var f interface{}
json.Unmarshal(b, &f)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case float64:
fmt.Println(k, "is float64", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
Can someone please suggest some pointers to look for. Thanks.
So it looks like you're trying to implement your own json marshalling. That's no biggie: the standard json package already supports that. Just have your type implement the MarshalJSON and UnmarshalJSON functions (cf first example on the docs). Assuming some fields will be shared (eg schema, id, type), you can create a unified type like this:
// poor naming, but we need this level of wrapping here
type Data struct {
Metadata
}
type Metadata struct {
Schema string `json:"$schema"`
Type string `json:"type"`
Description string `json:"description"`
Id string `json:"id"`
Properties json.RawMessage `json:"properties"`
Address *Address `json:"-"`
// other types go here, too
}
Now all properties will be unmarshalled into a json.RawMessage field (essentially this is a []byte field). What you can do in your custom unmarshall function now is something like this:
func (d *Data) UnmarshalJSON(b []byte) error {
meta := Metadata{}
// unmarshall common fields
if err := json.Unmarshal(b, &meta); err != nil {
return err
}
// Assuming the Type field contains the value that allows you to determine what data you're actually unmarshalling
switch meta.Type {
case "address":
meta.Address = &Address{} // initialise field
if err := json.Unmarshal([]byte(meta.Properties), meta.Address); err != nil {
return err
}
case "name":
meta.Name = &Name{}
if err := json.Unmarshal([]byte(meta.Properties), meta.Name); err != nil {
return err
}
default:
return errors.New("unknown message type")
}
// all done
d.Metadata = meta // assign to embedded
// optionally: clean up the Properties field, as it contains raw JSON, and is exported
d.Metadata.Properties = json.RawMessage{}
return nil
}
You can do pretty much the same thing for marshalling. First work out what type you're actually working with, then marshal that object into the properties field, and then marhsal the entire structure
func (d Data) MarshalJSON() ([]byte, error) {
var (
prop []byte
err error
)
switch {
case d.Metadata.Address != nil:
prop, err = json.Marshal(d.Address)
case d.Metadata.Name != nil:
prop, err = json.Marshal(d.Name) // will only work if field isn't masked, better to be explicit
default:
err = errors.New("No properties to marshal") // handle in whatever way is best
}
if err != nil {
return nil, err
}
d.Metadata.Properties = json.RawMessage(prop)
return json.Marshal(d.Metadata) // marshal the unified type here
}

Resources