REST API with gojsonschema: first path segment in URL cannot contain colon - go

I'm having issues making github.com/xeipuuv/gojsonschema work for my REST API that I'm currently building.
The procedure would look like this
User sends request to /api/books/create (in this case I'm sending a PUT request)
User inputs body parameters name and content
The server converts these body parameters into readable JSON
The server tries to validate the JSON using a json schema
The server performs the request
or that is how it should work.
I get this error when trying to validate the JSON and I have no clue how to fix it.
http: panic serving [::1]:58611: parse {"name":"1","content":"2"}: first path segment in URL cannot contain colon
type CreateParams struct {
Name string
Content string
}
func Create(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
data := &CreateParams{
Name: r.Form.Get("name"),
Content: r.Form.Get("Content"),
}
jsonData, err := json.Marshal(data)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(jsonData))
schema := `{
"required": [
"Name",
"Content"
],
"properties": {
"Name": {
"$id": "#/properties/Name",
"type": "string",
"title": "The Name Schema",
"default": "",
"examples": [
"1"
],
"minLength": 3,
"pattern": "^(.*)$"
},
"Content": {
"$id": "#/properties/Content",
"type": "string",
"title": "The Content Schema",
"default": "",
"examples": [
"2"
],
"pattern": "^(.*)$"
}
}
}`
schemaLoader := gojsonschema.NewStringLoader(schema)
documentLoader := gojsonschema.NewReferenceLoader(string(jsonData))
result, err := gojsonschema.Validate(schemaLoader, documentLoader)
if err != nil {
panic(err.Error())
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, desc := range result.Errors() {
fmt.Printf("- %s\n", desc)
}
}
}
My first thought was that it breaks because r.ParseForm() outputs things in a weird way, but I'm not sure anymore.
Note that I would like to have a "universal" method as I'm dealing with all kinds of requests: GET, POST, PUT, etc. But if you have a better solution I could work with that.
Any help is appreciated!

Related

Extract avro schema

Similar to this question
How to extract schema for avro file in python
Is there a way to read in an avro file in golang without knowing the schema beforehand and extract a schema?
How about something like this (adapted code from https://github.com/hamba/avro/blob/master/ocf/ocf.go):
package main
import (
"github.com/hamba/avro"
"log"
"os"
)
// HeaderSchema is the Avro schema of a container file header.
var HeaderSchema = avro.MustParse(`{
"type": "record",
"name": "org.apache.avro.file.Header",
"fields": [
{"name": "magic", "type": {"type": "fixed", "name": "Magic", "size": 4}},
{"name": "meta", "type": {"type": "map", "values": "bytes"}},
{"name": "sync", "type": {"type": "fixed", "name": "Sync", "size": 16}}
]
}`)
var magicBytes = [4]byte{'O', 'b', 'j', 1}
const (
schemaKey = "avro.schema"
)
// Header represents an Avro container file header.
type Header struct {
Magic [4]byte `avro:"magic"`
Meta map[string][]byte `avro:"meta"`
Sync [16]byte `avro:"sync"`
}
func main() {
r, err := os.Open("path/my.avro")
if err != nil {
log.Fatal(err)
}
defer r.Close()
reader := avro.NewReader(r, 1024)
var h Header
reader.ReadVal(HeaderSchema, &h)
if reader.Error != nil {
log.Println("decoder: unexpected error: %v", reader.Error)
}
if h.Magic != magicBytes {
log.Println("decoder: invalid avro file")
}
schema, err := avro.Parse(string(h.Meta[schemaKey]))
if err != nil {
log.Println(err)
}
log.Println(schema)
}
Both https://github.com/hamba/avro and https://github.com/linkedin/goavro can decode Avro OCF files (which it sounds like is what you have) without an explicit schema file.
Once you've created a new reader/decoder, you can retrieve the metadata, which includes the schema at key avro.schema: https://pkg.go.dev/github.com/hamba/avro/ocf#Decoder.Metadata and https://pkg.go.dev/github.com/linkedin/goavro#OCFReader.MetaData

Passing variables with graphql using a http client

I'm trying to interface with shopify storefront api in golang. But this is my first encounter with graphql and I'm a bit confused.
I need to pass variables for the request and did some research where I came to the conclusion that I could pass variables like shown below. But shopify keeps returning this error:
{"errors":[{"message":"Parse error on \"variables\" (IDENTIFIER) at [13, 3]","locations":[{"line":13,"column":3}]}]}
Here is my current code:
body := strings.NewReader(fmt.Sprintf(`
mutation ($checkoutId: ID!, $lineItems: [CheckoutLineItemInput!]!) {
checkoutLineItemsAdd(checkoutId: $checkoutId, lineItems: $lineItems) {
userErrors {
message
field
}
checkout {
id
}
}
}
variables: { "lineItems": [ { "quantity": 1, "variantId": "%s" } ], "checkoutId": "%s" }
`, productID, checkoutID))
req, err := http.NewRequest("POST", "https://myshop.myshopify.com/api/graphql", body)
if err != nil {
// handle err
fmt.Println(err)
}
req.Header.Set("Content-Type", "application/graphql")
req.Header.Set("X-Shopify-Storefront-Access-Token", "mytoken")
resp, err := http.DefaultClient.Do(req)
if err != nil {
// handle err
fmt.Println(err)
}
defer resp.Body.Close()
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
// handle err
fmt.Println(err)
}
My question is, what is the correct way of passing variables when using application/graphql?
After updating the code with the answer of #DanielRearden I'm now getting this error:
{"errors":[{"message":"Variable checkoutId of type ID! was provided invalid value","locations":[{"line":1,"column":11}],"extensions":{"value":null,"problems":[{"path":[],"explanation":"Expected value to not be null"}]}},{"message":"Variable lineItems of type [CheckoutLineItemInput!]! was provided invalid value","locations":[{"line":1,"column":29}],"extensions":{"value":null,"problems":[{"path":[],"explanation":"Expected value to not be null"}]}}]}
Updated code:
body := strings.NewReader(fmt.Sprintf(`{
"query": "mutation ($checkoutId: ID!, $lineItems: [CheckoutLineItemInput!]!) { checkoutLineItemsAdd(checkoutId: $checkoutId, lineItems: $lineItems) { userErrors { message field } checkout { id } }}",
"variables": {
"$lineItems": [
{ "quantity": 1, "variantId": "%s" }
],
"$checkoutId": "%s"
}
}`, productID, checkoutID))
...
...
req.Header.Set("Content-Type", "application/json")
Removing the $ token on the variables as shown below returns another error:
{
"query": "mutation ($checkoutId: ID!, $lineItems: [CheckoutLineItemInput!]!) { checkoutLineItemsAdd(checkoutId: $checkoutId, lineItems: $lineItems) { userErrors { message field } checkout { id } }}",
"variables": {
"checkoutId": "%s",
"lineItems": [
{ "quantity": 1, "variantId": "%s" }
]
}
}
And the error is: (status 500)
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="referrer" content="never" />
<title>Something went wrong</title>
But I guess this is not a graphql issue anymore but more of a shopify api problem.
You can't use variables when you set the Content Type to application/graphql because the entire request body is treated as a single GraphQL document. Variables cannot be included inside a GraphQL document -- they have to be submitted separately. Instead of using application/graphql, you should use application/json as the Content Type. Then, your request body should be a JSON string with two keys -- query and variables:
{
"query": "mutation ($checkoutId: ID!, $lineItems: [CheckoutLineItemInput!]!) {...}",
"variables": {
"checkoutId" "",
"lineItems": [
...
]
}
}

Parsing JSONSchema to struct type in golang

So, my use case consists of parsing varying JSON schemas into new struct types, which will be further used with an ORM to fetch data from a SQL database. Being compiled in nature, I believe there will not be an out-of-the-box solution in go, but is there any hack available to do this, without creating a separate go process. I tried with reflection, but could not find a satisfactory approach.
Currently, I am using a-h generate library which does generate the structs, but I am stuck at how to load these new struct types in go runtime.
EDIT
Example JSON Schema:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Address",
"id": "Address",
"type": "object",
"description": "address",
"properties": {
"houseName": {
"type": "string",
"description": "House Name",
"maxLength": 30
},
"houseNumber": {
"type": "string",
"description": "House Number",
"maxLength": 4
},
"flatNumber": {
"type": "string",
"description": "Flat",
"maxLength": 15
},
"street": {
"type": "string",
"description": "Address 1",
"maxLength": 40
},
"district": {
"type": "string",
"description": "Address 2",
"maxLength": 30
},
"town": {
"type": "string",
"description": "City",
"maxLength": 20
},
"county": {
"type": "string",
"description": "County",
"maxLength": 20
},
"postcode": {
"type": "string",
"description": "Postcode",
"maxLength": 8
}
}
}
Now, in the above-mentioned library, there is a command line tool, which generates the text for struct type for above json as below:
// Code generated by schema-generate. DO NOT EDIT.
package main
// Address address
type Address struct {
County string `json:"county,omitempty"`
District string `json:"district,omitempty"`
FlatNumber string `json:"flatNumber,omitempty"`
HouseName string `json:"houseName,omitempty"`
HouseNumber string `json:"houseNumber,omitempty"`
Postcode string `json:"postcode,omitempty"`
Street string `json:"street,omitempty"`
Town string `json:"town,omitempty"`
}
Now, the issue is that how to use this struct type without re-compilation in the program. There is a hack, where I can start a new go process, but that doesn't seem a good way to do it. One other way is to write my own parser for unmarshalling JSON schema, something like:
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"]}`)
var f interface{}
json.Unmarshal(b, &f)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case float64:
fmt.Println(k, "is float64", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
Can someone please suggest some pointers to look for. Thanks.
So it looks like you're trying to implement your own json marshalling. That's no biggie: the standard json package already supports that. Just have your type implement the MarshalJSON and UnmarshalJSON functions (cf first example on the docs). Assuming some fields will be shared (eg schema, id, type), you can create a unified type like this:
// poor naming, but we need this level of wrapping here
type Data struct {
Metadata
}
type Metadata struct {
Schema string `json:"$schema"`
Type string `json:"type"`
Description string `json:"description"`
Id string `json:"id"`
Properties json.RawMessage `json:"properties"`
Address *Address `json:"-"`
// other types go here, too
}
Now all properties will be unmarshalled into a json.RawMessage field (essentially this is a []byte field). What you can do in your custom unmarshall function now is something like this:
func (d *Data) UnmarshalJSON(b []byte) error {
meta := Metadata{}
// unmarshall common fields
if err := json.Unmarshal(b, &meta); err != nil {
return err
}
// Assuming the Type field contains the value that allows you to determine what data you're actually unmarshalling
switch meta.Type {
case "address":
meta.Address = &Address{} // initialise field
if err := json.Unmarshal([]byte(meta.Properties), meta.Address); err != nil {
return err
}
case "name":
meta.Name = &Name{}
if err := json.Unmarshal([]byte(meta.Properties), meta.Name); err != nil {
return err
}
default:
return errors.New("unknown message type")
}
// all done
d.Metadata = meta // assign to embedded
// optionally: clean up the Properties field, as it contains raw JSON, and is exported
d.Metadata.Properties = json.RawMessage{}
return nil
}
You can do pretty much the same thing for marshalling. First work out what type you're actually working with, then marshal that object into the properties field, and then marhsal the entire structure
func (d Data) MarshalJSON() ([]byte, error) {
var (
prop []byte
err error
)
switch {
case d.Metadata.Address != nil:
prop, err = json.Marshal(d.Address)
case d.Metadata.Name != nil:
prop, err = json.Marshal(d.Name) // will only work if field isn't masked, better to be explicit
default:
err = errors.New("No properties to marshal") // handle in whatever way is best
}
if err != nil {
return nil, err
}
d.Metadata.Properties = json.RawMessage(prop)
return json.Marshal(d.Metadata) // marshal the unified type here
}

Golang slow when looping over Collection

Im feeding my app a json from Redis which I then unmarshal and loop through.
Here is what the json Im feeding from Redis looks like:
[
{
"titel": "test 1",
"event": "some value",
"pair": "some value",
"condition": [
"or",
[
"contains",
"url",
"/"
],[
"notcontains",
"url",
"hello"
]
],
"actions": [
[
"option1",
"12",
"1"
],
[
"option2",
"3",
"1"
]
]
}, {
"titel": "test 2",
"event": "some value",
"pair": "some value",
"condition": [
"or",
[
"contains",
"url",
"/"
]
],
"actions": [
[
"option1",
"12",
"1"
],
[
"option2",
"3",
"1"
]
]
}
]
My struct to store the json looks like this:
type Trigger struct {
Event string `json:"event"`
Pair string `json:"pair"`
Actions [][]string `json:"actions"`
Condition Condition `json:"condition"`
}
type Condition []interface{}
func (c *Condition) Typ() string {
return (*c)[0].(string)
}
func (c *Condition) Val() []string {
xs := (*c)[1].([]interface{})
ys := make([]string, len(xs))
for i, x := range xs {
ys[i] = x.(string)
}
return ys
}
func (c *Condition) String() (string, string, string) {
return c.Val()[0], c.Val()[1], c.Val()[2]
}
type Triggers struct {
Collection []Trigger
}
And Im unmarshaling it into the struct like so:
var triggers Triggers
err := json.Unmarshal([]byte(triggersJson), &triggers.Collection)
if err != nil {
fmt.Println("Error while unmarshaling triggers json: ", err)
}
This works and performs perfectly fine while testing it with siege.
However, as soon as I want to start loop over the struct Im starting to see long response times and timeouts.
This is how I loop over it:
for _,element := range triggers.Collection {
if element.Event == "some value" {
fmt.Println("We got: some value")
}
}
Performance without the loop is 2ms on 500 concurrent connections.
With the loop its 600ihs ms on 500 concurrent with a bunch of timeouts
Ideally Id like to change structure of the json to not include:
"or",
But Im not in control over the json so this is unfortunately impossible.
Any ideas whats causing this and how it could be resolved?
Edit
I found whats slowing the whole thing down.
Editing my struct like:
type Trigger struct {
Event string `json:"event"`
Pair string `json:"pair"`
Actions [][]string `json:"actions"`
//Condition Condition `json:"condition"`
}
Takes it from 600ms to 100ms. However now Im not able to parse the Condition.
Now sure how to parse the Condition in another way than Im currently doing due it having two different formats
Sounds weird, but try to avoid copying Collection elements. Something like this:
for i := range triggers.Collection {
if triggers.Collection[i].Event == "some value" {
fmt.Println("We got: some value")
}

Unmarshaling values of JSON objects

If given the string, from a MediaWiki API request:
str = ` {
"query": {
"pages": {
"66984": {
"pageid": 66984,
"ns": 0,
"title": "Main Page",
"touched": "2012-11-23T06:44:22Z",
"lastrevid": 1347044,
"counter": "",
"length": 28,
"redirect": "",
"starttimestamp": "2012-12-15T05:21:21Z",
"edittoken": "bd7d4a61cc4ce6489e68c21259e6e416+\\"
}
}
}
}`
What can be done to get the edittoken, using Go's json package (keep in mind the 66984 number will continually change)?
When you have a changing key like this the best way to deal with it is with a map. In the example below I've used structs up until the point we reach a changing key. Then I switched to a map format after that. I linked up a working example as well.
http://play.golang.org/p/ny0kyafgYO
package main
import (
"fmt"
"encoding/json"
)
type query struct {
Query struct {
Pages map[string]interface{}
}
}
func main() {
str := `{"query":{"pages":{"66984":{"pageid":66984,"ns":0,"title":"Main Page","touched":"2012-11-23T06:44:22Z","lastrevid":1347044,"counter":"","length":28,"redirect":"","starttimestamp":"2012-12-15T05:21:21Z","edittoken":"bd7d4a61cc4ce6489e68c21259e6e416+\\"}}}}`
q := query{}
err := json.Unmarshal([]byte(str), &q)
if err!=nil {
panic(err)
}
for _, p := range q.Query.Pages {
fmt.Printf("edittoken = %s\n", p.(map[string]interface{})["edittoken"].(string))
}
}
Note that if you use the &indexpageids=true parameter in the API request URL, the result will contain a "pageids" array, like so:
str = ` {
"query": {
"pageids": [
"66984"
],
"pages": {
"66984": {
"pageid": 66984,
"ns": 0,
"title": "Main Page",
"touched": "2012-11-23T06:44:22Z",
"lastrevid": 1347044,
"counter": "",
"length": 28,
"redirect": "",
"starttimestamp": "2012-12-15T05:21:21Z",
"edittoken": "bd7d4a61cc4ce6489e68c21259e6e416+\\"
}
}
}
}`
so you can use pageids[0] to access the continually changing number, which will likely make things easier.

Resources