What I can be doing wrong in my handler?
type Patient struct {
FirstName string
LastName string
}
func createHandler(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
p := new(Patient)
err := json.NewDecoder(r.Body).Decode(&p)
if err != nil && err != io.EOF {
log.Fatal(err)
}
log.Print(p)
}
I always get from curl -X POST $PATH -d '{"firstName": "Julian"}'
2016/01/23 17:33:50 &{ }
Edit
I've added fmt.Println(r) and got this
2016/01/23 18:33:22 &{POST /patients/ HTTP/1.1 1 1 map[Accept:[*/*] Content-Type:[application/json] User-Agent:[curl/7.46.0]] 0x9b2820 0 [] false localhost:8081 map[] map[] <nil> map[] [::1]:35394 /patients/ <nil> <nil>}
after debug the previous line I realized it's a problem with the subrouter of gorilla toolkit
Two things:
Since you're using the JSON lib convention, then your post form data should be FirstName and not firstName. If you really wanted to use firstName, you need to add this struct tag to the field: json:"firstName".
Next, you don't need to put & in &p anymore because new(Patient) already returns a pointer. Just put p.
Although personally, I'd just do p := Patient{} just so I know it's not a pointer and use &p in the decoder.
You have to remove the r.ParseForm. You can read here
https://golang.org/pkg/net/http/#Request.ParseForm
For POST or PUT requests, it also parses the request body
So your request body will be empty after the call of ParseForm and you will receive an empty struct.
Related
I am trying to manipulate a golang sync.Map of sync.Map, but I have some issues with the casting.
I have the following code:
func (cluster *Cluster) action(object1, object2 MyObject) {
value, _ := cluster.globalMap.LoadOrStore(object1.name, sync.Map{})
localMap := value.(sync.Map)
localMap.Store(object2.Name, object2)
value2, _ := cluster.resourceInflight.Load(node.Name)
forComparison := value2.(sync.Map)
fmt.Println(localMap.Load(object2.Name))
fmt.Println(forComparison.Load(object2.Name))
}
{myObject map[] map[]} true
<nil> false
I am doing this since I wish to keep the content of localMap thread safe.
The problem is I am expecting to have the same result for my two print, as "forComparison" should be pointing to the same object than "localMap". But second result is nil.
I am suspecting that the problem is coming from the casting of the interface "value" into an actual "sync.Map". But I am not sure how I can actually call the .Store method with inline casting.
I thought about Storing localMap inside cluster.globalMap, but this seems incorrect to me as it would break the whole point of using a localSyncMap and create concurrency issues.
Any input on what I should do ?
As per the comments the issue was that you were copying a sync.Map; the following code will fail (output "Not found" - playground):
var sm sync.Map
var x interface{}
x = sm
sm2 := x.(sync.Map)
sm2.Store("test", "test")
result, ok := sm.Load("test")
if ok {
fmt.Printf("Found: %s\n", result)
} else {
fmt.Printf("Not found\n")
}
Whereas using a pointer works as expected:
var sm sync.Map
var x interface{}
x = &sm
sm2 := x.(*sync.Map)
sm2.Store("test", "test")
result, ok := sm.Load("test")
if ok {
fmt.Printf("Found: %s\n", result)
} else {
fmt.Printf("Not found\n")
}
Running go vet would probably have warned you about other issues (sync.Map contains a sync.Mutex and these "must not be copied after first use").
Note that the docs for Sync.Map state:
The Map type is specialized. Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content.
I have a reverse proxy API that reads the parameters of a localhost API call and then sends those parameters to a 3rd party API.
I'm able to get this working correctly if I only use one parameter. Like so:
http://localhost:8080/path?page=1
I want to be able to use more than one parameter however like so:
http://localhost:8080/path?page=1¶m=x
Please see my code below:
This function catches an HTTP request and then sends those parameters to another API.
func (s *Server) getReverseProxy(w http.ResponseWriter, r *http.Request) {
// when I try to append another query in the list a long with page, I get an error
keys, ok := r.URL.Query()["page"]
if !ok || len(keys[0]) < 1 {
log.Println("Url Param 'page' is missing")
return
}
// Query()["key"] will return an array of items,
// we only want the single item.
key := keys[0]
log.Println("Url Param 'page' is: " + string(key))
params := url.Values{
"page[size]": []string{"100"},
"page[number]": []string{""},
}
u := &url.URL{
Scheme: "https",
Host: "url.com",
Path: "/path",
RawQuery: params.Encode(),
}
}
Without having to refractor, am I missing something simple here? How can I add another parameter for my function to catch?
The line of code below ...
keys, ok := r.URL.Query()["page"]
it returns the param value of page, but in []string type. To retrieve more params, simply add similar statement with different param name. for example:
keysPage, ok := r.URL.Query()["page"]
keysParamA, ok := r.URL.Query()["ParamA"]
keysParamB, ok := r.URL.Query()["ParamB"]
keysParamC, ok := r.URL.Query()["ParamC"]
Or, you can also use the r.URL.Query().Get(key) to return the param value in string type.
page := r.URL.Query().Get("page")
paramA := r.URL.Query().Get("ParamA")
paramB := r.URL.Query().Get("ParamB")
paramC := r.URL.Query().Get("ParamC")
r.URL.Query() returns a map[string][]string
you can do a
keys, ok := r.URL.Query()
//browse through keys by
keys["params"]
keys["page"]
I have a Go project where I'm using Zap structured logging to log the contents of structs. That's how I initialise the logger:
zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
panic(err)
}
Initially I started with my own structs with json tags and it all worked perfectly:
zapLog.Info("Event persisted", zap.Any("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
I now switched to protobuf and I'm struggling to achieve the same result. Initially I just got the "reflected map" version, when using zap.Any():
zapLog.Info("Event persisted", zap.Any("event", &event))
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\",
type:\"updated\" value:{...}, context:<key: foo, value:bar>}
I tried marshalling the object with the jsonpb marshaller, which generated the correct output on itself, however, when I use it in zap.String(), the string is escaped, so I get an extra set of '\' in front of each quotation mark. Since there's processing of the logs at a later point, this causes problems there and hence I want to avoid it:
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
// handle error
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\",
\"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":"bar\"}}"}
I then tried using zap.Reflect() instead of zap.Any() which was the closest thing I could get to what I need, except that enums are rendered as their numerical values (the initial solution did not have enums, so that didn't work in the pre-protobuf solution either):
zapLog.Info("Event persisted", zap.Reflect("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
The only option I see so far is to write my own MarshalLogObject() function:
type ZapEvent struct {
event *Event
}
func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {
encoder.AddString("sourceType", z.event.SourceType.String()
// implement encoder for each attribute
}
func processEvent(e Event) {
...
zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}
But since it's a complex struct, I would rather use a less error prone and maintenance heavy solution. Ideally, I would tell zap to use the jsonpb marshaller somehow, but I don't know if that's possible.
Use zap.Any with a json.RawMessage. You can convert directly the byte output of jsonpb.Marshaler:
foo := &pb.FooMsg{
Foo: "blah",
Bar: 1,
}
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, foo); err != nil {
// handle error
}
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))
The bytes will be printed as:
Event persisted {"event": {"foo":"blah","bar":"1"}}`
I believe that's the easiest way, however I'm also aware of a package kazegusuri/go-proto-zap-marshaler (I'm not affiliated to it) that generates MarshalLogObject() implementations as a protoc plugin. You may want to take a look at that too.
I used another way to jsonify protos.
Since protos can be naturally marshaled, I just wrapped them in the strict-to-json marshaler.
And you can modify the internals to use protojson (newer jsonpb).
Unlike the marshaler in the previous solution, this one doesn't require ahead-of-logging processing.
type jsonObjectMarshaler struct {
obj any
}
func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
bytes, err := json.Marshal(j.obj)
// bytes, err := protojson.Marshal(j.obj)
if err != nil {
return nil, fmt.Errorf("json marshaling failed: %w", err)
}
return bytes, nil
}
func ZapJsonable(key string, obj any) zap.Field {
return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}
Then to use it, just
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))
I have a short Go program that runs the go list -json command for several packages, stores the output of each run of the command in a json.RawMessage, appends each json.RawMessage into a slice of json.RawMessages, and then returns the result to the server after concatenating each of the json.RawMessages together and compacting the json. However, there is an error message that gets produced when I run json.Compact that I can't locate the source of. Googling this error message reveals that most people who seem to encounter it--whether it's for an invalid , or some other character--have a hard time finding the source of it.
invalid character ',' looking for beginning of value
The code with comments is available to view here on play.golang.org (although it won't run there) and also below.
Question: can you explain the source of this error and how to prevent it?
(Note, some of the packages were included just for testing purposes)
package main
import (
"expvar"
"encoding/json"
"bytes"
"fmt"
"github.com/go-martini/martini"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
"go/build"
"log"
"math/rand"
"net/http"
_ "net/http/pprof"
"os/exec"
)
type myType struct {
J []json.RawMessage
}
var pack map[string]string
type GoList struct {
Imports []string
}
type Import struct {
Dir string
ImportPath string
Name string
Target string
Standard bool
Root string
GoFiles []string
Imports []string
Deps []string
}
const contentTypeJSON = "application/json"
func main() {
http.HandleFunc("/importgraph", func(w http.ResponseWriter, r *http.Request) { importGraph(w, r) })
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("Inside handler")
fmt.Fprintf(w, "Hello world from my Go program!")
}
func importGraph(w http.ResponseWriter, r *http.Request) {
pack = make(map[string]string)
var t myType
cmd := exec.Command("go", "list", "-json")
stdout, err := cmd.Output()
if err != nil {
println(err.Error())
return
}
var list GoList
err = json.Unmarshal(stdout, &list)
for _, d := range list.Imports {
//get the imports for each of the packages listed by go list -json
t.imports(d)
}
var buff bytes.Buffer
//concatenate the separate json.RawMessages together into json
buff.WriteByte('[')
for i, j := range t.J {
if i != 0 {
buff.WriteByte(',')
}
buff.Write([]byte(j))
}
buff.WriteByte(']')
var buffer bytes.Buffer
if err := json.Compact(&buffer, buff.Bytes()); err != nil {
println(err.Error()) //error message: invalid character ',' looking for beginning of value
return
}
w.Header().Set("Content-Type", contentTypeJSON)
w.Write(buffer.Bytes())
}
func (myObj *myType) imports(pk string) error {
cmd := exec.Command("go", "list", "-json", pk)
stdout, _ := cmd.Output()
pack[pk] = pk
var deplist Import
json.Unmarshal(stdout, &deplist)
var newj json.RawMessage
json.Unmarshal(stdout, &newj)
myObj.J = append(myObj.J, newj)
for _, imp := range deplist.Imports {
if _, ok := pack[imp]; !ok {
myObj.imports(imp) //recursive call to get the imports of the imports etc
}
}
return nil
}
First, as has been commented, are you sure you can't use
the go/build package directly rather than running go list?
I Wouldn't use println (or fmt.Println) inside HTTP handlers. It's much better to use log.Println and/or get the error into the ResponseWriter. Also, it's a good idea to wrap your ListenAndServe call with log.Fatal.
When printing/logging error values you can just use err, no need to have err.Error().
Further, when you actually want to do something more detailed than just reporting/logging the error message you can look at it's type and other info. For example, log.Printf("verbose error info: %#v", err) gives:
&json.SyntaxError{msg:"invalid character ',' looking for beginning of value", Offset:0}
I tried this because I know the json package returns various error types with additional info and I was hoping the offset value would be of help. If it had been then something like this might have been helpful:
if err := json.Compact(…) {
if err != nil {
log.Println("json.Compact:", err)
if serr, ok := err.(*json.SyntaxError); ok {
log.Println("Occurred at offset:", serr.Offset)
// … something to show the data in buff around that offset …
}
}
}
But offset zero isn't helpful :(
So although this doesn't identify you problem hopefully
it can be of some help to your further investigation.
Edit:
So after adding:
log.Println("Write file:", ioutil.WriteFile("data.json", buff.Bytes(), 0600))
to the above error handling block I then ran a JSON validator on the resultant file and it identified this piece:
"XTestImports": [
"io",
"log",
"net"
]
},,{
"Dir": "/usr/local/go/src/mime",
"ImportPath": "mime",
"Name": "mime",
Note the double ,,.
That should tell you whete the error in your code is.
But if not, you need to skip empty entries, either when processing t.J or when you build it. The later is better and just involves:
if len(newj) > 0 {
myObj.J = append(myObj.J, newj)
}
(where btw you don't check for errors from json.Unmarshal so it's not clear if that is supposed to ever be empty or if it's empty due to a preceeding error. Never ignore error returns!)
I also encountered the same error message in a Go program, but the error message was within the HTTP response error, in HTML format when my HTTP response parser expected JSON.
For me, the solution was to change my request to include setting the Content-Type header to application/json. How you do this depends on which http client library you happen to be using; if you have access to the http.Header core type, you can set the header with .Set(...).
I realize the scope of this fix for me may not apply to the original question, but I came here first after googling and thought this would help others, since the message was not particularly obvious at first glance. The hint is that the invalid < character is the first HTML character in the error/response, which is likely the result of the request type not being set to application/json, thus the server responds with a non JSON response.
For me the issue was I was trying to parse the already parsed JSON.
I was also facing this error "invalid character 'N' looking for beginning of value".
This error was coming while "unmarshalling the non-json response into a json". I was expecting a json response, so wrote go code to unmarshal it into a json. But, due to URL change, the response that I was getting was a text ie. "404 Not found" error, which cannot be unmarshalled into a json.
"invalid character 'N' looking for beginning of value"
So, to summarise, this error appears when we are trying to unmarshal a non-json response (text/html/xml) into json.
Reason for this eerie error message is :
// When unmarshaling quoted strings, invalid UTF-8 or
// invalid UTF-16 surrogate pairs are not treated as an error.
// Instead, they are replaced by the Unicode replacement
// character U+FFFD.
https://golang.org/src/encoding/json/decode.go
In my case I saved my json as string then parsed it by :
stringData = JSON.parse(myJsonString)
I also had the same error another time using gin-context-ShouldBind() (https://godoc.org/github.com/gin-gonic/gin#Context.ShouldBind) and mapping my json to go object:
error was because it needs a json as string, so I used : JSON.stringify(jsonObject) when sending my request from front-end part.
And in case someone has the same problem as me, I needed to call JSON.stringify on my post data.
I encountered a similar problem with my error message being:
invalid character 'I' looking for beginning of value
In my case, i was trying to decode BSON using json.Unmarshal. Json doesn't recognize the ISODate type, which caused this error.
I had a similar issue. For me I omitted the first letter of my authorization token. So instead of
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I used this
"yJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I'm building an http api and every one of my handlers returns JSON data, so I built a wrapper function that handles the JSON marshalling and http response (I've included the relevant section from the wrapper as well as one of the sample handlers below).
What is the best way to pass arbitrarily nested structs (the structs also contain arbitrary types/number of fields). Right now I've settled on a map with string keys and interface{} values. This works, but is this the most idiomatic go way to do this?
result := make(map[string]interface{})
customerList(httpRequest, &result)
j, err := json.Marshal(result)
if err != nil {
log.Println(err)
errs := `{"error": "json.Marshal failed"}`
w.Write([]byte(errs))
return
}
w.Write(j)
func customerList(req *http.Request, result *map[string]interface{}) {
data, err := database.RecentFiftyCustomers()
if err != nil {
(*result)["error"] = stringifyErr(err, "customerList()")
return
}
(*result)["customers"] = data//data is a slice of arbitrarily nested structs
}
If you do not know in advance what types, what structure and which nesting you get, there is no option but to decode it into something generic like map[string]interface{}. So nothing "idiomatic" or "non-idiomatic" here.
(Personally I'd try to somehow fix the structs and not have "arbitrary" nestings, and combinations.)