The use case is parsing idls in service A and parsing protobuf messages to json in service B.
Using jhump.protoreflect to parse idl and get desc.MessageDescriptor, then use protojson to marshal descriptorpb.DescriptorProto
h, _ := protojson.Marshal(md.Descriptor.AsDescriptorProto())
Using protojson to unmarshal h and get DescriptorProto
var z descriptorpb.รง
if err := protojson.Unmarshal(h, &z); err != nil {
fmt.Println(err.Error())
}
How to parse message by using DescriptorProto?
If I'm doing the whole work in the same service, I'm able to parse the message by using MessageDescriptor
dm := dynamic.NewMessage(md)
dm.Unmarshal(body)
dm.MarshalJSONPB(&jsonpb.Marshaler{OrigName: true, EnumsAsInts: true})
but the issue is parsing message is done in a different service, so I'm looking to keep some "message descriptor" in database and can be used by service B.
Any suggestion?
Related
I'm trying to use the K8S Go client to list the ClusterServiceVersions.
It could be enough to have the raw response body.
I've tried this:
data, err := clientset.RESTClient().Get().Namespace(namespace).
Resource("ClusterServiceVersions").
DoRaw(context.TODO())
if err != nil {
panic(err.Error())
}
fmt.Printf("%v", string(data))
But it returns the following error:
panic: the server could not find the requested resource (get ClusterServiceVersions.meta.k8s.io)
How do I specify to use the operators.coreos.com group?
Looking at some existing code I've also tried to add
VersionedParams(&v1.ListOptions{}, scheme.ParameterCodec)
But it result in this other error:
panic: v1.ListOptions is not suitable for converting to "meta.k8s.io/v1" in scheme "pkg/runtime/scheme.go:100"
It is possible to do a raw request using the AbsPath() method.
path := fmt.Sprintf("/apis/operators.coreos.com/v1alpha1/namespaces/%s/clusterserviceversions", namespace)
data, err := clientset.RESTClient().Get().
AbsPath(path).
DoRaw(ctx)
Also notice that if you want to define clientset using the interface (kubernetes.Interface) instead of the concrete type (*kubernetes.Clientset) the method clientset.RESTClient() is not directly accessible, but you can use the following one:
clientset.Discovery().RESTClient()
I know this code need to send a JSON instead of form data in the API
err := ctx.ShouldBindJSON(&modelAdd)
if err != nil {
return err
}
But I need to add file, is there anything like ShouldBindJSON but for FormData?
You can use ShouldBind to get data from form data as the documentation says
https://github.com/gin-gonic/gin#model-binding-and-validation
I have the following requirement: return errors from a REST API in the following format:
Error format
422
{
"name-of-field": [
"can't be blank",
"is too silly"
]
}
My code looks like this:
var PostFeedback = func(w http.ResponseWriter, r *http.Request) {
params := mux.Vars(r)
surveyId := params["id"]
feedback := &models.Feedback{}
err := json.NewDecoder(r.Body).Decode(feedback)
if err != nil {
jsonError := fmt.Sprintf(`{
"%s": [
"%s"
]
}`, "errors", err)
log.Printf("invalid input format, %v", jsonError)
resp := map[string]interface{}{"error": jsonError}
u.Respond(w, resp)
return
}
Questions:
How do I get the names of the offending fields?
How do I satisfy the requirement best?
The encoding/json package doesn't provide validation for "blank", nor "silly" values. It will return an error only if the data in the body is not a valid json, or if the field types in the json do not, according to the package's spec, match the field types of the structure into which you're trying to decode that json.
The 1st type of error would be the json.SyntaxError, if you get this it is not always possible to satisfy your requirements since there may be no actual fields which you could use in your response, or if there are json fields, they, and their values, may be perfectly valid json, but the cause of the error may lie elsewhere (see example).
In cases where the data holds actual json fields but it has, for example, non-json values you could use the Offset field of the SyntaxError type to find the closest preceding field in the data stream. Using strings.LastIndex you can implement a naive solution to look backwards for the field.
data := []byte(`{"foobar": i'm not json}`)
err := json.Unmarshal(data, &T{})
se, ok := err.(*json.SyntaxError)
if !ok {
panic(err)
}
field := string(data[:se.Offset])
if i := strings.LastIndex(field, `":`); i >= 0 {
field = field[:i]
if j := strings.LastIndex(field, `"`); j >= 0 {
field = field[j+1:]
}
}
fmt.Println(field) // outputs foobar
Playground link
NOTE: As you can see, for you to be able to look for the field, you need to have access to the data, but when you're using json.NewDecoder and passing it the request's body directly, without first storing its contents somewhere, you'll loose access to that data once the decoder's Decode method is done. This is because the body is a stream of bytes wrapped in a io.ReadCloser that does not support "rewinding", i.e. you cannot re-read bytes that the decoder already read. To avoid this you can use ioutil.ReadAll to read the full contents of the body and then json.Unmarshal to do the decoding.
The 2nd type of error would be the json.UnmarshalTypeError. If you look at the documentation of the error type and its fields you'll know that all you need to do is to type assert the returned value and you're done. Example
Validation against "blank" and "silly" values would be done after the json has been successfully decoded into your structure. How you do that is up to you. For example you could use a 3rd party package that's designed for validating structs, or you can implement an in-house solution yourself, etc. I actually don't have an opinion on which one of them is the "best" so I can't help you with that.
What I can say is that the most basic approach would be to simply look at each field of the structure and check if its value is valid or not according to the requirements for that field.
After using the protogen tool, I have a message type for sending messages:
type File struct {
Info string `protobuf:"bytes,1,opt,name=info,json=info" json:"info,omitempty"`
BytesValues []byte `protobuf:"bytes,2,opt,name=bytes_values,json=bytesValues,proto3" json:"bytes_values,omitempty"`
}
I am trying to send some binary data using the BytesValues field like so:
filePath := filepath.Join("test", "myfile.bin")
f, _ := ioutil.ReadFile(filePath) // error return value ignored for brevity
msg := File{BytesValues: f}
body, _ := proto.Marshal(msg) // encode
The server seems to have problems decoding the message I am sending to it. Is this the correct way to send binary data using a []byte field with protocol buffers?
In my case, the problem was actually the server not reading the raw bytes from the correct field.
The correct way to send raw bytes is to just set the bytes to the field. There is no need to encode the bytes in any way because protocol buffers is a binary format.
filePath := filepath.Join("test", "myfile.bin")
f, _ := ioutil.ReadFile(filePath) // error return value ignored for brevity
msg := File{BytesValues: f}
body, _ := proto.Marshal(msg) // encode
I'm running http request in golang
resp, err := client.Do(req)
if err != nil {
return "", err
}
So, it returns error back to the main function, that attempts to store it in the database:
_, err = db.Exec("UPDATE test SET error = $1 WHERE id = $2", error, id)
I receive the following error: sql: converting Exec argument #1's type: unsupported type errors.errorString, a struct exit status 1
So, I understand, that error has a different type, but I can't find information on how to pass the value of the error to the string. Could someone direct me in a right way.
Use the function:
error.Error()
to get a string representation of the error.
Tip: avoid naming variables with existing type names. error is a type name and it's also your variable name, which might lead to confusion.