What's the best way to convert this json object to protobuf?
JSON:
{
"name": "test",
"_list": {
"some1": { "value": 1 },
"some2": [
{ "value": 2 },
{ "value": 3 },
]
}
}
Proto:
message Something {
string name = 1;
message ListType {
repeated string = 1;
}
map<string, ListType> _list = 2;
}
Without having the _list in the message I would use jsonpb.Unmarsal, but I can't think of a way to define the Unmarshaler interface on a type that is generated in a diff package.
I also thought of having _list as a Any (json.RawMessage) and handle it after the Unmarshal (but can't make this to work; err message: Any JSON doesn't have '#type')
With _list being inconsistent (not just a list of strings/map of values/etc) and you mentioning you looked into using Any you could consider making your message:
message Something {
string name = 1;
google.protobuf.Struct _list = 2;
}
https://github.com/golang/protobuf/blob/master/ptypes/struct/struct.proto
With that you can marshal/unmarshal json to/from proto messages using github.com/golang/protobuf/jsonpb which is actually designed for use with the grpc gateway but you can use it too
Related
I'm fairly new to protocol buffers but have been trying to learn them as a means of sending data via MQTT. So far, I've been fine with creating proto messages and compiling them for python runtime, until I started to notice incompatibility between versions of my protobufs.
When I add a message type (no changes to existing messages/fields) to my server-side proto definitions without updating my client side proto definitions, decoding messages sent to the server give me non-deterministic results.
Here is an example of what I'm talking about:
Client proto:
message Wrapper {
optional uint32 id = 1;
optional string name = 2;
oneof payload {
Event event = 3;
Command command = 4;
}
}
message Event {
uint32 event_id = 1;
oneof event_payload {
LoginEvent login_event = 2;
LogoffEvent logoff_event = 3;
}
}
Server Proto:
message Wrapper {
optional uint32 id = 1;
optional string name = 2;
oneof payload {
Event event = 3;
Command command = 4;
}
message Event {
uint32 event_id = 1;
oneof event_payload {
LoginEvent login_event = 2;
LogoffEvent logoff_event = 3;
NewUserEvent new_user_event = 4;
}
}
I will encode and send a message from the client:
message Wrapper {
id = 12345;
name = John;
event = {
login_event = ...
}
}
And will decode the message on the server and get:
message Wrapper {
id = 12345;
name = John;
event = {
logoff_event = ...
}
}
NOTE: The decoded message type isn't deterministic and changes between messages
Can someone explain why adding an event type seems to screw up decode? Or any best-practices I should obey to improve version compatibility? Thanks in advance!
This may be a side-effect of the Python implementation rather than a bug.
You don't include the code you're using to (un)marshal the protobufs.
In your example, does logoff_event contain a LogoffEvent message type?
You say non-deterministic? Do you see different event_payload messages types on the server for the same message sent by the client?
What is returned by:
msg = Wrapper() # Your decoded message
assert msg.event.WhichOneof("event_payload")
# or
assert msg.event.HasField("login_event")
See Python Generated Code: OneOf
Update 210927
I'm unable to repro the behavior you observe; it works as expected for me.
Client:
import client_pb2
msg = client_pb2.Wrapper()
msg.id=0
msg.name="Test"
msg.event.event_id=1
msg.event.login_event.y="Hello Freddie"
print(msg)
f=open("message","wb")
f.write(msg.SerializeToString())
f.close()
Yields:
id: 0
name: "Test"
event {
event_id: 1
login_event {
y: "Hello Freddie"
}
}
Server:
import server_pb2
msg = server_pb2.Wrapper()
f=open("message","rb")
msg.ParseFromString(f.read())
f.close
assert msg.event.WhichOneof("event_payload")
assert msg.event.HasField("login_event")
print(msg)
Yields:
id: 0
name: "Test"
event {
event_id: 1
login_event {
y: "Hello Freddie"
}
}
So I have an Struct that holds data that has a AddedByUser which links to my User Struct.
What I want to be able to do it remove the UserLevel from the AddedByUser
Now I want to be able to do it from this function only, so using the json:"-" is not an option. That would remove it from all json output. I only want to remove it form this one function.
I should also say that these are Gorm models and when I have been trying to remove the 10 option (UserLevels) it only removes the outer data set not the UserLevel from all of the data.
{
"ID": 1,
"CreatedAt": "2019-01-08T16:33:09.514711Z",
"UpdatedAt": "2019-01-08T16:33:09.514711Z",
"DeletedAt": null,
"UUID": "00000000-0000-0000-0000-000000000000",
"Title": "title000",
"Information": "info999",
"EventDate": "2006-01-02T15:04:05Z",
"AddedByUser": {
"ID": 2,
"CreatedAt": "2019-01-08T15:27:52.435397Z",
"UpdatedAt": "2019-01-08T15:27:52.435397Z",
"DeletedAt": null,
"UUID": "b019df80-a7e4-4397-814a-795e7e84b4ca",
"Firstname": "Me",
"Surname": "admin",
"Password": "....",
"Email": "admin#email.co.uk",
"UserLevel": {
"ID": 0,
"CreatedAt": "0001-01-01T00:00:00Z",
"UpdatedAt": "0001-01-01T00:00:00Z",
"DeletedAt": null,
"LevelTitle": "",
"UserLevel": null
},
So this is what I have tried,
data := []models.MyData{}
data = append(data[0:2])
I have about 14 results, with out the append it loads all the results but with this is only loads two results. The idea was to remove either UpdateAt or Title. As I am not sure if the gorm model information is all 0 or if the slice sees them as 0,1,2,3,4 etc.
I have also tried to range over the slice of models, while I can access each of the sections, I can not seem to find a simple method to remove data by name from a struct? Maps seem to have that but not structs which I am not sure why?
Thanks.
UPDATE
This is the model I am using:
//Model
type MyData struct {
gorm.Model
UUID uuid.UUID
Title string
Information string
EventDate time.Time
AddedByUser Users `gorm:"ForeignKey:added_by_user_fk"`
AddedByUserFK uint
}
//Users Model
type Users struct {
gorm.Model
UUID uuid.UUID
Firstname string
Surname string
Password string
Email string
UserLevel UserLevels `gorm:"ForeignKey:user_level_fk" json:",omitempty"`
UserLevelFK uint
}
As mentioned in the comments, you cannot remove fields from a struct value, because that would yield a value of a different type.
However, you can set fields to their zero value. Combined with the omitempty JSON tag, you can exclude fields from the JSON encoding. To make this work properly, you have to change the UserLevel field to a pointer type (otherwise you end up with empty objects in the JSON document).
Types shortened for brevity:
package main
import (
"encoding/json"
"fmt"
)
type MyData struct {
Title string
AddedByUser Users
}
type Users struct {
ID int
UserLevel *UserLevels `json:",omitempty"` // pointer type with omitempty
}
type UserLevels struct {
LevelTitle string
}
func main() {
var x MyData
x.Title = "foo"
x.AddedByUser.ID = 2
x.AddedByUser.UserLevel = &UserLevels{}
f(x)
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("main:\n" + string(b))
}
func f(x MyData) {
// "unset" UserLevel. Since we are receiving a copy of MyData, this is
// invisible to the caller.
x.AddedByUser.UserLevel = nil
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("f:\n" + string(b))
}
// Output:
// f:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2
// }
// }
// main:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2,
// "UserLevel": {
// "LevelTitle": ""
// }
// }
// }
Try it on the playground: https://play.golang.org/p/trUgnYamVOA
Alternatively, you can define new types that exclude the AddedByUser field. However, since this field isn't at the top level, this is a lot of work, and it's easy to forget to update those types when new fields are added to the original types.
If the field were at the top level, the compiler would do most of the work for you, because types that only differ in their field tags can be directly converted to one another:
type MyData struct {
ID int
Title string
}
func main() {
var x MyData
x.ID = 1
x.Title = "foo"
f(x)
}
func f(x MyData) {
type data struct { // same as MyData, except the field tags
ID int
Title string `json:"-"`
}
b, _ := json.MarshalIndent(data(x), "", " ")
fmt.Println("main:\n" + string(b))
}
I have a json file (nested json) that I am unmarshalling its content into a map[string]interface. Now I have to implement pagination as the data is large. The client side will send as a query parameter the desired page, how can I slice the data I have?
This is a snippet of the data I am dealing with:
"packages":{
"pkg1": {
"meta": {
"description": "description1",
"name": "pkg1.1"
},
"name": "pkg1.1"
},
"pkg2": {
"meta": {
"description": "description2",
"name": "pkg2.2"
},
"name": "pkg2.2"
},
}
So what I did is that I recursively iterated through the data and created an array of a custom type containing the data I need (name, description) for each entry so that I can use it for pagination. Here is the code I used:
type Object struct {
name string
description string
}
func iterate(aMap map[string]interface{}, result *[]Object){
for key, val := range aMap {
switch val.(type) {
case map[string]interface{}:
if(key == "meta"){
switch reflect.TypeOf(val).Kind() {
case reflect.Map:
s := reflect.ValueOf(val)
var tmpData Object
if(s.MapIndex(reflect.ValueOf("name")).IsValid()){
tmpData.name = s.MapIndex(reflect.ValueOf("name")).Interface().(string)
}
if(s.MapIndex(reflect.ValueOf("description")).IsValid()){
tmpData.description = s.MapIndex(reflect.ValueOf("description")).Interface().(string)
}
*result = append(*result, tmpData)
}
}
iterate(val.(map[string]interface{}), result)
default: //DO NOTHING!!
}
}
}
If you're doing pagination, somewhere the data must be represented as a list instead of an object? I assume at some place in your JSON, you have a list of items, otherwise pagination doesn't make sense.
It shouldn't be very hard, something simple like this should work:
const (
itemsPerPage = 10
)
var data []map[string]interface{}
// pages start at 1, can't be 0 or less.
func GetDataPage(page int) []map[string]interface{} {
start := (page - 1) * itemsPerPage
stop := start + itemsPerPage
if start > len(data) {
return nil
}
if stop > len(data) {
stop = len(data)
}
return data[start:stop]
}
You are unmarshalling your json into a map which has no order by itself. In order to be able to paginate your results you need to order them in some way.
One way of doing it is to sort your data and then store it into an array. But in order to paginate you need to have ordered data and that is not possible with a map.
My proto is something like:
message PlatformConfig {
google.protobuf.Any source_adapter_config = 1;
};
and I want to set proto:
message SessionBundleSourceAdapterConfig {
SessionBundleConfig config = 1;
}
into source_adapter_config, How to write the this TextFormat of protobuf.
Finally, I found that Any type need a type_url to specify type, for example:
platform_configs {
key: "tensorflow"
value {
source_adapter_config {
type_url:"type.googleapis.com/tensorflow.serving.SavedModelBundleSourceAdapterConfig"
value: "..."
}
}
}
I have a simple object defined:
type Link struct {
Href string `json:"href"`
Title string `json:"href,omitempty"`
}
type Foo struct {
Links map[string]Link `json:"_links"`
}
foo := new(Foo)
foo.Links = make(map[string]Link, 0)
foo.Links["self"] = Link{Href: "/href"}
After marshalling it to JSON, I'd expect:
{
"_links": {
"self": {
"href": "/href"
}
}
}
But instead I get:
{
"_links": {
"self": {}
}
}
Any idea why? Here's a full example:
https://play.golang.org/p/3RA3Mrx3pt
You've defined json:"href" twice:
type Link struct {
Href string `json:"href"`
Title string `json:"href,omitempty"`
}
After changing the second to json:"title" it works: https://play.golang.org/p/uEbyqtHYF8.