Background
I am trying to use protobuff for one of our apps, but I am having trouble understanding the protocol and I need help creating a .proto file.
Data
The data I need to encode is a list of maps, with the following structure:
[
{
"AwayTeam": "Osasuna",
"Date": "2017-05-07",
"Div": "SP1",
"FTAG": 1,
"FTHG": 4,
"FTR": "H",
"HTAG": 0,
"HTHG": 2,
"HTR": "H",
"HomeTeam": "Valencia",
"Season": 201617
},
{
"AwayTeam": "Osasuna",
"Date": "2016-02-27",
"Div": "SP2",
"FTAG": 1,
"FTHG": 0,
"FTR": "A",
"HTAG": 0,
"HTHG": 0,
"HTR": "D",
"HomeTeam": "Cordoba",
"Season": 201516
}
]
Each map has the following structure:
{
"AwayTeam": string, required: true
"Date": string, required: true
"Div": string, required: true
"FTAG": integer, required: true
"FTHG": integer, required: true
"FTR": string, required: true
"HTAG": integer, required: true
"HTHG": integer, required: true
"HTR": string, required: true
"HomeTeam": string, required: true
"Season": integer, required: true
}
Research
My goal is to create .proto file using proto3. So I decided to read the documentation for .proto3 files:
https://developers.google.com/protocol-buffers/docs/proto3#maps
But I was even more confused. According to the docs, I cannot have a map holding values of different types:
https://developers.google.com/protocol-buffers/docs/proto3#maps
For that I would need the equivalent of the JSON object type and check the docs for .struct.proto but that page doesn't mention anything about it.
Question
So I am rather lost here. How do I represent the mentioned data structure in a .proto?
Answer
Turns out that I don't actually need a map, a list of objects (messages) would suffice:
syntax = "proto3";
message Result {
string AwayTeam = 1;
string Date = 2;
string Div = 3;
int32 FTAG = 4;
int32 FTHG = 5;
string FTR = 6;
int32 HTAG = 7;
int32 HTHG = 8;
string HTR = 9;
string HomeTeam = 10;
int32 Season = 11;
}
message Response {
repeated Result results = 1;
}
Related
Say I'm modeling a Business resource in Protobuf. It represents any generic kind of "Business" - a bakery, a restaurant, a bank, etc...
Concept, in JSON
In JSON I might model the structure as follows:
{
id: "...",
name: "...",
address: "...",
properties: {
type: "...",
...
}
The properties attribute is a different structure for each type of business.
For a bakery it might be:
properties: {
type: 'Bakery',
serves_coffee: true,
gluten_free: true
}
For a bank it might be:
properties: {
type: "Bank",
max_denomination_offered: 100000,
}
Protobuf
The main Business can be modeled as follows in Protobuf:
message Business {
string id = 1;
string name = 2;
string address = 3;
??? properties = 4
}
How do I model a generic Properties message that can have varying shape?
Is there some sort of "generic" object I can use? Or do I have to create BankProperties, BakeryProperties, etc... and add them in as:
Google.Protobuf.WellKnownTypes.Any properties = 4;
Is there an established or well-accepted pattern for this sort of thing?
Thanks!
What will your receiving code do? If it only knows about specific kinds of Business anyway, it would make sense to just list them in the .proto:
message BakeryProperties {
bool serves_coffee = 1;
bool serves_cake = 2;
}
message BankProperties {
int32 max_denomination = 1;
}
message Business {
string id = 1;
string name = 2;
string address = 3;
oneof details {
BakeryProperties bakery = 100;
BankProperties bank = 101;
}
}
That way you get the actual benefits of a schema: any code that knows how do deal with bakeries, deals with them in the same way. If the code doesn't know about the particular type, it can still use the common properties.
But if you are just going to e.g. display the properties, maybe what you want is just a generic key-value mapping:
map<string,string> properties = 10;
or
message Property {
string name = 1;
oneof {
string strvalue = 2;
int32 intvalue = 3;
}
}
repeated Property properties = 10;
I have data
{
"id": 1000,
"price": "99,01USA",
},
{
"id": 1001,
"price": "100USA",
},
{
"id": 1002,
"price": "780USA",
},
{
"id": 1003,
"price": "20USA",
},
How I sort order by price (ASC , DESC)
You can alter it a little to parse price to integer and then sort it
You can create a dynamic sort function that sorts objects by their value that you pass:
function dynamicSort(property) {
var sortOrder = 1;
if(property[0] === "-") {
sortOrder = -1;
property = property.substr(1);
}
return function (a,b) {
/* next line works with strings and numbers,
* and you may want to customize it to your needs
*/
var result = (a[property] < b[property]) ? -1 : (a[property] > b[property]) ? 1 : 0;
return result * sortOrder;
}
}
So you can have an array of objects like this:
var People = [
{Name: "Name", Surname: "Surname"},
{Name:"AAA", Surname:"ZZZ"},
{Name: "Name", Surname: "AAA"}
];
...and it will work when you do:
People.sort(dynamicSort("Name"));
People.sort(dynamicSort("Surname"));
People.sort(dynamicSort("-Surname"));
Actually this already answers the question. Below part is written because many people contacted me, complaining that it doesn't work with multiple parameters.
Multiple Parameters
You can use the function below to generate sort functions with multiple sort parameters.
function dynamicSortMultiple() {
/*
* save the arguments object as it will be overwritten
* note that arguments object is an array-like object
* consisting of the names of the properties to sort by
*/
var props = arguments;
return function (obj1, obj2) {
var i = 0, result = 0, numberOfProperties = props.length;
/* try getting a different result from 0 (equal)
* as long as we have extra properties to compare
*/
while(result === 0 && i < numberOfProperties) {
result = dynamicSort(props[i])(obj1, obj2);
i++;
}
return result;
}
}
Which would enable you to do something like this:
People.sort(dynamicSortMultiple("Name", "-Surname"));
Subclassing Array
For the lucky among us who can use ES6, which allows extending the native objects:
class MyArray extends Array {
sortBy(...args) {
return this.sort(dynamicSortMultiple(...args));
}
}
That would enable this:
MyArray.from(People).sortBy("Name", "-Surname");
I have proto file like this:
message StartAssignmentRequest {
string additional_comment = 3;
repeated RideSlip slips = 4;
}
message RideSlip{
string slip_name = 2;
string slip_ext = 3;
string slip_link = 4;
}
Now I want to create its request and I am doing something like this:
req := &api.StartAssignmentRequest{
AdditionalComment:"AdditionalComment",
Slips: &api.RideSlip[],
}
but not having idea how I can send RideSlip data properly.
Protobuffer (both 2 and 3) repeated fields are compiled to slices in Go.
Just append to it:
req := &api.StartAssignmentRequest{
AdditionalComment: "AdditionalComment",
}
req.Slips = append(req.Slips, &api.RideSlip{
SlipName: "foo",
SlipExt: "bar",
SlipLink: "https://stackoverflow.com",
})
Or assign to it a literal value:
req := &api.StartAssignmentRequest{
AdditionalComment: "AdditionalComment",
Slips: []*api.RideSlip{
{
SlipName: "foo",
SlipExt: "bar",
SlipLink: "https://stackoverflow.com",
},
},
}
I have this JSON
{
id: 142,
fields: [
{ fieldId: 50, value: 0 },
{ fieldId: 51, value: 0 },
{ fieldId: 52, value: 0 }, // <--- Notice the comma
]
}
As you can see, there is a comma after the last item.
When I parse with GSON to these objects:
class Foo {
public int id;
public List<Field> fields;
}
class Field {
public int fieldId;
public int value;
}
using this code:
Gson gson = new Gson();
Foo foo = gson.fromJson(json, Foo.class);
I get a foo object containing 4 items in the fields array.
Is this a problem with GSON or my JSON is not a correctly formatted JSON? I thought that this last comma in JavaScript was permitted...
From RFC 8259 (Note that I am not sure if this is the latest RFC for JSON):
5. Arrays
An array structure is represented as square brackets surrounding
zero or more values (or elements). Elements are separated by
commas.
array = begin-array [ value *( value-separator value ) ] end-array
There is no requirement that the values in an array be of the same
type.
Now GSON seems to do something like interpreting the last sentence so that null is a value. So after your last comma there is a null value.
You can also test what happens if you deserialize the following:
{
id: 142,
fields: [
{ fieldId: 50, value: 0 },,,,,,
{ fieldId: 51, value: 0 },
]
}
As you might guess there will be as many null objects as there are extra commas.
I would not say that this behavior is a problem nor can I say that it is against RFC.
I have currently run into an issue while toying with Lua and the alien module to use Win32 API and such from Lua scripts. So far I have only had a single issue with alien which is with the use of API that use certain structures such as CreateFontIndirect.
For example:
HFONT CreateFontIndirectA( const LOGFONT& lplf );
LOGFONT:
typedef struct tagLOGFONT {
LONG lfHeight;
LONG lfWidth;
LONG lfEscapement;
LONG lfOrientation;
LONG lfWeight;
BYTE lfItalic;
BYTE lfUnderline;
BYTE lfStrikeOut;
BYTE lfCharSet;
BYTE lfOutPrecision;
BYTE lfClipPrecision;
BYTE lfQuality;
BYTE lfPitchAndFamily;
TCHAR lfFaceName[LF_FACESIZE];
}LOGFONT, *PLOGFONT;
The issue lies with the font face name. I cannot get Lua to keep a string inside the structure itself, it always pushes a pointer into the structure. So there is no way, that I can figure out, to be able to use this API purely from Lua.
This is what I got working to a point:
LOGFONT = alien.defstruct {
{ 'lfHeight', 'long' },
{ 'lfWidth', 'long' },
{ 'lfEscapement', 'long' },
{ 'lfOrientation', 'long' },
{ 'lfWeight', 'long' },
{ 'lfItalic', 'byte' },
{ 'lfUnderline', 'byte' },
{ 'lfStrikeOut', 'byte' },
{ 'lfCharSet', 'byte' },
{ 'lfOutPrecision', 'byte' },
{ 'lfClipPrecision', 'byte' },
{ 'lfQuality', 'byte' },
{ 'lfPitchAndFamily', 'byte' },
{ 'lfFaceName', 'string' } -- This line isn't working properly.
}
gdi32 = alien.load( "gdi32.dll" )
gdi32.CreateFontIndirectA:types {
ret = 'long',
abi = 'stdcall',
'pointer'
}
An example to call it:
local lf = LOGFONT:new()
lf.lfHeight = 14
lf.lfWidth = 0
lf.lfEscapement = 0
lf.lfOrientation = 0
lf.lfWeight = 400
lf.lfItalic = 0
lf.lfUnderline = 0
lf.lfStrikeOut = 0
lf.lfCharSet = 0
lf.lfOutPrecision = 0
lf.lfClipPrecision = 0
lf.lfQuality = 0
lf.lfPitchAndFamily = 0
lf.lfFaceName = 'Terminal'
local hFont = gdi32.CreateFontIndirectA( lf() )
Debugging my application that runs my script shows that the api is being called properly, everything is passed properly except the font face. I've tried various different methods to get it working but I cant get it to go as needed.
Any tips on fixing this without hard-coding anything else into the exe?
Alien's string type is for pointers to strings only, that is why your example is not working. Try this:
-- LF_FACESIZE = ? -- put the value of the LF_FACESIZE constant here
LOGFONT = alien.defstruct {
{ 'lfHeight', 'long' },
{ 'lfWidth', 'long' },
{ 'lfEscapement', 'long' },
{ 'lfOrientation', 'long' },
{ 'lfWeight', 'long' },
{ 'lfItalic', 'byte' },
{ 'lfUnderline', 'byte' },
{ 'lfStrikeOut', 'byte' },
{ 'lfCharSet', 'byte' },
{ 'lfOutPrecision', 'byte' },
{ 'lfClipPrecision', 'byte' },
{ 'lfQuality', 'byte' },
{ 'lfPitchAndFamily', 'byte' },
{ 'lfFaceName', 'char' }
}
LOGFONT.size = LOGFONT.size + LF_FACESIZE - 1 -- so Alien allocates enough space
-- for the whole array
function get_lfname(lf) -- gets the lfFaceName field as a Lua string
local out = {}
local offset = LOGFONT.offsets.lfFaceName
local buf = lf()
for i = offset, offset+LF_FACESIZE-1 do
local c = buf:get(i, "char")
if c ~= 0 then
out[#out+1] = string.char(c)
else
break
end
end
return table.concat(out)
end
function set_lfname(lf, s) -- sets the Lua string s as the lfFaceName
local offset = LOGFONT.offsets.lfFaceName
local buf = lf()
for i = 1, LF_FACESIZE do
if i <= #s then
buf:set(offset+i, string.byte(string.sub(s, i, i)), "char")
else
buf:set(offset+i, 0, "char")
break
end
end
end
Now just allocate an LOFGONF structure as usual, but use the get_lfname and set_lfname functions to work with the lfFaceName attribute:
local lf = LOGFONT:new()
lf.lfHeight = 14
lf.lfWidth = 0
lf.lfEscapement = 0
lf.lfOrientation = 0
lf.lfWeight = 400
lf.lfItalic = 0
lf.lfUnderline = 0
lf.lfStrikeOut = 0
lf.lfCharSet = 0
lf.lfOutPrecision = 0
lf.lfClipPrecision = 0
lf.lfQuality = 0
lf.lfPitchAndFamily = 0
set_lfname(lf, 'Terminal')
local hFont = gdi32.CreateFontIndirectA( lf() )
Tacking an array at the end is a common pattern for structures in C programming that I forgot. I am going to put direct support for it in the next version of Alien.
Thanks much for the response mascarenhas, this solution worked. I did have to adjust your set_lfname function though, as the offset+i-1 was misaligned and was overwriting the lfPitchAndFamily byte in the structure, removed the -1 and it works great. :)