I am facing with the problem that protobuf I defined enum but its value is int32
Now I want someway or somehow to change all the protobuf defined to string
Or any code-hack for doing it in gateway without changing the protobuf.
Enum defined
enum TimeUnit {
seconds = 0;
minutes = 1;
hours = 2;
days = 3;
months = 4;
}
message CacheDuration {
uint32 Value = 1;
TimeUnit Units = 2;
}
What i got from generated code now is
And it is the return value for front end to use. So they would see the value of Units = int32 like this:
The services communicate by generated struct protobuf.
I want to make it change to
"Units":"days"
Thanks
You can use String method in your go code:
generatedTimeUnitEnum.String() // output: days
Related
I'm designing a protobuf to represent an event, where each event can hold extra fields.
There are a lot of possible extra fields (~100), but only a small portion of them will be used in each message (~3)
Each extra field will be used only once, but multiple of them can exist, therefore I would like to have a concept of an anyof message type, but unfortunately, there is no such thing in protobuf.
So to try and mock this behavior, and as mentioned in this discussion I thought I can put all my extra fields in a oneof, wrap it with a message, and use this message as repeated in my event:
message ExtraField {
oneof extra_field_value {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
repeated ExtraField extra_fields = 8;
}
Even though this solution is more explicit for my understanding, it isn't the most memory effective, and the repeated message with oneof implementation allows to add the same extra field more than once (unwanted behavior)
I can also just write all the extra fields as-is in an inner message, but most of them will be empty all the time
message ExtraFields {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
extraFields extra_fields = 8;
}
If I understand correctly, using empty fields in a message isn't going to make my serialized data larger, and therefore the second protobuf design is the preferred practice
Am I correct?
Is there another protobuf design for my needs?
In the following example:
try (ParquetWriter<Example> writer =
new ProtoParquetWriter<>(
new Path("file:/tmp/foo.parquet"),
Example.class,
SNAPPY,
DEFAULT_BLOCK_SIZE,
DEFAULT_PAGE_SIZE)) {
writer.write(
Example.newBuilder()
.setTs(System.currentTimeMillis())
.setTenantId("tenant")
.setSomeFlag(false)
.setSomeInt(1)
.setOtherInt(0)
.build());
}
}
And example .proto file:
syntax = "proto3";
package com.example;
message Example {
uint64 ts = 1;
string tenantId = 2;
bool someFlag = 3;
int32 someInt = 4;
int32 otherInt = 2;
}
The resulting parquet file won't have the fields someFlag and otherInt because they are false and 0 respectively.
Is there a way to make it write it anyway or should I handle this on the reader side?
In proto3, presence tracking was not enabled historically, and the only presence rule was around zero defaults. Fortunately this changed recently in new versions of protoc. The optional keyword can now be used in from of fields in proto3 to enable this. So: add optional, and any compliant implementation should do what you want. The defaults are still zero/false/etc, but if they are explicitly set: they are serialized.
syntax = "proto3";
package com.example;
message Example {
optional uint64 ts = 1;
optional string tenantId = 2;
optional bool someFlag = 3;
optional int32 someInt = 4;
optional int32 otherInt = 2; // [sic]
}
Also, the second 2 should be a 5
I want th serialize int/int64/double/float/uint32/uint64 into protobuf, which one should I use ? which one is more effective ?
For example :
message Test {
google.protobuf.Any any = 1; // solution 1
google.protobuf.Value value = 2; // solution 2
};
message Test { // solution 3
oneof Data {
uint32 int_value = 1;
double double_value = 2;
bytes string_value = 3;
...
};
};
In your case, you'd better use oneof.
You can not pack from or unpack to a built-in type, e.g. double, int32, int64, to google.protobuf.Any. Instead, you can only pack from or unpack to a message, i.e. a class derived from google::protobuf::Message.
google.protobuf.Value, in fact, is a wrapper on oneof:
message Value {
// The kind of value.
oneof kind {
// Represents a null value.
NullValue null_value = 1;
// Represents a double value.
double number_value = 2;
// Represents a string value.
string string_value = 3;
// Represents a boolean value.
bool bool_value = 4;
// Represents a structured value.
Struct struct_value = 5;
// Represents a repeated `Value`.
ListValue list_value = 6;
}
}
Also from the definition of google.protobuf.Value, you can see, that there's no int32, int64, or unint64 fields, but only a double field. IMHO (correct me, if I'm wrong), you might lose precision if the the integer is very large. Normally, google.protobuf.Value is used with google.protobuf.Struct. Check google/protobuf/struct.proto for detail.
I have a protobuf schema with a bunch of repeated structures. Something like
syntax = "proto3";
package My
message TopLevel
{
string swVersion = 3;
string reportMac = 4;
string reportSsid = 6
}
message Temperature
{
required uint64 ts = 1;
required uint32 source = 3;
repeated sint32 readings = 4;
}
message MyMessage
{
required TopLevel topLevel = 1;
repeated Temperature temperature = 2;
}
I compile with pbtools and get the structures and functions for Temperature and readings. However I am having a hard time figuring out how to add "Temperature" entries dynamically.
Or am I out of luck and pbtools requires telling it ahead of time how many entries I have. One problem is data is encoded as it is generated and I do not know how many of what I will have for each report.
I attached the generated code.
pbtools requires the length before adding any items.
File my_protocol.proto:
Message MyProtocol {
required int32 protocolId = 1;
required **unknownType** protocolBody = 2;
}
I want to parse protocolBody according to protocolId.
For example,
protocolId = 10001 represents a login event, then the protocolBody should be a login message and can be parsed by login.proto.
I don't know which type should be set on the unknownType.
Protocol buffers parses the whole message in the one go.
You could have a message
Message MyProtocolGetId {
required int32 protocolId = 1;
}
Parse that then reparse with the correct message id. Alternatively the ProtocolBody could be stored as bytes and passed seperately
You can fake it, but it will be awkward. You have to use optional fields, and indicate which one if present using a flag. Make sure to double check that the field is set in your receiver code:
Message OneOfManyTypes {
required int32 kind = 1;
optional MessageType1 ty1 = 2;
optional MessageType2 ty2 = 3;
...
}