I'm working with some protocol buffers (java) that have extensions. I'm seemingly able to parse the serialized protocol buffers ok (no errors anyway), but for debugging purposes (yet another issue), I'm printing them out to a log.
I'm getting these types of things scattered around the log:
data_config {
format: FORMAT_DELIMITED
1024: "\022\001\n"
}
And here's the message definition:
message DataConfig {
optional DataFormat format = 1;
extensions 1024 to max;
option (dwhio.data.message_reflection_config) = { reflect_extensions: true };
}
My question is 'Is the debug string in the log with '1024' correct (expected) or indicative of a class loading or other problem?'
I haven't figured out a way to print to a string involving a registry, simply 'merge', so I'm assuming that's not necessary?
The problem is probably that you did not provide an ExtensionRegistry when you parsed the message from binary. So, the extension was treated as an unknown field. When you later print the message, the extension is still unknown, so is printed as you see. The solution is to provide the registry at parse time, e.g. DataConfig.parseFrom(bytes, registry).
Related
I'm creating client side streaming method which proto file definition should be looking similar to this:
service DataService {
rpc Send(stream SendRequest) returns (SendResponse) {}
}
message SendRequest {
string id = 1;
bytes data = 2;
}
message SendResponse {
}
The problem here is that ID is sent with each streaming message even it is needed only once. What's your recommendation and most optimal way for such a use cases?
One hacky approach would be to set ID only once within first message and after that left if blank. But this API is supposed to be used by third party and method definition like above doesn't explaining that well.
I don't think something like this is supported either:
service DataService {
rpc Send(InitialSendRequest, stream DataOnlyRequest) returns (SendResponse) {}
}
I'm currently considering SendRequest message to be something like this, but will have to check how optimal is this compared to the first case considering proto marshaling:
message SendRequest {
oneof request{
string id = 1;
bytes data = 2;
}
}
Your approach of using a oneof with the fields clearly documented saying id is expected only in the first message on the stream and that server implementations will terminate the stream if id is set on subsequent messages on the stream sounds good to me.
The following is a usage of the above described pattern in grpc-lb-v1. Even though the grpc team is moving away from grpc-lb-v1, the above mentioned pattern is a commonly used one.
I'm not very sure about its implications with respect to proto marshaling. That might be a question for the protobuf team.
Hope this helps.
We are using Golang and .NET Core for our inter-communication microservices infrastructure.
All the data across the services are coming based on Protobuffs Protocols that we have created.
Here is an example of one of our Protobuffs:
syntax = "proto3";
package Protos;
option csharp_namespace = "Protos";
option go_package="Protos";
message EventMessage {
string actionType = 1;
string payload = 2;
bool auditIsActive = 3;
}
Golang is working well and the service is generating the content as needed and sending it to the SQS queue, once that happens the .NET core service is getting the data and trying to serialize it.
Here are the contents of the SQS message example:
{"#type":"type.googleapis.com/Protos.EventMessage","actionType":"PushPayload","payload":"<<INTERNAL>>"}
But we are getting an Exception that saying the wire-type is not defined as mentioned below:
Google.Protobuf.InvalidProtocolBufferException: Protocol message contained a tag with an invalid wire type.
at Google.Protobuf.UnknownFieldSet.MergeFieldFrom(CodedInputStream input)
at Google.Protobuf.UnknownFieldSet.MergeFieldFrom(CodedInputStream input)
at Google.Protobuf.UnknownFieldSet.MergeFieldFrom(UnknownFieldSet unknownFields, CodedInputStream input)
at Protos.EventMessage.MergeFrom(CodedInputStream input) in /Users/maordavidzon/projects/github_connector/GithubConnector/GithubConnector/obj/Debug/netcoreapp3.0/EventMessage.cs:line 232
at Google.Protobuf.MessageExtensions.MergeFrom(IMessage message, Byte[] data, Boolean discardUnknownFields, ExtensionRegistry registry)
at Google.Protobuf.MessageParser`1.ParseFrom(Byte[] data)
The Proto file is exactly the same in both of the services.
Is there any potential missing options or property that we need to add?
It looks like you're using the JSON format rather than the binary format. In that case, you want ParseJson(string json), not ParseFrom(byte[] data).
Note: the binary format is more efficient, if that matters to you. It also has better support across protobuf libraries / tools.
Basically there are two possible scenarios, or your protos files generated for .NET and GoLang are not in the same version or your data has been corrupted while transferring between GoLang and .NET application.
Protobuf is a binary protocol, check if you have any http filter or anything else that can change incoming or outgoing stream of bytes.
I would like to send a stream of different protobuf messages through the wires and be able to differentiate them at arrival as they are coming.
Let's say I have a *.proto like that:
message Book {
//...
}
message BlueRay{
//...
}
And then and the sender side, I serialize let's say this sequence (pseudo code in C#):
Book1.WriteDelimitedTo(myStream);
BlueRay1.WriteDelimitedTo(myStream);
Book2.WriteDelimitedTo(myStream);
How can I do to know the order/types of messages I'm getting on the receiver side? (The contract is available on both sender and receiver side of course)
Depending of my sender's state I can not presume/tell what is going to be sent and in which order...
I understood that there is no built-in way to do that like stated in the documentation, but for instance for the size of the message there was a helper (C# API helper WriteDelimited method to embedd size).
How can I do to get/map the type of a received message?
My server will be written in a given language (C# actually), but my clients should be "implementable" in any protobuf supported target, so I don not want to set up something that would serialize C#/CLR specific stuff in between...
I'm maybe using protobuf in a weird way? I'm trying to set up a kind of protocol.
I think I finally found out how to do that (only in C# at the moment).
Basically, I'm writing the following to the stream:
the coming message descriptor's target name (in proto file, as both end share this definition), using the primitive for string serilization
the coming message size, using the primitive for size serialization
the serialized message into the stream
This results in a method like the following:
public static void WriteToStream(Stream outputStream, IMessage message)
{
MessageDescriptor stateMsgDescriptor = message.Descriptor;
using (CodedOutputStream codedOutStr = new CodedOutputStream(outputStream, true))
{
codedOutStr.WriteString(stateMsgDescriptor.FullName);
int size = message.CalculateSize();
codedOutStr.WriteLength(size);
message.WriteTo(codedOutStr);
codedOutStr.Flush();
}
}
As stated here,
The Protocol Buffer wire format is not self-delimiting, so protocol
buffer parsers cannot determine where a message ends on their own
I am debugging though a wcf service. I have a service attached and debugging along with the exe. During the process, I get unsupported format name operation error when initializing queues with strings. However, I am pretty sure and double checked that the strings are of correct syntax, and I have all the permission and access to the queues.
RequestQueue = new System.Messaging.MessageQueue(correctString);
Any ideas are appreciated. Great thanks.
There are a couple of different formats that can be specified in the MessageQueue constructor, that use differing syntax, depending on if they are public/ private queues, dead-letter-queues, journal queues, and so on.
For example:
Public queue: MachineName\QueueName
Private queue: MachineName\Private$\QueueName
Can you post an example of what you are using?
Also, if you are using an Format Name, check the spelling of your format string:
FormatName:DIRECT=OS:YOURMACHINENAME\private$\YourQueueName
Please note that the first part FormatName:DIRECT is case-sensitive. (More in-depth documentation about the syntax can be found in the MSDN here: Direct Format Names)
I am building an SNMP Agent for a Windows application using the Microsoft WinSNMP API. Currently everything is working for single-item get and set-request, and also for get-next to allow walking the defined tree (albeit with some caveats that are not relevant to this question).
I am now looking at multi-item get and also get-bulk.
My current procedure is to iterate through the list of requested items (the varbindlist within the PDU), treating each one individually, effectively causing an internal get. The result is added to the VBL, set into the PDU, and then sent back to the SNMP Manager, taking into account invalid requests, etc.
My question is how should I handle "too much" data (data that cannot fit into a single transport layer message)? Or more accurately, is there a way to test whether data is "too big" without actually attempting to transmit? The only way I can see in the API is to try sending, check the error, and try again.
In the case of a get-request this isn't a problem - if you can't return all of the requested data, you fail: so attempt sending, and if the error report is SNMPAPI_TL_PDU_TOO_BIG, send a default "error" PDU.
However, it is allowable for a response to bulk-get to return partial results.
The only way I can see to handle this is a tedious (?) loop of removing an item and trying again. Something similar to the following (some detail removed for brevity):
// Create an empty varbindlist
vbl = SnmpCreateVbl(session, NULL, NULL);
// Add all items to the list
SnmpSetVb(vbl, &oid, &value); // for each OID/Value pair
// Create the PDU
pdu = SnmpCreatePdu(session, SNMP_PDU_RESPONSE, ..., vbl);
bool retry;
do {
retry = false;
smiINT failed = SnmpSendMsg(session, ..., pdu);
if (failed && SNMPAPI_TL_PDU_TOO_BIG == SnmpGetLastError()) {
// too much data, delete the last vb
SnmpDeleteVb(vbl, SnmpCountVbl(vbl));
SnmpSetPduData(pdu, ..., vbl);
retry = true;
};
} while(retry);
This doesn't seem like an optimal approach - so is there another way that I've missed?
As a side-note, I know about libraries such as net-snmp, but my question is specific to the Microsoft API.
The RFC does require you to do what you pasted,
https://www.rfc-editor.org/rfc/rfc3416
Read page 16.
There does not seem to be a function exposed by WinSNMP API that can do this for you, so you have to write your own logic to handle it.