My company is consider using protobuf3 as an message communication schema between our many microservices, and we struggle with find a solution for information lost when handling a message that transfer between more then 2 services and have more then one version, and I'll explain by writing an example:
I have three microservices, A,B,C that communicate with each other using the same schema:
message FooV1 {
int32 x = 1;
}
Where A send messages to B, B alter the messages and send the result to C (A->B->C).
Now let's say I alter the schema and add another field:
message FooV2 {
int32 x = 1;
int32 y = 2;
}
Then, I update only services A and C with the new schema, and B will remain with the old schema, like this: A:V2, B:V1, C:V2.
Supposed A send x=1 and y=2 to B. B knows only tag 1, so he don't parse y at all. Then he alter x by changing it, for instance, to be 0. Then B send it to C. C now get x that equal to 0, but in the process y is 'vanished' and even though A and C both support the new schema, and B doesn't care about y, we still lose data in the process.
I want C to know y even thought B doesn't recognize y
Is there any way to tell protobuf that even thought the service won't parse an unknown tag he won't get rid of the data but append it back to the message on sending? If there is a way to do that, how does it work? Thanks.
I found this paragraph on google's protobuf3 documentation:
Originally, proto3 messages always discarded unknown fields during parsing, but in version 3.5 we reintroduced the preservation of unknown fields to match the proto2 behavior. In versions 3.5 and later, unknown fields are retained during parsing and included in the serialized output.
so using protobuf3.5+ was the answer for me
Related
We have a communication channel on which we send protobufs. In order to be able to send more than one type of protobuf, we double-serialise:
message Coprolite {
enum Bezoar {
KBezoarUndef = 0;
KBezoarFriedEggs = 1;
KBezoarHam = 2;
}
Bezoar payload_type = 1;
bytes payload = 2;
}
If I have a FriedEggs protobuf, I serialise it, assign it to the payload of a Coprolite, set the payload_type to KBezoarFriedEggs, serialise the Coprolite, and send it on its way.
On receipt, I deserialise, check what I've got, and deserialise that.
This works on all of our platforms. I've not, however, found examples of others doing this this way (nor any other way, really). So this suggests I should ask for advice. Is there a better strategy or a reason that I should be wary of this?
If you want to prevent having to set payload_type you can use an oneof. Oneof's implicitly serialize the payload type by adding the tag number in front. Just like any other field serialized. So you have to write less administrative code.
There is however one advantage in your approach over using oneof's. Deepening on your programming language and how oneof's and byte arrays are implemented on your platform and in your protobuf library. An oneof implemented as an union might allocate memory the size of the largest nested message. So depending on your situation dynamically allocating the bytes array might use less memory when you send a lot of small message and only sometimes a big one.
There are two common approaches here; the simplest (especially when the types are predictable in advance) is usually oneof:
message Coprolite {
oneof payload_type {
FriedEggs eggs = 1;
Ham ham = 2;
}
}
This acts like a discriminated union, where you can check the embedded type.
In some niche scenarios, you may prefer to use Any
I was writing an external app in python that uses the message system in odoo.
So, I need to use, the mail_message, and the mail_notification tables.
I tried to put elements individually via INSERT into the table filling the necessary elements to make this work, and it works perfectly, the messages appear in the "inbox" of messages in Odoo and the notification appears correctly.
But checking the rest of the fields in this table, I see that message_id got a tag format (between <>) and a series of numbers (that I haven't found any correlation) followed by "-openerp-'res_id'-'model'-#'company'".
So, I don't know how to fill this field, my proofs determined that is not a necessary field, but in a serious implementation I don't know if left this field empty can cause some issues.
Anyone can explain me the reason of this field and how to fill it?
Thanks
You can check the code in tools/mail.py and do something similar
def generate_tracking_message_id(res_id):
"""Returns a string that can be used in the Message-ID RFC822 header field
Used to track the replies related to a given object thanks to the "In-Reply-To"
or "References" fields that Mail User Agents will set.
"""
try:
rnd = random.SystemRandom().random()
except NotImplementedError:
rnd = random.random()
rndstr = ("%.15f" % rnd)[2:]
return "<%.15f.%s-openerp-%s#%s>" % (time.time(), rndstr, res_id, socket.gethostname())
Now that we've reached Swift 2.0, I've decided to convert my, as yet unfinished, OS X app to Swift. Making progress but I've run into some issues with using termios and could use some clarification and advice.
The termios struct is treated as a struct in Swift, no surprise there, but what is surprising is that the array of control characters in the struct is now a tuple. I was expecting it to just be an array. As you might imagine it took me a while to figure this out. Working in a Playground if I do:
var settings:termios = termios()
print(settings)
then I get the correct details printed for the struct.
In Obj-C to set the control characters you would use, say,
cfmakeraw(&settings);
settings.c_cc[VMIN] = 1;
where VMIN is a #define equal to 16 in termios.h. In Swift I have to do
cfmakeraw(&settings)
settings.c_cc.16 = 1
which works, but is a bit more opaque. I would prefer to use something along the lines of
settings.c_cc.vim = 1
instead, but can't seem to find any documentation describing the Swift "version" of termios. Does anyone know if the tuple has pre-assigned names for it's elements, or if not, is there a way to assign names after the fact? Should I just create my own tuple with named elements and then assign it to settings.c_cc?
Interestingly, despite the fact that pre-processor directives are not supposed to work in Swift, if I do
print(VMIN)
print(VTIME)
then the correct values are printed and no compiler errors are produced. I'd be interested in any clarification or comments on that. Is it a bug?
The remaining issues have to do with further configuration of the termios.
The definition of cfsetspeed is given as
func cfsetspeed(_: UnsafeMutablePointer<termios>, _: speed_t) -> Int32
and speed_t is typedef'ed as an unsigned long. In Obj-C we'd do
cfsetspeed(&settings, B38400);
but since B38400 is a #define in termios.h we can no longer do that. Has Apple set up replacement global constants for things like this in Swift, and if so, can anyone tell me where they are documented. The alternative seems to be to just plug in the raw values and lose readability, or to create my own versions of the constants previously defined in termios.h. I'm happy to go that route if there isn't a better choice.
Let's start with your second problem, which is easier to solve.
B38400 is available in Swift, it just has the wrong type.
So you have to convert it explicitly:
var settings = termios()
cfsetspeed(&settings, speed_t(B38400))
Your first problem has no "nice" solution that I know of.
Fixed sized arrays are imported to Swift as tuples, and – as far as I know – you cannot address a tuple element with a variable.
However,Swift preserves the memory layout of structures imported from C, as
confirmed by Apple engineer Joe Groff:. Therefore you can take the address of the tuple and “rebind” it to a pointer to the element type:
var settings = termios()
withUnsafeMutablePointer(to: &settings.c_cc) { (tuplePtr) -> Void in
tuplePtr.withMemoryRebound(to: cc_t.self, capacity: MemoryLayout.size(ofValue: settings.c_cc)) {
$0[Int(VMIN)] = 1
}
}
(Code updated for Swift 4+.)
Am trying to read the lotus notes document using VB6.I can able to read the values of the but suddenly type mismatch error is throwed.When i reintialise the vb6 variable it works but stops after certain point.
ex; address field in lotus notes
lsaddress=ImsField(doc.address)
private function ImsField(pValue)
ImsField=pValue(0)
end function
Like this I am reading the remaining fields but at certain point the runtime error "13" type mismatch error throwed.
I have to manually reintialize by
set doc=view.getdocumentbykey(doclist)
The type mismatch error occurs for a certain field. The issue should be a data type incompatibility. Try to figure out which field causes the error.
Use GetItemValue() instead of short notation for accessing fields and don't use ImsField():
lsaddress=doc.GetItemValue("address")(0)
The type mismatch is occurring because you are encountering a case where pValue is not an array. That will occur when you attempt to reference a NotesItem that does not exist. I.e., doc.MissingItem.
You should not use the shorthand notation doc.itemName. It is convenient, but it leads to sloppy coding. You should use getItemValue as everyone else is suggesting, and also you should check to see if the NotesItem exists. I.e.,
if doc.hasItem("myItem") then
lsaddress=doc.getItemValue("myItem")(0)
end if
Notes and Domino are schema-less. There are no data integrity checks other than what you write yourself. You may think that the item always has to be there, but the truth is that there is nothing that will ever guarantee that, so it is always up to you to write your code so that it doesn't assume anything.
BTW: There are other checks that you might want to perform besides just whether or not the field exists. You might want to check the field's type as well, but to do that requires going one more level up the object chain and using getFirstItem instead of getItemValue, which I'm not going to get into here. And the reason, once again, is that Notes and Domino are schema-less. You might think that a given item must always be a text list, but all it takes is someone writing sloppy code in an one-time fix-it agent and you could end up having a document in which that item is numeric!
Checking your fields is actually a good reason (sometimes) to encapsulate your field access in a function, much like the way you have attempted to do. The reason I added "sometimes" above is that your code's behavior for a missing field isn't necessarily always going to be the same, but for cases where you just want to return a default value when the field doesn't exist you can use something like this:
lsaddress ImsField("address","")
private function ImsField(fieldName,defaultValue)
if doc.hasItem(fieldName) then
lsaddress=doc.getItemValue(fieldName)(0)
else
lsaddress=defaultValue
end if
end function
Type mismatch comes,
When you try to set values from one kind of datatype variable to different datatype of another variable.
Eg:-
dim x as String
Dim z as variant
z= Split("Test:XXX",":")
x=z
The above will through the error what you mentioned.
So check the below code...
lsaddress = ImsField(doc.address)
What is the datatype of lsaddress?
What is the return type of ImsField(doc.address)?
If the above function parameter is a string, then you should pass the parameter like (doc.address(0))
When I run the following .Net code:
using (var c = Shared.DataSources.BSS1.CreateCommand())
{
c.CommandText = "\r\nSelect c1, c2, c3, rowid \r\nFrom someSpecificTable \r\nWhere c3 = :p0";
var p = c.CreateParameter() as Oracle.DataAccess.Client.OracleParameter;
c.Parameters.Add(p);
p.OracleDbType = Oracle.DataAccess.Client.OracleDbType.Varchar2;
p.DbType = System.Data.DbType.AnsiString;
p.Size = 20;
p.Value = "007";
p.ParameterName = ":p0";
using (var r = c.ExecuteReader())
{
r.Read();
}
}
I get the following error:
ORA-01460: unimplemented or unreasonable conversion requested
ORA-02063: preceding line from XXX
This is not my database, and I don't have control over the select statements that I get, that table IS from a database link.
The funny thing is that if I add the following code just before the ExecuteReader it runs fine.
c.CommandText = c.CommandText.Replace("\r\n", " ");
Unfortunately that is not a good solution in my case as I can't control to SQL nore can I change it that way.
As for the table itself, the columns are:
c1 Number(5)
c2 varchar2(40)
c3 varchar2(20).
I know that ORA-02063 that comes after indicate something about a database link, but I looked in the synonim table and it didn't come from any database_link, and also I don't think that \r\n should affect database link.
I tried running the query without bound parameters, and it did work - but again bad practice to do so in a general term.
The trouble is that a competing tool that is not .Net based, is working and thus it's not a general problem.
I also couldn't reproduce the problem in my own environment, this is a customer database and site.
I am using instant client 11.1.6.20 and also tested it with instant client 11.2.3.0
The db is 10 and the db link is to an oracle v8 database
Any help would be appreciated
This problem can be recreated with straight forward steps. That is, any SQL query having a string literal, in where clause, more than 4000 characters in length gives an error "ORA-01704: string literal too long"
But, when the same query is executed through JDBC it gives "ORA-01460: unimplemented or unreasonable conversion requested"
Finally I found the answer!!!
After investigating and reflecting into the code I found that by changing the Direction of the Parameter to input output - the problem was resolved.
p.Direction = ParameterDirection.InputOutput;
After much investigation I found out that it's all about the fact that we have bound parameters that are used from ODP.NET and targeting tables from a DBLINK to a V8 Oracle server.
Once I eliminated the bound parameters it all worked.
It was a while back, but I think it's had something to do with varying string lengths of the strings sent to the bound parameter.
It seems that it ignored the size property, so if in the first query I sent a string with length 10 and in the second string I sent a string with length 12, I'll get that error.
I also found the oracle articles about it :
https://community.oracle.com/thread/2460796?tstart=0
and the patch for it:
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=745005.1
But - I found a fix in my code that actually solved it - see my next answer.
Hope this helps anyone.
The accepted answer didn't work for me. However, after reading the attached links, I applied the following – although it does involve editing the SQL.
In my case, I knew the maximum left of the bind variable (the length reducing after the first call is what causes the issue). So I padded the .NET string, and added a TRIM in the SQL. Following your example:
c.CommandText = "\r\nSelect c1, c2, c3, rowid \r\nFrom someSpecificTable \r\nWhere c3 = TRIM(:p0)";
...
p.Value = "007".PadRight(10);