i am planning to call following message using grpcurl.
How to send a .google.protobuf.BytesValue via grpcurl?
message TestRequest {
.google.protobuf.BytesValue TestJson = 1;
string Temp = 2;
}
BytesValues is a message (wrapper) for bytes.
Try Base64 encoding the bytes field and passing it that way.
Protobuf's JSON encoding of bytes is a Base64 string.
I have a message and I would like to package it into an any repeated google proto type::
Is there a way to encode an repeated any message type?
Can I even use repeated tag with google.protobuf.any?
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
/** Any Message **/
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
I am looking for an example, currently using nanopb to encode.
Sure, it is just a regular message.
https://github.com/nanopb/nanopb/tree/master/tests/any_type shows how to encode a single Any message, encoding many is like encoding any array. You'll have a choice between allocating statically, allocating dynamically or using callbacks. Or you can just encode a single subfield at a time into output stream, because concatenating encoded concatenates arrays in protobuf format.
I think I found my issue, I cannot use(repeated tag on the google.protobuf.any, as I would like to append the RepeatedAny messages in the final binary):
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
Instead I should use something like this:
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message SensorAny{
google.protobuf.any sensor = 1;
}
message RepeatedAny{
repeated SensorAny sensors = 1;
}
I should not use the repeated tag on the google.protobuf.any, I should be using it on a message that contains the google.protobuf.any instead, so that the protobinary can contain the format (sensors1), (sensors2).....(sensorsN), one or more SensorAny messages.
Below is the sample code, if someone finds this question in the future for nanopb:
/* First encode the SensorAny message by setting the value of the first field,
The first field of this message is of type google.protobuf.any, so it should have
1. sensor.type_url
2. sensor.value
*/
void* pBufAny = calloc(1, sBufSize);
pb_ostream_t ostream_any = pb_ostream_from_buffer(pBufAny, sBufSize);
SensorAny SensorAnyProto = SensorAny_init_default;
SensorAnyProto.has_message = true;
SensorAnyProto.sensor.type_url.arg = "type.googleapis.com/SensorAny.proto";
SensorAnyProto.sensor.type_url.funcs.encode = Proto_encode_string;
ProtoEncodeBufferInfo_t BufInfo = {
.Buffer = pBuf, /* I have already filled and encoded Onesensor message previously as pBuf */
.BufferSize = ostream.bytes_written,
};
SensorAnyProto.sensor.value.funcs.encode = Proto_encode_buffer;
SensorAnyProto.sensor.value.arg = &BufInfo;
pb_encode(&ostream_any, SensorAny_fields, &SensorAnyProto);
free(pBuf);
// Now Use the above encoded Any message buffer pBufAny to set the first repeated field in RepeatedAny
RepeatedAny SensorAnyRepeated = RepeatedAny_init_default;
ProtoEncodeBufferInfo_t AnyBufInfo = {
.Buffer = pBufAny,
.BufferSize = ostream_any.bytes_written,
};
AnyRepeated.sensors.arg=&AnyBufInfo;
AnyRepeated.sensors.funcs.encode = Proto_encode_buffer;
void* pBufAnyRepeated = calloc(1, sBufSize);
pb_ostream_t ostream_repeated = pb_ostream_from_buffer(pBufAnyRepeated, sBufSize);
!pb_encode(&ostream_repeated, RepeatedAny_fields, &AnyRepeated);
free(pBufAny);
I'm trying to covert the string input in the serial to byte of array to write in a block in RFID card
String puta = "PUTANGINA";
byte blockcontent[16] = {puta};
I expect the output will be the string will be byte array.
You can use the getBytes() this copies the string's characters to the supplied buffer.
Syntax:
string.getBytes(buf, len)
Parameters:
string: a variable of type String
buf: the buffer to copy the characters into (byte [])
len: the size of the buffer (unsigned int)
Returns
None
https://www.arduino.cc/en/Reference.StringGetBytes
I want to access data in a database created by Rails for use by non-Ruby code. Some fields use attr_encrypted accessors, and the library in use is the symmetric-encryption gem. I consistently get a "wrong final block length" error if I try to decrypt the data with, e.g., the NodeJS crypto library.
I suspect this has to do either with character encoding or with padding, but I can't figure it out based on the docs.
As an experiment, I tried decrypting data from symmetric-encryption in Ruby's own OpenSSL library, and I get either a "bad decrypt" error or the same problem:
SymmetricEncryption.cipher = SymmetricEncryption::Cipher.new(
key: "1234567890ABCDEF",
iv: "1234567890ABCDEF",
cipher_name: "aes-128-cbc"
)
ciphertext = SymmetricEncryption.encrypt("Hello world")
c = OpenSSL::Cipher.new("aes-128-cbc")
c.iv = c.key = "1234567890ABCDEF"
c.update(ciphertext) + c.final
That gives me a "bad decrypt" error.
Interestingly, the encrypted data in the database can be decrypted by the symmetric-encryption gem, but isn't the same as the output of SymmetricEncryption.encrypt (and OpenSSL doesn't successfully decrypt it, either).
Edit:
psql=# SELECT "encrypted_firstName" FROM people LIMIT 1;
encrypted_firstName
----------------------------------------------------------
QEVuQwBAEAAuR5vRj/iFbaEsXKtpjubrWgyEhK5Pji2EWPDPoT4CyQ==
(1 row)
Then
irb> SymmetricEncryption.decrypt "QEVuQwBAEAAuR5vRj/iFbaEsXKtpjubrWgyEhK5Pji2EWPDPoT4CyQ=="
=> "Lurline"
irb> SymmetricEncryption.encrypt "Lurline"
=> "QEVuQwAAlRBeYptjK0Fg76jFQkjLtA=="
Looking at the source for the symmetric-encryption gem, by default it adds a header to the output and base64 encodes it, although both of these are configurable.
To decrypt using Ruby’s OpenSSL directly, you will need to decode it and strip off this header, which is 6 bytes long in this simple case:
ciphertext = Base64.decode64(ciphertext)
ciphertext = ciphertext[6..-1]
c = OpenSSL::Cipher.new("aes-128-cbc")
c.decrypt
c.iv = "1234567890ABCDEF"
c.key = "1234567890ABCDEF"
result = c.update(ciphertext) + c.final
Of course, you may need to alter this depending on what settings you are using in symmetric-encryption, e.g. the header length may vary. In order to decrypt the result from the database you will need to parse the header. Have a look at the source.
Based on the Rust implementation done by #Shepmaster in my other question (and the source code for the symmetric-encryption gem), I have a working version in TypeScript. #matt is close with his answer, but the header can actually have additional bytes containing metadata about the encrypted data. Note that this doesn't handle (1) compressed encrypted data, or (2) setting the encryption algorithm from the header itself; neither situation is relevant to my use case.
import { createDecipher, createDecipheriv, Decipher } from "crypto";
// We use two types of encoding with SymmetricEncryption: Base64 and UTF-8. We
// define them in an `enum` for type safety.
const enum Encoding {
Base64 = "base64",
Utf8 = "utf8",
}
// Symmetric encryption's header contains the following data:
interface IHeader {
version: number, // The version of the encryption algo
isCompressed: boolean, // Whether the data is compressed (TODO: Implement)
hasIv: boolean, // Whether the header itself has the IV
hasKey: boolean, // Whether the header itself has the Key
hasCipherName: boolean, // Whether the header contains the cipher name
hasAuthTag: boolean, // Whether the header has an authorization tag
offset: number, // How many bytes into the encoded ciphertext the actual encrypted data starts
iv?: Buffer, // The IV, present only if `hasIv` is true
key?: Buffer, // The key, present only if `hasKey` is true
// The cipher name, present only if `hasCipherName` is true. Currently ignored.
cipherName?: string,
authTag?: string, // The authorization tag, present only if // `hasAuthTag` is true
}
// Byte 6 of the header contain bit flags
interface IFlags {
isCompressed: boolean,
hasIv: boolean,
hasKey: boolean,
hasCipherName: boolean,
hasAuthTag: boolean
}
// The 7th byte until the end of the header have the actual values. If all
// of the flags are false, the header ends at the 6th byte.
interface IValues {
iv?: Buffer,
key?: Buffer,
cipherName?: string,
authTag?: string,
size: number,
}
/**
* Represent the encoded ciphertext, complete with the SymmetricEncryption header.
*/
class Ciphertext {
// Bit flags corresponding to the data encoded in byte 6 of the
// header.
readonly FLAG_COMPRESSED = 0b1000_0000;
readonly FLAG_IV = 0b0100_0000;
readonly FLAG_KEY = 0b0010_0000;
readonly FLAG_CIPHER_NAME = 0b0001_0000;
readonly FLAG_AUTH_TAG = 0b0000_1000;
// The literal data encoded in bytes 1 - 4 of the header
readonly MAGIC_HEADER = "#EnC";
// If any of the values represented by the bit flags is present, the first 2
// bytes of the data tells us how long the actual value is. In other words,
// the first 2 bytes aren't the value itself, but rather give the info about
// the length of the rest of the value.
readonly LENGTH_INFO_SIZE = 2;
public header: IHeader | null;
public data: Buffer;
private cipherBuffer: Buffer;
constructor(private input: string) {
this.cipherBuffer = new Buffer(input, Encoding.Base64);
this.header = this.getHeader();
const offset = this.header ? this.header.offset : 0; // If no header, then no offset
this.data = this.cipherBuffer.slice(offset);
}
/**
* Extract the header from the data
*/
private getHeader(): IHeader | null {
let offset = 0;
// Bytes 1 - 4 are the literal `#EnC`. If that's absent, there's no
// SymmetricEncryption header.
if (this.cipherBuffer.toString(Encoding.Utf8, offset, offset += 4) != this.MAGIC_HEADER) {
return null;
}
// Byte 5 is the version
const version = this.cipherBuffer.readInt8(offset++); // Post increment
// Byte 6 is the flags
const rawFlags = this.cipherBuffer.readInt8(offset++);
const flags = this.readFlags(rawFlags);
// Bytes 7 - end are the values.
const values = this.getValues(offset, flags);
offset += values.size;
return Object.assign({ version, offset }, flags, values);
}
/**
* Get the values for `iv`, `key`, `cipherName`, and `authTag`, if any are
* set, based on the bitflags. Return that data, plus how many bytes in the
* header those values represent.
*
* #param offset - What byte we're on when we get to the values. Should be 7
* #param flags - The flags we've extracted, showing us which values to expect
*/
private getValues(offset: number, flags: IFlags): IValues {
let iv: Buffer | undefined = undefined;
let key: Buffer | undefined = undefined;
let cipherName: string | undefined = undefined;
let authTag: string | undefined = undefined;
let size = 0; // If all of the bit flags are false, there is no additional data.
// For each value, see if the flag is set to true. If it is, we need to
// read the value. Keys and IVs need to be `Buffer` types; other values
// should be strings.
[iv, size] = flags.hasIv ? this.readBuffer(offset) : [undefined, size];
[key, size] = flags.hasKey ? this.readBuffer(offset + size) : [undefined, size];
[cipherName, size] = flags.hasCipherName ? this.readString(offset + size) : [undefined, size];
[authTag, size] = flags.hasAuthTag ? this.readString(offset + size) : [undefined, size];
return { iv, key, cipherName, authTag, size };
}
/**
* Parse the 16-bit integer representing the bit flags into an object for
* easier handling
*
* #param flags - The 16-bit integer that contains the bit flags
*/
private readFlags(flags: number): IFlags {
return {
isCompressed: (flags & this.FLAG_COMPRESSED) != 0,
hasIv: (flags & this.FLAG_IV) != 0,
hasKey: (flags & this.FLAG_KEY) != 0,
hasCipherName: (flags & this.FLAG_CIPHER_NAME) != 0,
hasAuthTag: (flags & this.FLAG_AUTH_TAG) != 0
}
}
/**
* Read a string out of the value at the specified offset. Return the value
* itself, plus the number of bytes consumed by the value (including the
* 2-byte encoding of the length of the actual value).
*
* #param offset - The offset (bytes from the beginning of the encoded,
* encrypted Buffer) at which the value in question begins
*/
private readString(offset: number): [string, number] {
// The length is the first 2 bytes, encoded as a little-endian 16-bit integer
const length = this.cipherBuffer.readInt16LE(offset);
// The total size occupied in the header is the 2 bytes encoding length plus the length itself
const size = this.LENGTH_INFO_SIZE + length;
const value = this.cipherBuffer.toString(Encoding.Base64, offset + this.LENGTH_INFO_SIZE, offset + size);
return [value, size];
}
/**
* Read a Buffer out of the value at the specified offset. Return the value
* itself, plus the number of bytes consumed by the value (including the
* 2-byte encoding of the length of the actual value).
*
* #param offset - The offset (bytes from the beginning of the encoded,
* encrypted Buffer) at which the value in question begins
*/
private readBuffer(offset: number): [Buffer, number] {
// The length is the first 2 bytes, encoded as a little-endian 16-bit integer
const length = this.cipherBuffer.readInt16LE(offset);
// The total size occupied in the header is the 2 bytes encoding length plus the length itself
const size = this.LENGTH_INFO_SIZE + length;
const value = this.cipherBuffer.slice(offset + this.LENGTH_INFO_SIZE, offset + size);
return [value, size];
}
}
/**
* Allow decryption of data encrypted by Ruby's `symmetric-encryption` gem
*/
class SymmetricEncryption {
private key: Buffer;
private iv?: Buffer;
constructor(key: string, private algo: string, iv?: string) {
this.key = new Buffer(key);
this.iv = iv ? new Buffer(iv) : undefined;
}
public decrypt(input: string): string {
const ciphertext = new Ciphertext(input);
// IV can be specified by the user. But if it's encoded in the header
// itself, go with that instead.
const iv = (ciphertext.header && ciphertext.header.iv) ? ciphertext.header.iv : this.iv;
// Key can be specified by the user. but if it's encoded in the header,
// go with that instead.
const key = (ciphertext.header && ciphertext.header.key) ? ciphertext.header.key : this.key;
const decipher: Decipher = iv ?
createDecipheriv(this.algo, key, iv) :
createDecipher(this.algo, key);
// Terse version of `update()` + `final()` that passes type checking
return Buffer.concat([decipher.update(ciphertext.data), decipher.final()]).toString();
}
}
const s = new SymmetricEncryption("1234567890ABCDEF", "aes-128-cbc", "1234567890ABCDEF");
console.log(s.decrypt("QEVuQwAADWK0cKzgFIovdIThq9Scrg==")); // => "Hello world"
because there is no extend in proto3, so I combine the base message with google.protobuf.Any type message, but it's binary length is too long
.proto file
message TradeMessage {
google.protobuf.Any message = 1;
string code = 2;
}
message Connect {
int32 seq = 1;
string appid = 2;
string clientid = 3;
string ver = 4;
}
...
.java file
TradeProtocol.Connect inner = TradeProtocol.Connect.newBuilder()
.setSeq(1)
.setAppid("test")
.build();
TradeProtocol.TradeMessage packet = TradeProtocol.TradeMessage.newBuilder()
.setMessage(Any.pack(inner))
.setCode(2)
.build();
service send packet to client, client can decode all message to base TradeMessage, the problem is the inner's length is 8 bytes, while packet's length is 56 bytes. the same function implement use proto2's extend just ten more bytes, so is there any way to implement extend function in proto3 or reduce the packet's length ? thanks
One alternative is to use oneof:
message Connect {
int32 seq = 1;
string appid = 2;
string clientid = 3;
string ver = 4;
}
message TradeMessage {
string code = 1;
oneof inner {
Connect inner_connect = 2;
SomeOtherMessage inner_other = 3;
...
}
}
The encoded size will still be larger than with extend, but only by 1-2 bytes.