I have a hash of key values pairs.
I do not have the keys, nor the values.
I can assume most will be there, but sometimes keys are removed and others are added. Is there any way for me to have a message with variable keys?
{
"knownkey1": "value",
"knownkey2": {
"unknown-key1": "value",
"unknown-key2": "value"
}
}
or is the best ways to just serialize it using json stringify in the message? i would think that would defeat the whole purpose of using grpc.
message Rate {
string ticker = 1;
string value = 2;
}
message GetAllResponse {
string lastUpdated = 1;
repeated Rate payload = 2;
}
Looks like you can just use the maps type as outlined here:
https://developers.google.com/protocol-buffers/docs/proto3#maps
Related
Suppose I have a proto message like this:
message WorkflowParameters {
oneof parameters {
WorkflowAParams a = 1;
WorkflowBParams b = 2;
}
}
And I want to have another message where the type of workflow can be specified. Something like this:
message ListWorkflowsRequest {
// The type of workflows to fetch
WorkflowParameters.parameters workflow_type = 1;
}
The above doesn't work (it throws "WorkflowParameters.parameters" is not a type.) What's the recommended way of doing this?
It's not possible. oneof is only a thin syntatic sugar/behavior change, and has no effect on the actual schema. It affects the generated code's behavior, but not the serialized format. In the following example, these two messages are interchangeable and wire-compatible:
message WorkflowParameters {
oneof parameters {
WorkflowAParams a = 1;
WorkflowBParams b = 2;
}
}
message WorkflowParameters2 {
WorkflowAParams a = 1;
WorkflowBParams b = 2;
}
Now, if you just want to specify which part of a oneof will be set, you could theoretically use the generated code constants, and a simple int field:
message ListWorkflowsRequest {
// The field number of WorkflowParameters that should be filled.
int32 workflow_type = 1;
}
All language generators should have convenient enough constants created, like WorkflowParameters::A_FIELD_NUMBER for C++.
I'm trying to reverse engineer a client that uploads an audio file to a server, and then uploads the file's metadata in a separate request afterwards. The metadata is serialized in Protobuf, and it uses a fairly simple and readable structure. Here's what it looks like after protoc --decode_raw:
1 {
1: "title"
2: "This is the audio title"
}
1 {
1: "tags"
2 {
}
}
1 {
1: "loc"
2: "This is the location"
}
1 {
1: "transcription"
2: "A transcript"
}
1 {
1: "mapping"
2 {
1 {
1: 6
2 {
3: 840
}
}
2: 6
}
}
It seems to be just a repeated message at field 1 that holds a key-value pair each time, but sometimes the value is a string, and sometimes it's a more complex message. How can they assign both a string and a message to field 2 if Protobuf only lets you use one value type per field? If I'm going to craft my own request, I need something like this:
message KeyValuePair {
string key = 1;
oneof value {
string str_value = 2;
MessageValue msg_value = 2;
}
}
But that doesn't work, because Field number 2 has already been used in "package.name" by field "str_value". Any ideas? I'll be using Python to create and send the request data.
There is an official way to achieve this: google.protobuf.Any
If a protobuf schema defines an any at the top level, e.g.:
message Root {
repeated google.protobuf.Any value = 1;
}
message Title {
string title= 2;
}
message Tags {
string name = 1;
repeated string tags = 2;
}
Then messages of any Protobuf-defined type can be serialized in a list.
However, I don't think that's what the existing code is doing:
the raw output for Any normally includes the type.googleapis.com type url
Using Any, the title / loc fields would be encapsulated in a nested object, not a string at the same level.
E.g.:
1 {
1: "type.googleapis.com/Title"
2 {
1: "the title"
}
}
I have Redis DB consisting of hashes and strings.
I got all keys from DB using this code:
Set<byte[]> keys = redisTemplate.getConnectionFactory().getConnection().keys("*".getBytes());
Iterator<byte[]> it = keys.iterator();
while(it.hasNext()){
byte[] data = (byte[])it.next();
String key = (new String(data, 0, data.length));
System.out.println(key);
}
from here: How to get all Keys from Redis using redis template
Since the key can be hash or string, how to determine when I can use opsForHash and opsForValue i.e. how to check if it is hash or it is a string in spring boot using restTemplate.
The Redis command to get the type of a key is TYPE: https://redis.io/commands/type
You can use RedisTemplate's public DataType type(K key) method to execute this: https://docs.spring.io/spring-data/redis/docs/current/api/org/springframework/data/redis/core/RedisTemplate.html#type-K-
Here's an example:
Set<byte[]> keys = redisTemplate.getConnectionFactory().getConnection().keys("*".getBytes());
Iterator<byte[]> it = keys.iterator();
while(it.hasNext()){
byte[] data = (byte[])it.next();
String key = (new String(data, 0, data.length));
DataType type = redisTemplate.type(key);
if (type == DataType.HASH) {
// ...
} else if (type == DataType.STRING) {
// ...
}
}
Edit: one more piece of advice is that you may want to use SCAN instead of KEYS * (mentioned in one answer to the SO question you linked). Scan is generally better in production since it doesn't try to get and return all keys at once.
I am new to Redis and I'm trying to write a simple project that collects information from SQL database and caches into Redis. As I'm more comfortable with C#, I've chosen StackExchange.Redis to do that.
Let's say I have a table in my db with a schema like this Persons(ID, Name, Address, Age, BirthDate).
I have a Person class in my project with corresponding fields.
I have also a function GetPersonByID(ID), that requests the Redis, if a key with the ID doesn't exist it executes another function called GetPersonByID_SQL(ID), when an sql query is being executed, after getting information from db it creates an object Person, adds that object to Redis(using hashTable) and returns the object. If the key existed in Redis the function just gets information from Redis, creates an object Person, maps the corresponding values values to fields and returns that object.
Here is the code of how I do this.
public static Person GetPersonByID(string ID)
{
redis = ConnectionMultiplexer.Connect("127.0.0.1");
IDatabase db = redis.GetDatabase();
Person p;
if (!db.KeyExists(key))
{
p = Person.GetPersonByID_SQL(ID);
db.HashSet(key, "Name", p.Name);
db.HashSet(key, "Address", p.Address);
db.HashSet(key, "Age", p.Age);
db.HashSet(key, "BirthDate", p.BirthDate);
}
else
{
HashEntry[] values = db.HashGetAll(key);
p = new Person();
for (int i = 0; i < values.Length; i++)
{
HashEntry hashEntry = values[i];
switch (hashEntry.Name)
{
case "Name": p.Name = hashEntry.Value; break;
case "Address": p.Address = hashEntry.Value; break;
case "Age": p.Age = hashEntry.Value; break;
case "BirthDate": p.BirthDate = hashEntry.Value; break;
}
}
}
return p;
}
My question is, Is there any way I can Bind automatically the value of Redis (that is in my way a HashTable) to my existing object?
My question is, Is there any way I can Bind automatically the value of Redis (that is in my way a HashTable) to my existing object?
No, this is not a feature of StackExchange.Redis - values can only be stored as the most basic types (strings and numbers). You're expected to do the conversions to more complex values yourself.
So you can either store your person object as multiple hash fields (as you've done in your code), or you can store your person object as a single serialized string against a key (you can use the STRING data structure for this instead). Then you can perform the necessary deserializations when you want to retrieve a value.
If you always want to retrieve every value from your Person data structure, I would recommend going for the second option. You'll only then need to use a single command command for each get/set:
// SET
_redis.StringSet(key, serializedPerson);
// GET
string serializedPerson = _redis.StringGet(key);
How to send message type as object in ProtoBuf - Proto3 Syntax?
I want to transfer object instead of string or number.
Example
{
name: 'One',
date: 'date',
some: 'some',
...
...
}
syntax = "proto3";
package db;
service Proxy
{
rpc myFunction(Request) returns (Response);
}
message Request
{
string name = 1;
}
message Response
{
object keyvalue = 1;
}
Here, I am getting error
throw Error("no such Type or Enum '" + path + "' in " + this);
^
Error: no such Type or Enum 'object' in Type
--
Workaround
I can convert it to string in server side, and then I can JSON.parse() at client.
But I want to know, if there any better way to do it.
protocol-buffer does not support the object data type!
But you can emulate your data as hierarchically by using protocol buffer message type itself.
syntax = "proto3";
package db;
service Proxy
{
rpc myFunction(Request) returns (Response);
}
message Request
{
string name = 1;
}
message Response
{
message obj {
string key1 = 1;
string key2 = 2
}
obj keyvalue = 1; // Here you have created your own type obj.
}
In the above example, you can see that the Response message now has "keyvalue" field of type obj(which is a custom type you have just built).
Now you will pass Object in a callback from the server instead of primitive type.
callback(null, { keyvalue: { key1: "value1", key2: "value2" } });
Let's say if keys are unknown to you but key/value pair data type is same and known to you then, in this case, you can use
map<type, type>
message Response
{
map<string, string> keyvalue = 1;
}
callback(null, { keyvalue: { "key1": "value1", "key5": "value2" } });
References:-
https://developers.google.com/protocol-buffers/docs/proto3#other
https://developers.google.com/protocol-buffers/docs/proto#scalar
https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Struct
Depending on what you're trying to accomplish, you can likely use one of three types.
The bytes type represents an arbitrary string of bytes. You can choose to encode your types in anyway you feel appropriate.
The any type represents an arbitrary protobuf message, allowing the client or server to encode non-predefined messages. This is a string of bytes, with a unique identifier per message type and language implementations will have ways to unpack the message.
You mention using Javascript. You could likely define a Javascript object message using a combination of oneof for the basic types and map for object types.
If all of the keys and values you want to send have the same type, you can use a map type. Your example shows string keys and string values, so that could use the type map<string, string>.
If you want to send arbitrary object data with similar structure to JSON objects, you can use the google.protobuf.Struct message type.