protobuf:
message test
{
int16 a:1
message testdata{
int16 b:1 `
int16 c:2
repeated testdata test_data=2
}
}
dt=test()
dt.a=11
dt1=d.testdata.add()
dt1.b=2222
dt1.c=3333
send=dt.SerializeToString()
now while deserialising code as below:
t=test()
t.ParseFromString(send)
t1=t.testdata()
print(t1.test_data)----> empty string
print(t1.b)--> value is not printing as expected, its printing object name instead of value.
how to access the nested protobuf elements and any input or suggestions are welcome.
I encourage you to read the documentation.
Your protobuf file is invalid.
A working equivalent of what you've written could be:
example.proto:
syntax = "proto3";
message test {
int32 a=1;
message testdata {
int32 b=1;
int32 c=2;
}
repeated testdata test_data=2;
}
Compiled using:
protoc \
--proto_path=${PWD} \
--python_out=${PWD} \
${PWD}/example.proto
Example Python code (with apologies for my poor variable naming choices):
import example_pb2
from google.protobuf.message import DecodeError
# Create a test message (f)
f = example_pb2.test()
f.a = 1
# Add a repeated property of type testdata
g=f.test_data.add()
g.b=1
g.c=1
# Add a repeated property of type testdata
g=f.test_data.add()
g.b=2
g.c=2
# Add a repeated property of type testdata
g=f.test_data.add()
g.b=3
g.c=3
# Output it
print(f)
# Serialize it and output it
x=f.SerializeToString()
print(x)
# Create a new test message (h) and parse x into it
h=example_pb2.test()
h.ParseFromString(x)
print(h)
# Create incorrectly formatted message (y) as a variant of x
y=b'\x08\x01\x12\x12\x04\x08\x01\x10\x01\x12\x04\x08\x02\x10\x02\x12\x04\x08\x03\x10\x03'
j=example_pb2.test()
try:
j.ParseFromString(y)
print(j)
except DecodeError as e:
print(e)
Yielding:
a: 1
test_data {
b: 1
c: 1
}
test_data {
b: 2
c: 2
}
test_data {
b: 3
c: 3
}
b'\x08\x01\x12\x04\x08\x01\x10\x01\x12\x04\x08\x02\x10\x02\x12\x04\x08\x03\x10\x03'
a: 1
test_data {
b: 1
c: 1
}
test_data {
b: 2
c: 2
}
test_data {
b: 3
c: 3
}
Error parsing message
Related
i have a serialized bin file of protobufs, written mainly in protobufs-net.
i want to decompile it, and see the structure of it.
i used some toolds like :
https://protogen.marcgravell.com/decode
and i also used protoc:
protoc --decode_raw < ~/Downloads/file.bin
and this is part of the result i get:
1 {
1: "4f81b7bb-d8bd-e911-9c1f-06ec640006bb"
2: 0x404105b1663ef93a
3: 0x4049c6158c593f36
4: 0x40400000
5 {
1: "53f8afde-04c6-e811-910e-4622e9d1766e"
2 {
1: "e993fba0-8fc9-e811-9c15-06ec640006bb"
}
2 {
1: "9a7c7210-3aca-e811-9c15-06ec640006bb"
2: 1
}
2 {
1: "2d7d12f1-2bc9-e811-9c15-06ec640006bb"
}
3: 18446744073709551615
}
6: 46
7: 1571059279000
}
how i can decompile it? i want to know the structure and change data in it and make a new bin file.
Reverse engineering a .proto file is mostly a case of looking at the output of the tools such as you've mentioned, and trying to write a .proto that looks similar. Unfortunately, a number of concepts are ambiguous if you don't know the schema, as multiple different data types and shapes share the same encoding details, but... we can make guesses.
Looking at your output:
1 {
...
}
tells us that our root message probably has a sub-message at field 1; so:
message Root {
repeated Foo Foos = 1;
}
(I'm guessing at the repeated here; if the 1 only appears once, it could be single)
with everything at the next level being our Foo.
1: "4f81b7bb-d8bd-e911-9c1f-06ec640006bb"
2: 0x404105b1663ef93a
3: 0x4049c6158c593f36
4: 0x40400000
5: { ... }
6: 46,
7: 1571059279000
this looks like it could be
message Foo {
string A = 1;
sfixed64 B = 2;
sfixed64 C = 3;
sfixed32 D = 4;
repeated Bar E = 5; // again, might not be "repeated" - see how many times it occurs
int64 F = 6;
int64 G = 7;
}
however; those sfixed64 could be double, or fixed64; and those sfixed32 could be fixed32 or float; likewise, the int64 could be sint64 or uint64 - or int32, sint32, uint32 or bool, and I wouldn't be able to tell (they are all just "varint"). Each option gives a different meaning to the value!
our Bar definitely has some kind of repeated, because of all the 2:
1: "53f8afde-04c6-e811-910e-4622e9d1766e"
2 { ... }
2 { ... }
2 { ... }
3: 18446744073709551615
let's guess at:
message Bar {
string A = 1;
repeated Blap B = 2;
int64 C = 3;
}
and finally, looking at the 2 from the previous bit, we have:
1: "e993fba0-8fc9-e811-9c15-06ec640006bb"
and
1: "9a7c7210-3aca-e811-9c15-06ec640006bb"
2: 1
and
1: "2d7d12f1-2bc9-e811-9c15-06ec640006bb"
so combining those, we might guess:
message Blap {
string A = 1;
int64 B = 2;
}
Depending on whether you have more data, there may be additional fields, or you may be able to infer more context. For example, if an int64 value such as Blap.B is always 1 or omitted, it might actually be a bool. If one of the repeated elements always has at most one value, it might not be repeated.
The trick is to to play with it until you can deserialize the data, re-serialize it, and get the exact same payload (i.e. round-trip).
Once you have that: you'll want to deserialize it, mutate the thing you wanted to change, and serialize.
For protofile:
syntax = "proto3";
package messagepb;
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
option (gogoproto.marshaler_all) = true;
option (gogoproto.sizer_all) = true;
option (gogoproto.unmarshaler_all) = true;
option (gogoproto.goproto_getters_all) = false;
service KV {
// Put puts the given key into the store.
// A put request increases the revision of the store,
// and generates one event in the event history.
rpc Put(PutRequest) returns (PutResponse) {}
}
message PutRequest {
bytes key = 1;
bytes value = 2;
}
message ResponseHeader {
repeated PutRequest l = 3;
}
I get the following proto struct:
type ResponseHeader struct {
L []*PutRequest `protobuf:"bytes,3,rep,name=l" json:"l,omitempty"`
}
But how do I get following protostruct:
type ResponseHeader struct {
L []PutRequest `protobuf:"bytes,3,rep,name=l" json:"l,omitempty"`
}
That is I want having data locality (and thus slices of contig data, not pointers to spread out stuff)
I needed to use: [(gogoproto.nullable) = false]
as in:
repeated PutRequest l = 3 [(gogoproto.nullable) = false];
And got:
L []PutRequest `protobuf:"bytes,3,rep,name=l" json:"l"`
I'm creating a compression/decompression library in Rust using Huffman encoding. One of the first steps is creating a data structure that contains all unique characters and the number of occurrences. I'm starting with just a simple text file, and having issues related to newline 'characters'.
My first attempt at solving this problem was constructing a BTreeMap, essentially a key-value pair of unique characters and their occurrences, respectively. Unfortunately, a newline 'character' is \n, which I think is not being handled corrected due to being two characters. I then converted the BTreeMap into a Vec to order by value, but that didn't solve the newline issue.
Here's my initial attempt at my comp binary package. Calling the binary is done using cargo, and my sample file is reproduced at the end of this question:
cargo run <text-file-in> <compressed-output-file>
main.rs
extern crate comp;
use std::env;
use std::process;
use std::io::prelude::*;
use comp::Config;
fn main() {
// Collect command-line args into a vector of strings
let mut stderr = std::io::stderr();
let config = Config::new(env::args()).unwrap_or_else(|err| {
writeln!(&mut stderr, "Parsing error: {}", err).expect("Could not write to stderr");
process::exit(1)
});
println!("Filename In: {}", config.filename_in);
println!("Filename Out: {}", config.filename_out);
if let Err(e) = comp::run(config) {
writeln!(&mut stderr, "Application error: {}", e).expect("Could not write to stderr");
process::exit(1);
}
}
lib.rs
use std::collections::btree_map::BTreeMap;
use std::error::Error;
use std::fs::File;
use std::io::Read;
use std::iter::FromIterator;
pub struct Config {
pub filename_in: String,
pub filename_out: String
}
impl Config {
pub fn new(mut args: std::env::Args) -> Result<Config, &'static str> {
args.next();
let filename_in = match args.next() {
Some(arg) => arg,
None => return Err("Didn't get a filename_in string"),
};
let filename_out = match args.next() {
Some(arg) => arg,
None => return Err("Didn't get a filename_out string"),
};
Ok(Config {
filename_in: filename_in,
filename_out: filename_out,
})
}
}
pub fn run(config: Config) -> Result<(), Box<Error>> {
let mut f = File::open(config.filename_in)?;
let mut contents = String::new();
f.read_to_string(&mut contents)?;
for line in contents.lines() {
println!("{}", line);
}
// Put unique occurrences into a BTreeMap
let mut count = BTreeMap::new();
for c in contents.chars() {
*count.entry(c).or_insert(0) += 1;
}
// Put contents into a Vec to order by value
let mut v = Vec::from_iter(count);
v.sort_by(|&(_, a), &(_, b)| b.cmp(&a));
// Print key-value pair of input file
println!("Number of occurrences of each character");
for &(key, value) in v.iter() {
println!("{}: {}", key, value);
}
Ok(())
}
Sample text file, poem.txt:
I'm nobody! Who are you?
Are you nobody, too?
Then there's a pair of us — don't tell!
They'd banish us, you know.
How dreary to be somebody!
How public, like a frog
To tell your name the livelong day
To an admiring bog!
Usage:
$ cargo run poem.txt poem
Compiling comp v0.1.0 (file:///home/chris/Projects/learn_rust/comp-rs)
Finished dev [unoptimized + debuginfo] target(s) in 1.96 secs
Running `target/debug/comp poem.txt poem`
Filename In: poem.txt
Filename Out: poem
I'm nobody! Who are you?
Are you nobody, too?
Then there's a pair of us — don't tell!
They'd banish us, you know.
How dreary to be somebody!
How public, like a frog
To tell your name the livelong day
To an admiring bog!
Number of occurrences of each character
: 36
o: 24
e: 15
a: 10
n: 10
y: 10
< What's going on here?
: 9 < What's going on here?
r: 9
d: 8
l: 8
b: 7
i: 7
t: 7
u: 7
h: 6
s: 5
!: 4
': 4
T: 4
g: 4
m: 4
,: 3
w: 3
?: 2
H: 2
f: 2
k: 2
p: 2
.: 1
A: 1
I: 1
W: 1
c: 1
v: 1
—: 1
Unfortunately, a newline 'character' is \n, which I think is not being handled corrected due to being two characters.
No, it is not. A newline character (UTF-8 codepoint 0x0A) is a single character.
I think I need to newline character to be a key in my key-value pair, but it's currently two keys.
No, it is not. Such a thing cannot happen "accidentally" either. If we somehow had two keys, you would have to call insert twice; there's no built-in concept of a multi-key map.
All that's happening here is that a newline character is printed as... a newline!
y: 10
: 9
If you take the time to create a MCVE, you'd see this quickly:
fn main() {
let c = '\n';
println!(">{}<", c);
println!(">{:?}<", c);
}
>
<
>'\n'<
The newline character is actually an escape sequence character. This means that if you write it as \n in the code which shows up as two characters, it's actually a placeholder for a single character - a new line - and should be treated as 'one character' in the program during runtime.
The core issue you have here is that you're using println to print it out to the command line and actually printing an new line, as the \n is interpreted to mean "A new line". This is why, when you use println here, you get the behavior you see. This is typical of most languages.
While this adds a little additional bit of code, you may wish to do something like this instead to specially-handle new-line data being printed:
// Print key-value pair of input file
println!("Number of occurrences of each character");
for &(key, value) in v.iter() {
if key == '\n' {
println!("\\n": {}, value);
} else {
println!("{}: {}", key, value);
}
}
Consider as explained by Shepmaster though to create an MCVE to thoroughly test things, it helps rule out misinterpretation of what is actually happening behind the scenes.
(NOTE: I am not a Rust master; there is probably a better way to achieve the above, but this is the shortest solution I came up with in a short period of time)
I used PycParser to generate an abstract syntax tree for a C function, but I'm still trying to figure out how to convert the parse tree into a string:
from pycparser import c_parser
src = '''
int hello(a, b){
return a + b;
}
'''
ast = c_parser.CParser().parse(src)
aString = ast.show()
print(aString) #This prints "None" instead of printing the parse tree
Would it be possible to generate a string (or an array of strings) from the parse tree that has been generated?
This is the abstract syntax tree that was printed, but I can't figure out how to convert it to a string:
FileAST:
FuncDef:
Decl: hello, [], [], []
FuncDecl:
ParamList:
ID: a
ID: b
TypeDecl: hello, []
IdentifierType: ['int']
Compound:
Return:
BinaryOp: +
ID: a
ID: b
The show method accepts a buf keyword parameter - you can pass a StringIO there. sys.stdout is the default. See https://github.com/eliben/pycparser/blob/master/pycparser/c_ast.py#L30
Try this, please
from pycparser import c_parser
src = '''
int hello(a, b){
return a + b;
}
'''
ast = c_parser.CParser().parse(src)
ast_buffer_write=open("buffer.txt","w") # here will be your AST source
ast.show(ast_buffer_write)
ast_buffer_write.close()
ast_buffer_read=open("buffer.txt","r")
astString=ast_buffer_read.read()
print(astString)
While learning Scalaz 6, I'm trying to write type-safe readers returning validations. Here are my new types:
type ValidReader[S,X] = (S) => Validation[NonEmptyList[String],X]
type MapReader[X] = ValidReader[Map[String,String],X]
and I have two functions creating map-readers for ints and strings (*):
def readInt( k: String ): MapReader[Int] = ...
def readString( k: String ): MapReader[String] = ...
Given the following map:
val data = Map( "name" -> "Paul", "age" -> "8" )
I can write two readers to retrieve the name and age:
val name = readString( "name" )
val age = readInt( "age" )
println( name(data) ) //=> Success("Paul")
println( age(data) ) //=> Success(8)
Everything works fine, but now I want to compose both readers to build a Boy instance:
case class Boy( name: String, age: Int )
My best take is:
val boy = ( name |#| age ) {
(n,a) => ( n |#| a ) { Boy(_,_) }
}
println( boy(data) ) //=> Success(Boy(Paul,8))
It works as expected, but the expression is awkward with two levels of applicative builders. Is there a way, to get the following syntax to work ?
val boy = ( name |#| age ) { Boy(_,_) }
(*) Full and runnable implementation in: https://gist.github.com/1891147
Update: Here is the compiler error message that I get when trying the line above or Daniel suggestion:
[error] ***/MapReader.scala:114: type mismatch;
[error] found : scalaz.Validation[scalaz.NonEmptyList[String],String]
[error] required: String
[error] val boy = ( name |#| age ) { Boy(_,_) }
[error] ^
How about this?
val boy = (name |#| age) {
(Boy.apply _).lift[({type V[X]=ValidationNEL[String,X]})#V]
}
or using a type alias:
type VNELStr[X] = ValidationNEL[String,X]
val boy = (name |#| age) apply (Boy(_, _)).lift[VNELStr]
This is based on the following error message at the console:
scala> name |#| age apply Boy.apply
<console>:22: error: type mismatch;
found : (String, Int) => MapReader.Boy
required: (scalaz.Validation[scalaz.NonEmptyList[String],String],
scalaz.Validation[scalaz.NonEmptyList[String],Int]) => ?
So I just lifted Boy.apply to take the required type.
Note that since Reader and Validation (with a semigroup E) are both Applicative, their composition is also Applicative. Using scalaz 7 this can be expressed as:
import scalaz.Reader
import scalaz.Reader.{apply => toReader}
import scalaz.{Validation, ValidationNEL, Applicative, Kleisli, NonEmptyList}
//type IntReader[A] = Reader[Int, A] // has some ambigous implicit resolution problem
type IntReader[A] = Kleisli[scalaz.IdInstances#Id, Int, A]
type ValNEL[A] = ValidationNEL[Throwable, A]
val app = Applicative[IntReader].compose[ValNEL]
Now we can use a single |#| operation on the composed Applicative:
val f1 = toReader((x: Int) => Validation.success[NonEmptyList[Throwable], String](x.toString))
val f2 = toReader((x: Int) => Validation.success[NonEmptyList[Throwable], String]((x+1).toString))
val f3 = app.map2(f1, f2)(_ + ":" + _)
f3.run(5) should be_==(Validation.success("5:6"))