How to write my own code generator of protobuf - protocol-buffers

Question

Google protobuf is a nice IDL for RPC. But I want to know how to write my own code generator for protobuf.

Answer 1

The protoc compiler can output a protobuf-formatted description of the .proto file. That way most of the parsing has been done for you already, and you only need to generate the output you want.
The .proto schema for the .proto file description is here:
https://github.com/google/protobuf/blob/master/src/google/protobuf/descriptor.proto
As an additional step, you can make your generator runnable via an "-mygenerator-out=." option on protoc itself:
https://developers.google.com/protocol-buffers/docs/reference/other
Here is one (albeit a bit convoluted) example on how a code generator can be written in Python:
https://github.com/nanopb/nanopb/blob/master/generator/nanopb_generator.py

Answer 2

A protoc plugin is a binary that takes a protobuf message of type CodeGeneratorRequest and returns a response of type CodeGeneratorResponse to standard out.
The binary must be called protoc-gen-NAME and can be used by invoking the protoc command with:
protoc --plugin=./path/to/protoc-gen-NAME --NAME_out=./test/generated ./test.proto
Note specifically that names are important. This will not work, it will invoke the java generator:
protoc --plugin=./path/to/protoc-gen-NAME --java_out=./test/generated ./test.proto
This will not work, because the binary does not have the correct name:
protoc --plugin=./path/to/whatever-NAME --NAME_out=./test/generated ./test.proto
In order to process the incoming CodeGeneratorRequest and generate a valid response, your binary must itself be able to parse the protobuf message as per the protocol file plugin.proto from the protocolbuffers repository.
Historically this was difficult to do in a self-contained manner, but you can do this 'end-to-end' entirely in rust simply with the protobuf crate, like this trivial example demonstrates:
[dependencies]
protobuf="3.0.2"
use protobuf::plugin::{code_generator_response, CodeGeneratorRequest, CodeGeneratorResponse};
use protobuf::Message;
use std::io;
use std::io::{BufReader, Read, Write};
fn main() {
// Read message from stdin
let mut reader = BufReader::new(io::stdin());
let mut incoming_request = Vec::new();
reader.read_to_end(&mut incoming_request).unwrap();
// Parse as a request
let req = CodeGeneratorRequest::parse_from_bytes(&incoming_request).unwrap();
// Generate the content for each output file
let mut response = CodeGeneratorResponse::new();
for proto_file in req.proto_file.iter() {
let mut output = String::new();
output.push_str(&format!("// from file: {:?}\n", &proto_file.name));
output.push_str(&format!("// package: {:?}\n", &proto_file.package));
for message in proto_file.message_type.iter() {
output.push_str(&format!("\nmessage: {:?}\n", &message.name));
for field in message.field.iter() {
output.push_str(&format!(
"- {:?} {:?} {:?}\n",
field.type_,
field.type_name,
field.name(),
));
}
}
// Add it to the response
let mut output_file = code_generator_response::File::new();
output_file.content = Some(output);
output_file.name = Some(format!("{:?}/out.txt", &proto_file.name.as_ref().unwrap()));
response.file.push(output_file);
}
// Serialize the response to binary message and return it
let out_bytes: Vec<u8> = response.write_to_bytes().unwrap();
io::stdout().write_all(&out_bytes).unwrap();
}
Obviously this trivial example doesn't generate code, just text files, but it shows the basic process. You should also iterate over service and deal with all the additional properties on each type.
What this basically gives you is an AST matching the .proto files; the codegen side of it can be done however you like.
Helpful hints:
Do not log to stdout in your plugin, eg. for debugging. The only permitted output to stdout is a protobuf format CodeGeneratorResponse message.
The plugin does not write files, the protoc command does that; it should generate content and then return an array of files, along with content and metadata.
For more information on plugins, carefully read the plugin.proto file linked above; it has extensive details.