Does protobuf support embedding non functional metadata into the protobuf schema without affecting the message serialization/de-serialization? I am attempting to embed business information (ownership, contact info) into a large shared protobuf schema but do NOT want to impact functionality at all.
A structured comment or custom_option that does not get serialized would work. I would also like to parse the information from the .proto file for auditing purposes.
TIA
message Bar {
optional int32 a = 1 [(silent_options).owner = "team1", (silent_options).email = "team1#company.com"];
optional int32 b = 2;
}
To do what you want, you will need to use the extend keyword to create a custom FieldOption and then create a protoc plugin. I recommend using Go or C++ for custom plugins, their API is easier.
First we create a custom field option with extend. You can do so like this:
syntax = "proto3";
import "google/protobuf/descriptor.proto";
option go_package = "test.com/proto";
message SilentOptions {
string owner = 1; // maybe an enum if you know all teams
string email = 2;
}
extend google.protobuf.FieldOptions {
SilentOptions silent_options = 1000;
}
There are few things to note. The first one is that we are extending google.protobuf.FieldOptions which is available because we import descriptor.proto. Then, in my case my go module is called "test.com" and this file is in a folder called proto. So the go_package is "test.com/proto".
Once we have this, we need to create a protoc plugin. There are few tutorials on the web but here is a quick solution which is only working for your example:
package main
import (
"log"
"google.golang.org/protobuf/compiler/protogen"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
silentOption "test.com/proto"
)
func main() {
protogen.Options{}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
for _, msg := range sourceFile.Messages {
for _, field := range msg.Fields {
opt := field.Desc.Options().(*descriptorpb.FieldOptions)
if opt == nil { // no FieldOption
continue
}
ext1 := proto.GetExtension(opt, silentOption.E_SilentOptions)
if ext1 != nil { // no SilentOptions
log.Println(ext1)
}
}
}
}
return nil
})
}
For this to run you will need to go get -u google.golang.org/protobuf and run protoc on the silent_options.proto: protoc --go_out=. --go_opt=module=test.com proto/*.proto. For the protoc command to run you need to run the following commands: go get google.golang.org/protobuf/cmd/protoc-gen-go. This is getting the official plugin to generate go code from protobuf.
Finally, build the main.go with a command similar to go build -o protoc-gen-test (note that the name of the binary should start with protoc-gen and then is followed by you plugin name). Then, copy the generated binary to a place in your PATH. And with that you should be able to run: protoc --test_out=. test.proto where test.proto is:
syntax = "proto3";
import "proto/silent_options.proto"; // import the proto file for options
option go_package = "another_test.com"; // can be in a different package
message Bar {
optional int32 a = 1 [
(silent_options) = {
owner: "team1",
email: "team1#company.com"
}
];
optional int32 b = 2;
}
It should print something like:
owner:"team1" email:"team1#company.com"
Obviously this is quite an involved process but I would be happy to update this answer if anything is unclear or if you need more details on the code.
Edit
I forgot to mention that I chose the tag 1000 but there might already have other plugins have the same tag. If you plan to use other plugins later, you should be careful with that and check the tag already in use (protobuf documentation had a page on this, can't find it yet) otherwise you might have conflicts.
Edit 2
Here is the list of the extension tags that are already taken.
I have a system I need to operate as a daemon, which has been developed in go. Ordinarily I could just manage this though something like systemd or even just something as simple as './project &' but this particular project needs to get some input before it can become a daemon - it needs a password to be entered.
This password cannot be provided as command line args, placed in a config file, hard coded in the source etc. It must be manually provided by someone who has knowledge of the password at system startup.
At startup, the project needs to collect the password, verify that things are correct, and only if the correct password has been provided (verified by attempting to decrypt a certificate) can it then actually begin to operate as the daemon.
Is there any way in go for the project to startup, accept input from stdin, perform whatever validation is needed etc, and only then become a daemon process (detaching from stdin etc)?
Currently I can simulate the required behavior by starting it, and after the password has been entered using 'ctrl+z' to suspend it and bg to push the process to the background. Surely there has to be a way of doing that from within the process itself...
here is a program that does what I think you meant.
so the program only checks the password. The actual daemon is webserver which gets called only when password checks out.
package main
import (
"golang.org/x/crypto/ssh/terminal"
"os/exec"
)
func main() {
pwd, err := terminal.ReadPassword(1)
if err != nil {
panic(err)
}
if string(pwd) == "test" {
c := exec.Command("./webserver")
c.Run()
}
return
}
the daemon program I used here was taken from here
https://github.com/sevlyar/go-daemon/blob/master/examples/cmd/gd-simple/simple.go
Its not possible to run a single program both as waiting for input and as a daemon.
You could use flags and carefully control your app flow and run itself as a daemon if a given flag (such as encrypted credentials) is passed. Or even store them in a temporary file, database or anywhere.
func main() {
cred := flag.String("cred", "", "raw cred")
flag.Parse()
if *cred == "" {
fmt.Print("Enter credentials:\n")
decryptedCred, _ := bufio.NewReader(os.Stdin).ReadString('\n')
if !validCred(decryptedCred) {
os.Exit(1)
}
encryptedCred := encryptCredentials(decryptedCred)
cmd := exec.Command("./project", fmt.Sprintf("-cred=%s", encryptedCred), "&")
cmd.Start()
fmt.Printf("Started project with pid: %d\n", cmd.Process.Pid)
os.Exit(0)
}
for {
// start app
// use *cred here
}
}
Whatever the approach I would probably keep track of the pid, etc.
./project
# Enter credentials:
myCredentialsString
# Started project with pid: 11702
ps ax | grep project
# 11667 s001 R 0:02.47 ./project -cred=N/esPq8wsWn4/+Gco16ddl9UnJ0= &\012
Hope this helps
I'm a little confused about the use of sync.Mutex in Go. I understand the basic concept (calling Lock() will prevent other goroutines from executing the code between it and Unlock()), but I'm not sure if I need it here. I've seen a fair few C++ answers for this but in each example they all seem to be modifying and accessing a variable directly.
This is my code from a package called configuration, which I will use throughout the application to get (surprisingly) configuration and settings information.
package config
import (
"encoding/json"
"fmt"
"os"
"sync"
log "github.com/sirupsen/logrus"
)
/*
ConfigurationError is an implementation of the error interface describing errors that occurred while dealing with this
package.
*/
type ConfigurationError string
/*
Error prints the error message for this ConfigurationError. It also implements the error interface.
*/
func (ce ConfigurationError) Error() string {
return fmt.Sprintf("Configuration error: %s", string(ce))
}
/*
configuration is a data struct that holds all of the required information for setting up the program. It is unexported
to prevent other packages creating more instances. Other packages that need settings information should call Current()
to access a copy of the unexported programConfig package variable.
*/
type configuration struct {
sync.RWMutex
LogLevel log.Level `json:"logLevel,omitempty"` //Logging
LogLocation string `json:"logLocation,omitempty"` //-
HttpPort int `json:"port,omitempty"` //Web
LoginUri string `json:"loginUri"` //-
WebBaseUri string `json:"webBaseUri"` //-
StaticBaseUri string `json:"staticBaseUri"` //-
ApiBaseUri string `json:"apiBaseUri"` //-
StaticContentLocalPath string `json:"staticContentLocalPath"` //-
MaxSimultaneousReports int `json:"maxSimultaneousReports"` //Reporting
}
var programConfig configuration
/*
Current returns a copy of the currently loaded program configuration.
*/
func Current() configuration {
programConfig.RLock()
defer programConfig.RUnlock()
return programConfig
}
/*
Load attempts to load a JSON settings file containing a representation of the Configuration struct. It will then set
the value of the package-level config struct to the loaded values. Some settings changes will require a restart of the
server application.
filepath - the full path of the settings file including a leading slash or drive name (depending on the OS).
*/
func Load(filepath string) error {
//Open the file for reading.
settingsFile, err := os.Open(filepath)
if err != nil {
return ConfigurationError(err.Error())
}
defer settingsFile.Close()
//Decode JSON into package level var.
decoder := json.NewDecoder(settingsFile)
newSettings := configuration{}
err = decoder.Decode(&newSettings)
if err != nil {
return ConfigurationError(err.Error())
}
programConfig.Lock() //I'm not 100% sure this is the correct use of a mutex for this situation, so check up on that.
programConfig = newSettings
programConfig.Unlock()
return nil
}
As you can see, I've used mutex in two places.
In Current(). Do I need this here if the function is not returning a pointer but a copy of the programConfig variable? The only way the underlying package variable will be modified is through the Load() function.
In the Load() function. This can be called at any time by any goroutine, although it rarely will be.
Given that, am I using them correctly and why do I need one when reading a copy of the data (if so)?
When you read data which can be written at the same time you need a mutex. Otherwise it might happen that you read while it's written and get half of the old data and half of the new data.
So your example seems to be just fine. Because you are probably reading the config very often but writing it rarely your usage of a RWLock makes sense. This means that multiple threads can read at the same time as long as it's not written.
What in your code looks dangerous is:
programConfig.Lock()
programConfig = newSettings
programConfig.Unlock()
Because programConfig contains the Mutex you are doing the Lock and Unlock on different instances which will lead to deadlocks. You should add the Mutex to the "parent" of the instance. In this case probably the package.
How can import a file realtive to the revel basefolder in revel framework. Currently I do the following to get hold of some config values.
file, err := ioutil.ReadFile("conf/config.conf")
...
This results in my server only working if i stand in the app directory when starting revel with
revel run myapp
Is there a way to access the base folder?
There are exported global variables in the revel package, you can use any of these:
var (
// App details
AppName string // e.g. "sample"
BasePath string // e.g. "/Users/revel/gocode/src/corp/sample"
AppPath string // e.g. "/Users/revel/gocode/src/corp/sample/app"
ViewsPath string // e.g. "/Users/revel/gocode/src/corp/sample/app/views"
ImportPath string // e.g. "corp/sample"
SourcePath string // e.g. "/Users/revel/gocode/src"
// Revel installation details
RevelPath string // e.g. "/Users/revel/gocode/src/revel"
// Where to look for templates and configuration.
// Ordered by priority. (Earlier paths take precedence over later paths.)
CodePaths []string
ConfPaths []string
TemplatePaths []string
)
If it is empty for you, that is most likely because you started your app from the base folder.
Note that these paths are set by the Init(mode, importPath, srcPath string) function. It's documentation states:
srcPath - the path to the source directory, containing Revel and the app.
If not specified (""), then a functioning Go installation is required.
Also check out: how to reference a relative file from code and tests
I use this method:
In conf/app.conf add a line with the configuration path of this way:
projectname.path = "/foldersnames/"
and in the controller Write a method like this:
func info(field string) string {
config, err := revel.LoadConfig("app.conf")
if err != nil || config == nil {
log.Fatalln("Failed to load configuration file", err)
}
return config.StringDefault(field, "empty")
}
You can build a helper with this code and take configurations variables from all applications.
You must call of this way:
info("projectname.path")
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
The community reviewed whether to reopen this question 12 months ago and left it closed:
Opinion-based Update the question so it can be answered with facts and citations by editing this post.
Improve this question
What is the preferred way to handle configuration parameters for a Go program (the kind of stuff one might use properties files or ini files for, in other contexts)?
The JSON format worked for me quite well. The
standard library offers methods to write the data structure indented, so it is quite
readable.
See also this golang-nuts thread.
The benefits of JSON are that it is fairly simple to parse and human readable/editable
while offering semantics for lists and mappings (which can become quite handy), which
is not the case with many ini-type config parsers.
Example usage:
conf.json:
{
"Users": ["UserA","UserB"],
"Groups": ["GroupA"]
}
Program to read the configuration
import (
"encoding/json"
"os"
"fmt"
)
type Configuration struct {
Users []string
Groups []string
}
file, _ := os.Open("conf.json")
defer file.Close()
decoder := json.NewDecoder(file)
configuration := Configuration{}
err := decoder.Decode(&configuration)
if err != nil {
fmt.Println("error:", err)
}
fmt.Println(configuration.Users) // output: [UserA, UserB]
Another option is to use TOML, which is an INI-like format created by Tom Preston-Werner. I built a Go parser for it that is extensively tested. You can use it like other options proposed here. For example, if you have this TOML data in something.toml
Age = 198
Cats = [ "Cauchy", "Plato" ]
Pi = 3.14
Perfection = [ 6, 28, 496, 8128 ]
DOB = 1987-07-05T05:45:00Z
Then you can load it into your Go program with something like
type Config struct {
Age int
Cats []string
Pi float64
Perfection []int
DOB time.Time
}
var conf Config
if _, err := toml.DecodeFile("something.toml", &conf); err != nil {
// handle error
}
Viper is a golang configuration management system that works with JSON, YAML, and TOML. It looks pretty interesting.
I usually use JSON for more complicated data structures. The downside is that you easily end up with a bunch of code to tell the user where the error was, various edge cases and what not.
For base configuration (api keys, port numbers, ...) I've had very good luck with the gcfg package. It is based on the git config format.
From the documentation:
Sample config:
; Comment line
[section]
name = value # Another comment
flag # implicit value for bool is true
Go struct:
type Config struct {
Section struct {
Name string
Flag bool
}
}
And the code needed to read it:
var cfg Config
err := gcfg.ReadFileInto(&cfg, "myconfig.gcfg")
It also supports slice values, so you can allow specifying a key multiple times and other nice features like that.
Just use standard go flags with iniflags.
go flags
import "flag"
var nFlag = flag.Int("n", 1234, "help message for flag n")
iniflags
package main
import (
"flag"
...
"github.com/vharitonsky/iniflags"
...
)
var (
flag1 = flag.String("flag1", "default1", "Description1")
...
flagN = flag.Int("flagN", 123, "DescriptionN")
)
func main() {
iniflags.Parse() // use instead of flag.Parse()
}
Standard go flags have the following benefits:
Idiomatic.
Easy to use. Flags can be easily added and scattered across arbitrary packages your project uses.
Flags have out-of-the-box support for default values and description.
Flags provide standard 'help' output with default values and description.
The only drawback standard go flags have - is management problems when the number of flags used in your app becomes too large.
Iniflags elegantly solves this problem: just modify two lines in your main package and it magically gains support for reading flag values from ini file. Flags from ini files can be overriden by passing new values in command-line.
See also https://groups.google.com/forum/#!topic/golang-nuts/TByzyPgoAQE for details.
I have started using Gcfg which uses Ini-like files. It's simple - if you want something simple, this is a good choice.
Here's the loading code I am using currently, which has default settings and allows command line flags (not shown) that override some of my config:
package util
import (
"code.google.com/p/gcfg"
)
type Config struct {
Port int
Verbose bool
AccessLog string
ErrorLog string
DbDriver string
DbConnection string
DbTblPrefix string
}
type configFile struct {
Server Config
}
const defaultConfig = `
[server]
port = 8000
verbose = false
accessLog = -
errorLog = -
dbDriver = mysql
dbConnection = testuser:TestPasswd9#/test
dbTblPrefix =
`
func LoadConfiguration(cfgFile string, port int, verbose bool) Config {
var err error
var cfg configFile
if cfgFile != "" {
err = gcfg.ReadFileInto(&cfg, cfgFile)
} else {
err = gcfg.ReadStringInto(&cfg, defaultConfig)
}
PanicOnError(err)
if port != 0 {
cfg.Server.Port = port
}
if verbose {
cfg.Server.Verbose = true
}
return cfg.Server
}
have a look at gonfig
// load
config, _ := gonfig.FromJson(myJsonFile)
// read with defaults
host, _ := config.GetString("service/host", "localhost")
port, _ := config.GetInt("service/port", 80)
test, _ := config.GetBool("service/testing", false)
rate, _ := config.GetFloat("service/rate", 0.0)
// parse section into target structure
config.GetAs("service/template", &template)
https://github.com/spf13/viper and https://github.com/zpatrick/go-config are a pretty good libraries for configuration files.
Use toml like this article Reading config files the Go way
I wrote a simple ini config library in golang.
https://github.com/c4pt0r/cfg
goroutine-safe, easy to use
package cfg
import (
"testing"
)
func TestCfg(t *testing.T) {
c := NewCfg("test.ini")
if err := c.Load() ; err != nil {
t.Error(err)
}
c.WriteInt("hello", 42)
c.WriteString("hello1", "World")
v, err := c.ReadInt("hello", 0)
if err != nil || v != 42 {
t.Error(err)
}
v1, err := c.ReadString("hello1", "")
if err != nil || v1 != "World" {
t.Error(err)
}
if err := c.Save(); err != nil {
t.Error(err)
}
}
===================Update=======================
Recently I need an INI parser with section support, and I write a simple package:
github.com/c4pt0r/cfg
u can parse INI like using "flag" package:
package main
import (
"log"
"github.com/c4pt0r/ini"
)
var conf = ini.NewConf("test.ini")
var (
v1 = conf.String("section1", "field1", "v1")
v2 = conf.Int("section1", "field2", 0)
)
func main() {
conf.Parse()
log.Println(*v1, *v2)
}
You might also be interested in go-libucl, a set of Go bindings for UCL, the Universal Configuration Language. UCL is a bit like JSON, but with better support for humans: it supports comments and human-readable constructs like SI multipliers (10k, 40M, etc.) and has a little bit less boilerplate (e.g., quotes around keys). It's actually pretty close to the nginx configuration file format, if you're already familiar with that.
I agree with nemo and I wrote a little tool to make it all real easy.
bitbucket.org/gotamer/cfg is a json configuration package
You define your config items in your application as a struct.
A json config file template from your struct is saved on the first run
You can save runtime modifications to the config
See doc.go for an example
I tried JSON. It worked. But I hate having to create the struct of the exact fields and types I might be setting. To me that was a pain. I noticed it was the method used by all the configuration options I could find. Maybe my background in dynamic languages makes me blind to the benefits of such verboseness. I made a new simple config file format, and a more dynamic-ish lib for reading it out.
https://github.com/chrisftw/ezconf
I am pretty new to the Go world, so it might not be the Go way. But it works, it is pretty quick, and super simple to use.
Pros
Super simple
Less code
Cons
No Arrays or Map types
Very flat file format
Non-standard conf files
Does have a little convention built-in, which I now if frowned upon in general in Go community. (Looks for config file in the config directory)