How to make config file in Golang elegantly? - go

I'm a newbie with Golang.
I want to write a program to manage my Redis instances so that I can create a Redis connection with specific config file. But I don't know how to create the config file for Redis instances elegantly.
I found "text/template" before, is that a good idea?

It depends on the file format you want to support for those configs.
One library able to read most of those format (from a simple ini file to a JSON one) would be spf13/viper:
Viper is a complete configuration solution for go applications. It has been designed to work within an application to handle all types of configuration. It supports
setting defaults
reading from yaml, toml and json config files
reading from environment variables
reading from remote config systems (Etcd or Consul)
reading from command line flags
setting explicit values

Redis configuration files have a simple text format. You can generate a configuration file using the fmt package:
fmt.Fprintf(w, "pidfile %s\n", pidFile)
fmt.Fprintf(w, "port %d\n", port)
where w is io.Writer for the output.
The text/template package is also a viable option. Given the template
pidfile {{.PidFile}}
port {{.Port}}
you can execute it with
t.Execute(w, map[string]interface{}{
"PidFile": pidFile,
"Port": port,
})

If you want to make a config file for development, testing, staging and production, I would recommend to use the // +build possibility offered by Go.
Set up your Go program
You create 4 files in a config packages as followed :
src/program/config
|
|--config_dev.go
|--config_test.go
|--config_staging.go
|--config_prod.go
In the config files
Then in each file, you define the tag used to use that file during the go build (or run, ...) process.
It means for instance in config_dev.go :
// +build dev
package config
// Development ready configuration const based on the build tags.
const (
MYSETTINGS = "dev blablabla"
ISREADY = false
)
In the config_test.go, that would look like :
// +build test
package config
// Development ready configuration const based on the build tags.
const (
MYSETTINGS = "test blablabla"
ISREADY = true
)
Note the // +build dev and // +build test.
That are the tags you are going to use during build to tell which config file you want to use.
In any other Go file, you just have to call config.ISREADY without importing anything else that "config" in your file.
Build
Then to build your app, you just have to run :
go build -tags dev to build with the development config
or
go build -tags test to build with the testing config.

As redis config file has very simple structure I'd suggest you to look at encoding/csv package with Reader.Comma delimiter set just to blank space. It allow you to both read, parse and write configuration easily. Seems to me "slaveof {{.Host}} {{.Port}}" as template looks not very handy. But it's sure correct approach. Just matter of taste.

I would suggest to use some configuration library. I like Viper for is completeness.

Related

Import type definitions from other packages in protobuf

I am trying to create a grpc service with a very basic single action which is GetDeployment, takes a namespace and a name as an input, and returns a Kubernetes deployment. The thing is that I do not want to define my own message for the Deployment as it already exists on the official Kubernetes repository.
I am pretty new to grpc and probably do not understand well enough how it works but can I import this message to my own file in a way I could then write the following .proto file ?
syntax = "proto3";
package api;
import "google/api/annotations.proto";
import "k8s.io/kubernetes/pkg/api/v1/generated.proto";
message GetDeploymentOptions {
string namespace = 1;
string name = 2;
}
service AppsV1 {
rpc GetDeployment(GetDeploymentOptions) returns (k8s.io.kubernetes.pkg.api.v1.Deployment) {}
}
Thank you in advance
GRPC codegen is just a protoc plugin. It generates code for service and rpc but it follows the normal protobuf rules for imports.
In your example, if your file is in src/api.proto and the k8s api repo is a git submodule checked out into thirdparty/k8s.io/api folder you would generate the files you'd need by running:
root>protoc.exe -I thirdparty k8s.io/api/core/v1/generated.proto --go_out=go
root>protoc.exe -I thirdparty src/api.proto --go_out=plugins=grpc:go
The first command is generating the .pb.go file which contains the k8s messages, while the second command is generating the .pb.go file which contains your messages and your service.
Looking at the transient imports of that file, you may also need to checkout api-machinery into k8s.io/apimachinery and run protoc on that file as well.

Get project root path on runtime to read config file

I have a Go project with the following structure and Im struggling to read config file which is located in my project,I need to read the config yaml (which inside the root project) and I should read it inside other package under sub root directory and I got error of not found
myproject
- config.yaml
- cmd
--com
---ftp
----fs.go
Inside the fs.go I need to read the config.yaml and in not having success with it. I try with os.Getwd and also ex, err := os.Executable() and also "../../../" without success, any idea ?
#VonC - suggested to use https://github.com/gobuffalo/packr which can help I guess but the problem is that I need to call it inside the fs.go file and I need to pass this as parameter from the main.go file, is there a better approach ? because I need to pass this parameter in lots of functions...
does viper can help? https://github.com/spf13/viper
My program is CLI program which will be used as bin.
2018: If the binary is built in GOPATH/bin, while your sources are in GOPATH/src, then the relative path would be (at runtime) ../src/myproject.
But a cleaner way would be to embed that file in your binary.
See for instance gobuffalo/packr.
Update Q1 2021: with Go 1.16, you would use the embed package
Go source files that import "embed" can use the //go:embed directive to initialize a variable of type string, []byte, or FS with the contents of files read from the package directory or subdirectories at compile time.
//go:embed hello.txt
var f embed.FS
data, _ := f.ReadFile("hello.txt")
print(string(data))

K8s Go client library fails to find package on go get

We wrote some Go code to talk to our Kubernetes cluster and fetch the IP of a Service exposed. We do it like so:
(import "gopkg.in/kubernetes/kubernetes.v1/pkg/client/restclient")
(import kubectl "gopkg.in/kubernetes/kubernetes.v1/pkg/client/unversioned")
svc, err := c.Services(k8sNS).Get(svcName)
if err != nil {
panic(l.Errorf("Could not retrieve svc details. %s", err.Error()))
}
svcIP := svc.Status.LoadBalancer.Ingress[0].IP
go get works fine, and our script executes when we do go run ... and everybody is happy. Now, as of yesterday (from the time this question is posted) on the same script - go get fails. The error is like so:
[09.07.2016 10:56 AM]$ go get
package k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install: cannot find package "k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install" in any of:
/usr/local/go/src/k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install (from $GOROOT)
/home/ckotha/godir/src/k8s.io/kubernetes/pkg/apis/authentication.k8s.io/install (from $GOPATH)
We have not specifically used authentication package in our code. Are we importing kubernetes libraries correctly? is there another way to do this ?
ls on $GOPATH/k8s.io/kubernetes/pkg/apis/ and found this:
:~/godir/src/k8s.io/kubernetes/pkg/apis
[09.07.2016 10:53 AM]$ ls
abac apps authentication authorization autoscaling batch certificates componentconfig extensions imagepolicy OWNERS policy rbac storage
It looks like a package you imported has changed.
You can update existing repositories:
go get -u
The -u flag instructs get to use the network to update the named
packages and their dependencies. By default, get uses the network to
check out missing packages but does not use it to look for updates to
existing packages.
You do use gopkg.io to pin the version to v1, but I think you want to be more specific, eg, v1.3.6 (EDIT: this won't work because gopkg.in doesn't permit package selectors more specific than the major version.).
Alternatively, a good way to ensure code stays the same is to compile your binary and execute that, instead of using go run.

Go: embed JS files with bindata

This question is a follow up to an earlier question of mine. I've closed the question so I hope its okay that I ask a fresh but related question here. Go: embed static files in binary
How do I serve JS files with go-bindata? Do I pass it into html like this
hi.html
<script>{{.Bindata}}></script>
Doesn't seem to work even though I have no compile or JS errors.
Using https://github.com/elazarl/go-bindata-assetfs
Assuming you have the following structure:
myprojectdirectory
├───api
├───cmd
├───datastores
└───ui
├───css
└───js
Where ui is the directory structure you'd like to wrap up and pack into your app...
Generate a source file
The go-bindata-assetfs tool is pretty simple. It will look at the directories you pass to it and generate a source file with variables that can contain the binary data in those files. So make sure your static files are there, and then run the following command from myprojectdirectory:
go-bindata-assetfs ./ui/...
Now, by default, this will create a source file in the package main. Sometimes, this is ok. In my case, it isn't. You can generate a file with a different package name if you'd like:
go-bindata-assetfs.exe -pkg cmd ./ui/...
Put the source file in the correct location
In this case, the generated file bindata_assetfs.go is created in the myprojectdirectory directory (which is incorrect). In my case, I just manually move the file to the cmd directory.
Update your application code
In my app, I already had some code that served files from a directory:
import (
"net/http"
"github.com/gorilla/mux"
)
// Create a router and setup routes
var Router = mux.NewRouter()
Router.PathPrefix("/ui").Handler(http.StripPrefix("/ui", http.FileServer(http.Dir("./ui"))))
// Start listening
http.ListenAndServe("127.0.0.1:3000", Router)
Make sure something like this works properly, first. Then it's trivial to change the FileServer line to:
Router.PathPrefix("/ui").Handler(http.StripPrefix("/ui", http.FileServer(assetFS())))
Compile the app
Now you have a generated source file with your static assets in them. You can now safely remove the 'ui' subdirectory structure. Compile with
go install ./...
And you should have a binary that serves your static assets properly.
Use https://github.com/elazarl/go-bindata-assetfs
From the readme:
go-bindata-assetfs data/...
In your code setup a route with a file server
http.Handle("/", http.FileServer(assetFS()))
Got my answer here: Unescape css input in HTML
var safeCss = template.CSS(`body {background-image: url("paper.gif");}`)

Using sbt-native-packager, how can I simply prepend a directory to my bash script's ${app_classpath}?

My project uses sbt-native-packager with packageArchetype.java_application.
During sbt stage I have a task that generates some final Typesafe style configuration file that I then copy to:
target/universal/stage/conf/application.conf
I'd like to prepend this directory to the runtime classpath in the bash script, and am looking for the simplest way to do that. I'd hate to maintain a separate src/main/templates/bash-template for something so simple, and am not seeing exactly how to go about it otherwise.
Thanks!
Short Answer
Define a package mapping
mappings in Universal <+= (packageBin in Compile, sourceDirectory ) map {
(_, src) =>
// we are using the reference.conf as default application.conf
// the user can override settings here
val conf = src / "main" / "resources" / "reference.conf"
conf -> "conf/application.conf"
}
Create a jvmopts in src/universal/conf with
-Dconfig.file=/<installation-path>/conf/application.conf
Add to build.sbt
bashScriptConfigLocation := Some("${app_home}/../conf/jvmopts")
Example for server_archetype:
Follow the example application. A bit of description can be found here.
Long answer
Changing the classpath is not supported directly by the sbt-native-packager, because it can cause problems like
classpath ordering
security issues
Like Typesafe Config, most libraries which use config files, provide a parameter to define the location of the configuration file. Use the parameters describe in the documentation.
It seems your are trying to run a server, which means you can use the
packageArchetype.java_server
which is designed to read external configurations. Take a look at the example application how to use it.
The following setting:
scriptClasspath in bashScriptDefines ~= (cp => "../conf" +: cp),
Allows you to do exactly what you need.
In this specific example I prepend the "../conf" directory to the classpath entries.
Also, you need to import the following configuration keys to your build SBT:
import com.typesafe.sbt.packager.Keys.bashScriptDefines
import com.typesafe.sbt.packager.Keys.scriptClasspath
The question you asked in the title is a bit different from the description. I am answering the question in the title here - i.e. how can you prepend a directory into the classpath:
This is very hacky and brittle in that it will probably break if/when the sbt-native-packager makes changes to how the script is generated, but for now it works for me:
private lazy val ClasspathPattern = "declare -r app_classpath=\"(.*)\"\n".r
bashScriptDefines := bashScriptDefines.value.map {
case ClasspathPattern(classpath) => "declare -r app_classpath=\"/path/to/some/external/lib/*:" + classpath + "\"\n"
case _#entry => entry
},

Resources