When I am starting the docker daemon I am modifying the dns server so that the containers have a modified /etc/resolv.conf. Looking at the usage message I see:
$ docker --help
Usage: docker [OPTIONS] COMMAND [arg...]
A self-sufficient runtime for linux containers.
Options:
--api-enable-cors=false Enable CORS headers in the remote API
-b, --bridge="" Attach containers to a prexisting network bridge
use 'none' to disable container networking
--bip="" Use this CIDR notation address for the network bridge's IP, not compatible with -b
-D, --debug=false Enable debug mode
-d, --daemon=false Enable daemon mode
--dns=[] Force Docker to use specific DNS servers
--dns-search=[] Force Docker to use specific DNS search domains
-e, --exec-driver="native" Force the Docker runtime to use a specific exec driver
... etc ...
The --dns is what I want to pass, it shows a 'list' with the [], which after much trial and error I finally got this to work:
--dns 127.0.0.1 --dns 8.8.8.8
which deposits :
nameserver 127.0.0.1
nameserver 8.8.8.8
in to the /etc/resolv.conf file.
Is this the correct way to provide a list to docker (and presumably any go) program?
This is a way of passing multiple arguments to a program in Go but certainly not the only way. This is accomplished by defining a type that implements the Value interface. The flag package on flag.Parse() iterates though the argument list matching the name to a registered Value and calling the Set(string) function on the Value. You can use this to append each value of a given name to a slice.
type numList []int
func (l *numList) String() string {
return "[]"
}
func (l *numList) Set(value string) error {
number, err := strconv.Atoi(value)
if err != nil {
return fmt.Errorf("Unable to parse number from value \"%s\"", value)
}
*l = append(*l, number)
return nil
}
This new type can be registered as a flag variable. In the following example the application takes n num command line arguments that are converted to an integer and added to a list.
var numbers numList
func main() {
flag.Var(&numbers, "num", "A number to add to the summation"
flag.Parse()
sum := 0
for _, num := range numbers {
sum += num
}
fmt.Printf("The sum of your flag arguments is %d.\n", sum)
}
This could have been easily done with a string flag and have users pass a delimited list. There is no established convention within the Go language and each application can provide whatever implementation of best fit.
Related
SSRF vul should forbidden local ip request.
How did go net.LookupIP work with Base 8 (octal )?
#go code
net.LookupIP("0177.0.0.01") // NOT_OK 0177.0.0.01
net.LookupIP("0266.075.0310.07") // OK 182.61.200.7
#shell
ping 0177.0.0.01 //OK 127.0.0.1
ping 0266.075.0310.07 // OK 182.61.200.7
input http:address to chrome is the same as ping ,but go not stablize
debug-snapshot-img net.LookupIP("0177.0.0.01") Has Not Decode Base8
debug-snapshot-img ping 0177.0.0.01 in shell Has Decode Base8
debug-snapshot-img net.LookupIP("0266.075.0310.07") Has Decode Base8
The problem
I think you cannot for a simple reason: that A.B.C.D format of IPv4 addresses is called "dot-decimal" (or "dotted-decimal") for a reason: the numbers representing the octets of an address are in the decimal format, and no "special modifiers"—such as 0x, 0, 0o etc—to make it "not decimal" are allowed.
I'm inclined to think that the fact ping is OK to parse the address with some octets in octal is some weird artefact of its implementation; for instance, it also parses hexadecimal:
$ ping 0xAB.0.0.0x01
PING 0xAB.0.0.0x01 (171.0.0.1) 56(84) bytes of data.
^C
Also note that there's no such thing as "the standard ping tool": any OS can have any implementation of it; for instance, ping available on Windows definitely shares no source code with ping available on a typical GNU/Linux-based OS, and that one is probably different from ping implementations available on systems tracing their heritage to *BSD.
To further illustrate my point, consider host:
$ host -t PTR 0177.0.0.01
Host 0177.0.0.01 not found: 3(NXDOMAIN)
This tool, as you can see, failed to cope with the leading 0s.
So, I think it's sensible to assume that the fact net.LookupIP (and net.ParseIP for that matter) definitely cannot be blamed for not supporting "octals" when they parse the dotted-decimal representation of an IPv4 address.
Possible solutions
Possible solutions depend on what really you're after.
If you have to accept IP addresses as strings in such a "dotted-not-so-decimal" format, I'd propose to write a simplistic custom parser which would "preprocess" the strings before calling net.LookupIP on them.
Such a preprocessor would split the string and try to parse each octet, then assemble the address back.
Go has strconv.ParseInt in its standard library, which accepts the "base" argument telling it which base the digits forming the string representation to parse are, and if you pass 0 as the base, the function uses the usual heuristics to guess the base; in particular, the leading 0 signalizes the base 8—just as you'd want it to.
So, we could roll something like this:
package main
import (
"errors"
"fmt"
"net"
"strconv"
"strings"
)
func normalizeIPv4Addr(s string) (string, error) {
parts := strings.SplitN(s, ".", 4)
if len(parts) != 4 || strings.IndexByte(parts[3], '.') != -1 {
return "", errors.New("invalid IPv4 address: invalid number of octets")
}
var dst [4]byte
for i, p := range parts {
n, err := strconv.ParseInt(p, 0, 8)
if err != nil {
return "", fmt.Errorf("invalid IPv4 address: invalid octet %d: %s", i+1, err)
}
if n < 0 || 255 < n {
return "", fmt.Errorf("invalid IPv4 address: invalid octet %d: out of range", i+1)
}
dst[i] = byte(n)
}
return net.IP(dst[:]).String(), nil
}
func main() {
fmt.Println(normalizeIPv4Addr("0177.0.0.01"))
}
Playground.
I tried using clientset.CoreV1().Endpoints(namespace).Get(context.TODO(),name string , metav1.GetOptions{})
endpoints, err2 := clientset.CoreV1().Endpoints(namespace2).Get(context.TODO(), namespace2, metav1.GetOptions{})
if err2 != nil {
log.Println(err2.Error())
}
fmt.Printf("GetPodList There are %v endpoints in the cluster\n", (endpoints))
But I am unsure of the argument to give for name string (the second argument) and metav1.GetOptions{}. (third argument)
You should use the List function rather than Get: List allows you to retrieve multiple endpoints matching certain criteria, Get allows you to retrieve a specific endpoint (by name).
Thus:
endpoints, err := clientset.CoreV1().Endpoints(namespace2).List(context.TODO(), metav1.ListOptions{})
// ...
fmt.Printf("GetPodList there are %v endpoints in the cluster\n", len(endpoints.Items)
If you want all the endpoints in a namespace, you don’t need to specify any list options and passing an empty structure is fine.
go version go1.15.6 windows/amd64
dev os Windows [Version 10.0.19041.630]
I have a Go app in which I am running the AWS CLI using exec.Cmd.Run(). I build out the Cmd class and populate the arguments.
Before I run the Cmd, I use the .String() method to view the command to be ran. If I take this value, copy it to a shell, the command executes with no modifications to the output given to me with no issues reported.
However, when I run the command, it fails returning an error. When I debug the script, it is failing because it says the AWS CLI is saying a parameter is incorrect.
Questions:
Is it possible to see the 100% raw representation of what is being ran? It does not match the return value of .String()
Is there a better way to call an os level command that I am missing?
Real Example:
cmd := &exec.Cmd{
Path: awsPath,
Args: args,
Stdout: &stdout,
Stderr: &stderr,
}
fmt.Printf("Command: %s\n", cmd.String())
// c:\PROGRA~1\Amazon\AWSCLIV2\aws.exe --profile testprofile --region us-east-1 --output json ec2 describe-network-interfaces --filters Name=group-id,Values=sg-abc123
// Running above works 100% of the time if ran from a shell window
err := cmd.Run()
// always errors out saying the format is incorrect
GoPlayground Replication of Issue
https://play.golang.org/p/mvV9VG8F0oz
From cmd.String source:
// String returns a human-readable description of c.
// It is intended only for debugging.
// In particular, it is not suitable for use as input to a shell.
// The output of String may vary across Go releases.
You are seeing the reverse, but the problem is the same: eye-balling a printed command string does not show the exact executable path (is there a rogue space or unprintable character?), same with the arguments (rogue characters?).
Use fmt.Printf("cmd : %q\n", cmd.Path) to show any hidden unicode characters etc. And use the same technique with each of the arguments.
EDIT:
I have found the root cause of your problem you met: os/exec
// Path is the path of the command to run.
//
// This is the only field that must be set to a non-zero
// value. If Path is relative, it is evaluated relative
// to Dir.
Path string
// Args holds command line arguments, including the command as **Args[0]**.
// If the Args field is empty or nil, Run uses {Path}.
//
// In typical use, both Path and Args are set by calling Command.
Args []string
So if you have declare the Cmd.Path := "/usr/local/bin/aws", you have to declare Cmd. Args like this: Args: []string{"", "s3", "help"}, because the Args including the command as Args[0] in above document link.
Final, I think you can exec command like this for simple and effectively:
package main
import (
"bytes"
"fmt"
"os/exec"
)
func main() {
stdout := &bytes.Buffer{}
stderr := &bytes.Buffer{}
name := "/usr/local/bin/aws"
arg := []string{"s3", "help"}
cmd := exec.Command(name, arg...)
cmd.Stderr = stderr
cmd.Stdout = stdout
fmt.Printf("Command: %q\n", cmd.String())
err := cmd.Run()
if err != nil {
fmt.Println("Error: ", stderr.String())
}
fmt.Println("Output: ", stdout.String())
}
=========
$ go run main.go
Command: "/usr/local/bin/aws s3 help"
Done.
I have been trying to run a command and parse the output in golang. Here is a sample of what I am trying to do:
package main
import (
"fmt"
"os/exec"
)
func main() {
out,err := exec.Command("ls -ltr").Output()
if err != nil {
fmt.Println("Error: %s", err)
}
fmt.Printf("%s",out)
}
Now, when I am trying to run "ls -ltr", I get this error:
Error: %s exec: "ls -ltr": executable file not found in $PATH
So, basically go is looking for whole "ls -ltr" in PATH. And it's not there obviously. Is there any way I can pass a flag to any argument?TIA.
You pass arguments to the program by passing more arguments to the function - it's variadic:
out,err := exec.Command("ls","-ltr").Output()
https://golang.org/pkg/os/exec/#Command
This is a pretty common convention with exec-style functions which you will see in most languages. The other common pattern is builders.
Sometimes the layout of arguments you need to pass won't be known at compile-time (though it's not a good idea to send arbitrary commands to the system - stay safe!). If you want to pass an unknown number of arguments, you can use an array with some special syntax:
// Populate myArguments however you like
myArguments := []string{"bar","baz"}
// Pass myArguments with "..." to use variadic behaviour
out,err := exec.Command("foo", myArguments...).Output()
In Node.js I use the nconf module to house environment variables like S3 keys, GCM keys, etc for each of my projects.
I haven't been able to find a similar solution in Go.
What are the generally accepted tools to help manage environment variables for each Go project?
Thanks in advance.
I would strongly recommend using github.com/namsral/flag instead.
It's like the built in flag except you can also supply the parameters via environment variables.
For example, suppose you have this code:
package main
import "fmt"
import "github.com/namsral/flag"
func main() {
var port = 3000
flag.IntVar(&port, "port", port, "Port number")
flag.Parse()
fmt.Println("You seem to prefer", port)
}
Then you can supply the values with either a command line option or an environment variable:
:~/dev/GO$ go run dummy.go
You seem to prefer 3000
:~/dev/GO$ go run dummy.go -port=1234
You seem to prefer 1234
:~/dev/GO$ PORT=4321 go run dummy.go
You seem to prefer 4321
:~/dev/GO$ PORT=4321 go run dummy.go -port=5555
You seem to prefer 5555
This might matter when it's hard to supply command line args. For example, if you use gin to automatically restart a server you have no way to supply command line arguments since gin is just calling go run on the main code without any arguments passed along.
I did some reading on this a while back when I was getting started with Go. According to this link, http://peter.bourgon.org/go-in-production/, they recommend using CLI flags (parameters) instead of environment vars - they even convert environment vars to flags to their CLI apps.
It took some getting used to; but, I really do see the advantages of going pure CLI flags between development, staging and production environments - having specific scripts for each environment.
For example, here's a little web app I wrote recently:
// global flags
var isdebug bool
var port int
var cert string
var key string
var dbdsn string
var dbmaxidle int
var dbmaxopen int
var imguri string
// init is the entry point for the entire web application.
func init() {
log.Println("Starting wwwgo ...")
// setup the flags
//flag.StringVar(&host, "host", "", "Specify a host to redirect to. Use this to redirect all traffic to a single url.")
flag.IntVar(&port, "port", 8080, "Specify the port to listen to.")
flag.BoolVar(&isdebug, "isdebug", false, "Set to true to run the app in debug mode. In debug, it may panic on some errors.")
flag.StringVar(&cert, "cert", "", "Enables listening on 443 with -cert and -key files specified. This must be a full path to the certificate .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.")
flag.StringVar(&key, "key", "", "Enables listening on 443 with -cert and -key files specified. This must be a full path to the key .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.")
flag.StringVar(&dbdsn, "dbdsn", "root:root#tcp(localhost:3306)/dev_db?timeout=5s&tls=false&autocommit=true", "Specifies the MySql DSN connection.")
flag.IntVar(&dbmaxidle, "dbmaxidle", 0, "Sets the database/sql MaxIdleConns.")
flag.IntVar(&dbmaxopen, "dbmaxopen", 500, "Sets the database/sql MaxOpenConns.")
flag.StringVar(&imguri, "imguri", "/cdn/uploads/", "Set this to the full base uri of all images, for example on a remote CDN server or local relative virtual directory.")
flag.Parse()
// log our flags
if isdebug != false {
log.Println("DEBUG mode enabled")
}
if cert != "" && key != "" {
log.Println("Attempting SSL binding with supplied cert and key.")
}
if dbdsn != "" {
log.Printf("Using default dbdsn: %s", dbdsn)
}
...
}
This really becomes nice in staging/production environments.
A quick ./wwwgo -h for "what the heck was that parameter?" gives you full documentation:
admin#dev01:~/code/frontend/src/wwwgo [master]$ ./wwwgo -h
Usage of ./wwwgo:
-cert="": Enables listening on 443 with -cert and -key files specified. This must be a full path to the certificate .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.
-dbdsn="root:root#tcp(localhost:3306)/dev_db?timeout=5s&tls=false&autocommit=true": Specifies the MySql DSN connection.
-dbmaxidle=0: Sets the database/sql MaxIdleConns.
-dbmaxopen=500: Sets the database/sql MaxOpenConns.
-imguri="/cdn/uploads/": Set this to the full base uri of all images, for example on a remote CDN server or local relative virtual directory.
-isdebug=false: Set to true to run the app in debug mode. In debug, it may panic on some errors.
-key="": Enables listening on 443 with -cert and -key files specified. This must be a full path to the key .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.
-port=8080: Specify the port to listen to.
Very nice to have many options at CLI, and no documentation required - it's built into the flags package.
You can clearly see the defaults immediately.
With this type of documentation, I tend to setup all the defaults for common "development environments" that the team uses. We all have root/root access to our local databases. We all are using port 8080 for this particular web app during development, etc. That way, you just have to run:
go build
./wwwgo
And the app runs with all defaults - defaults that are documented. In production, just override the defaults. The built-in flag parsers will panic the application if any parameters are in the wrong format, which is also very nice.
We have used this for a large scale microservice application. This doesn't involve use of any third party libraries, just plain go lang using reflection. It is very simple to use and follows DRY principle. Adding new environment variables and configuring defaults is quite easy. You can also set loading from a config file as a fallback by changing just 2 lines of code. Checkout github page for more info.
package config
import (
"fmt"
"os"
"reflect"
)
/* Tag names to load configuration from environment variable */
const (
ENV = "env"
DEFAULT = "default"
)
type Configuration struct {
Port string `env:"port" default:"3009"`
MongoURL string `env:"MongoUrl" default:"mongodb://localhost:27017/test"`
UserService string `env:"UserService" default:"http://localhost:3005"`
AuthService string `env:"AuthService" default:"http://localhost:3050"`
Debug string `env:"Debug" default:"true"`
}
/* Non-exported instance to avoid accidental overwrite */
var serviceConfig Configuration
func setConfig() {
// ValueOf returns a Value representing the run-time data
v := reflect.ValueOf(serviceConfig)
for i := 0; i < v.NumField(); i++ {
// Get the field tag value
tag := v.Type().Field(i).Tag.Get(ENV)
defaultTag := v.Type().Field(i).Tag.Get(DEFAULT)
// Skip if tag is not defined or ignored
if tag == "" || tag == "-" {
continue
}
a := reflect.Indirect(reflect.ValueOf(serviceConfig))
EnvVar, Info := loadFromEnv(tag, defaultTag)
if Info != "" {
fmt.Println("Missing environment configuration for '" + a.Type().Field(i).Name + "', Loading default setting!")
}
/* Set the value in the environment variable to the respective struct field */
reflect.ValueOf(&serviceConfig).Elem().Field(i).SetString(EnvVar)
}
}
func loadFromEnv(tag string, defaultTag string) (string, string) {
/* Check if the tag is defined in the environment or else replace with default value */
envVar := os.Getenv(tag)
if envVar == "" {
envVar = defaultTag
/* '1' is used to indicate that default value is being loaded */
return envVar, "1"
}
return envVar, ""
}
/*GetConfiguration :Exported function to return a copy of the configuration instance */
func GetConfiguration() Configuration {
return serviceConfig
}
func init() {
setConfig()
fmt.Printf("Service configuration : %+v\n ", serviceConfig)
}
Well I prefer go-arg for setting environment variables. It is easy to use and has nice features.
For example:
package configs
import (
"fmt"
"github.com/alexflint/go-arg"
)
type config struct {
DBHost string `arg:"env:DBHost, -D, --dbhost" help:"Host of the database" placeholder:"DBHost"`
}
var Config config
func main(){
arg.MustParse(&Config)
fmt.Println(Config.DBHost)
}
With this library either you can take the variable from your env or you can pass through args.
export DBHost=127.0.0.1
or
go run ./main.go --dbhost=127.0.0.1
In nodejs, nconf is a great tool. It gives more than storing secret key.
In Golang, as far as i know there is two great package to use .env file easily , godotenv and viper , I prefer godotenv beacuse it's much easier .
Pros: Developers won't see your production secrets. You can use different secrets in dev, test, and production, without having to modify the code.
Cons: Malicious code can read your secrets. The bulk of your application's code is probably open-source libraries. Bad code may creep in without you knowing it.
Example for using godotenv
First run this command inside your terminal,
go get github.com/joho/godotenv
In your .env file
S3_BUCKET=YOURS3BUCKET
SECRET_KEY=YOURSECRETKEYGOESHERE
In your main.go file
package main
import (
"github.com/joho/godotenv"
"log"
"os"
)
func main() {
err := godotenv.Load()
if err != nil {
log.Fatal("Error loading .env file")
}
s3Bucket := os.Getenv("S3_BUCKET")
secretKey := os.Getenv("SECRET_KEY")
// now do something with s3 or whatever
}