exec,Cmd.Run() does not properly run command w/arguments - windows

go version go1.15.6 windows/amd64
dev os Windows [Version 10.0.19041.630]
I have a Go app in which I am running the AWS CLI using exec.Cmd.Run(). I build out the Cmd class and populate the arguments.
Before I run the Cmd, I use the .String() method to view the command to be ran. If I take this value, copy it to a shell, the command executes with no modifications to the output given to me with no issues reported.
However, when I run the command, it fails returning an error. When I debug the script, it is failing because it says the AWS CLI is saying a parameter is incorrect.
Questions:
Is it possible to see the 100% raw representation of what is being ran? It does not match the return value of .String()
Is there a better way to call an os level command that I am missing?
Real Example:
cmd := &exec.Cmd{
Path: awsPath,
Args: args,
Stdout: &stdout,
Stderr: &stderr,
}
fmt.Printf("Command: %s\n", cmd.String())
// c:\PROGRA~1\Amazon\AWSCLIV2\aws.exe --profile testprofile --region us-east-1 --output json ec2 describe-network-interfaces --filters Name=group-id,Values=sg-abc123
// Running above works 100% of the time if ran from a shell window
err := cmd.Run()
// always errors out saying the format is incorrect
GoPlayground Replication of Issue
https://play.golang.org/p/mvV9VG8F0oz

From cmd.String source:
// String returns a human-readable description of c.
// It is intended only for debugging.
// In particular, it is not suitable for use as input to a shell.
// The output of String may vary across Go releases.
You are seeing the reverse, but the problem is the same: eye-balling a printed command string does not show the exact executable path (is there a rogue space or unprintable character?), same with the arguments (rogue characters?).
Use fmt.Printf("cmd : %q\n", cmd.Path) to show any hidden unicode characters etc. And use the same technique with each of the arguments.

EDIT:
I have found the root cause of your problem you met: os/exec
// Path is the path of the command to run.
//
// This is the only field that must be set to a non-zero
// value. If Path is relative, it is evaluated relative
// to Dir.
Path string
// Args holds command line arguments, including the command as **Args[0]**.
// If the Args field is empty or nil, Run uses {Path}.
//
// In typical use, both Path and Args are set by calling Command.
Args []string
So if you have declare the Cmd.Path := "/usr/local/bin/aws", you have to declare Cmd. Args like this: Args: []string{"", "s3", "help"}, because the Args including the command as Args[0] in above document link.
Final, I think you can exec command like this for simple and effectively:
package main
import (
"bytes"
"fmt"
"os/exec"
)
func main() {
stdout := &bytes.Buffer{}
stderr := &bytes.Buffer{}
name := "/usr/local/bin/aws"
arg := []string{"s3", "help"}
cmd := exec.Command(name, arg...)
cmd.Stderr = stderr
cmd.Stdout = stdout
fmt.Printf("Command: %q\n", cmd.String())
err := cmd.Run()
if err != nil {
fmt.Println("Error: ", stderr.String())
}
fmt.Println("Output: ", stdout.String())
}
=========
$ go run main.go
Command: "/usr/local/bin/aws s3 help"
Done.

Related

In Go, why does exec.Command() fail but os.StartProcess() succeed launching "winget.exe"?

exec.Command() works for executing C:\Windows\System32\notepad.exe
But exec.Command() doesn't work for executing C:\Users\<username>\AppData\Local\Microsoft\WindowsApps\winget.exe. Fails with the error message:
exec: "C:\\Users\\<username>\\AppData\\Local\\Microsoft\\WindowsApps\\winget.exe": file does not exist
However, os.StartProcess() works for executing C:\Users\<username>\AppData\Local\Microsoft\WindowsApps\winget.exe
Can someone tell me why?
This code fragment does not work. winget.exe isn't launched.
wingetPath := filepath.Join(os.Getenv("LOCALAPPDATA"),
"Microsoft\\WindowsApps\\winget.exe")
cmd := exec.Command(wingetPath, "--version")
err := cmd.Start()
fmt.Println(err)
// exec: "C:\\Users\\<username>\\AppData\\Local\\Microsoft\\WindowsApps\\winget.exe": file does not exist
But this works:
wingetPath := filepath.Join(os.Getenv("LOCALAPPDATA"),
"Microsoft\\WindowsApps\\winget.exe")
procAttr := new(os.ProcAttr)
procAttr.Files = []*os.File{nil, nil, nil}
// The argv slice will become os.Args in the new process,
// so it normally starts with the program name
_, err := os.StartProcess(wingetPath, []string{wingetPath, "--version"}, procAttr)
fmt.Println(err)
// <nil>
Go version:
> go version
go version go1.18 windows/amd64
Bug in Golang
So apparently this is a bug in the Windows implementation of Go and has been filed on GitHub many times - the oldest I could find is this issue which was filed years ago.
The bug is caused by the fact that exec.Command() internally uses os.Stat() which does not read files with reparse points correctly. os.Lstat() can.
Windows Store apps use App Execution Aliases, which are essentially zero-byte files with reparse points. This post has some additional details.
Workarounds
Workaround is to use os.StartProces() - a lower level API which can be a bit painful to use especially when compared to os.Exec().
Important: In os.StartProcess(), the argv slice will become os.Args in the new process, so you should normally pass the program name as the first argument:
wingetPath := filepath.Join(os.Getenv("LOCALAPPDATA"),
"Microsoft\\WindowsApps\\winget.exe")
procAttr := new(os.ProcAttr)
procAttr.Files = []*os.File{nil, nil, nil}
/*
To redirect IO, pass in stdin, stdout, stderr as required
procAttr.Files = []*os.File{os.Stdin, os.Stdout, os.Stderr}
*/
args = []string { "install", "git.git" }
// The argv slice will become os.Args in the new process,
// so it normally starts with the program name
proc, err := os.StartProcess(wingetPath,
append([]string{wingetPath}, arg...), procAttr)
fmt.Println(err) // nil
Another approach to work around this bug is to (create and) execute a .cmd file (for example) which would (correctly resolve and) execute the file with reparse points. See this (and also this directory) for an example.

Why am I getting a nil pointer error depending on where I call BindPFlag?

I've just recently started working with Go, and I've run into some
behavior working with Cobra and Viper that I'm not sure I understand.
This is a slightly modified version of the sample code you get by
running cobra init. In main.go I have:
package main
import (
"github.com/larsks/example/cmd"
"github.com/spf13/cobra"
)
func main() {
rootCmd := cmd.NewCmdRoot()
cobra.CheckErr(rootCmd.Execute())
}
In cmd/root.go I have:
package cmd
import (
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var cfgFile string
func NewCmdRoot() *cobra.Command {
config := viper.New()
var cmd = &cobra.Command{
Use: "example",
Short: "A brief description of your application",
PersistentPreRun: func(cmd *cobra.Command, args []string) {
initConfig(cmd, config)
},
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("This is a test\n")
},
}
cmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.example.yaml)")
cmd.PersistentFlags().String("name", "", "a name")
// *** If I move this to the top of initConfig
// *** the code runs correctly.
config.BindPFlag("name", cmd.Flags().Lookup("name"))
return cmd
}
func initConfig(cmd *cobra.Command, config *viper.Viper) {
if cfgFile != "" {
// Use config file from the flag.
config.SetConfigFile(cfgFile)
} else {
config.AddConfigPath(".")
config.SetConfigName(".example")
}
config.AutomaticEnv() // read in environment variables that match
// If a config file is found, read it in.
if err := config.ReadInConfig(); err == nil {
fmt.Fprintln(os.Stderr, "Using config file:", config.ConfigFileUsed())
}
// *** This line triggers a nil pointer reference.
fmt.Printf("name is %s\n", config.GetString("name"))
}
This code will panic with a nil pointer reference at the final call to
fmt.Printf:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x6a90e5]
If I move the call to config.BindPFlag from the NewCmdRoot
function to the top of the initConfig command, everything runs
without a problem.
What's going on here? According to the Viper docs regarding the use of
BindPFlags:
Like BindEnv, the value is not set when the binding method is
called, but when it is accessed. This means you can bind as early as
you want, even in an init() function.
That's almost exactly what I'm doing here. At the time I call
config.BindPflag, config is non-nil, cmd is non-nil, and the
name argument has been registered.
I assume there's something going on with my use of config in a
closure in PersistentPreRun, but I don't know exactly why that is
causing this failure.
I thought this was interesting so I did some digging and found your exact problem documented in an issue. The problematic line is this:
config.BindPFlag("name", cmd.Flags().Lookup("name"))
// ^^^^^^^
You created a persistent flag, but bound the flag to the Flags property. If you change your code to bind to PersistentFlags, everything will work as intended even with this line in NewCmdRoot:
config.BindPFlag("name", cmd.PersistentFlags().Lookup("name"))
I don't have any issue if I use cmd.PersistentFlags().Lookup("name").
// *** If I move this to the top of initConfig
// *** the code runs correctly.
pflag := cmd.PersistentFlags().Lookup("name")
config.BindPFlag("name", pflag)
Considering you just registered persistent flags (flag will be available to the command it's assigned to as well as every command under that command), it is safer to call cmd.PersistentFlags().Lookup("name"), rather than cmd.Flags().Lookup("name").
The latter returns nil, since the PersistentPreRun is only called when rootCmd.Execute() is called, which is after cmd.NewCmdRoot().
At cmd.NewCmdRoot() levels, flags have not yet been initialized, even after some were declared "persistent".
This ends up being a little more complex than it might appear at first glance, so while the other answers here helped me resolve the problem I'd like to add a little detail.
There are some nuances in the documentation that aren't particularly clear if you're just starting to work with Cobra. Let's start with the documentation for the PersistentFlags method:
PersistentFlags returns the persistent FlagSet specifically set in the current command.
The key is in ...in the current command. In my NewCmdRoot root method, we can use cmd.PersistentFlags() because the root command is the current command. We can even use cmd.PersistentFlags() in the PersistentPreRun method, as long as we're not processing a subcommand.
If we were to re-write cmd/root.go from the example so that it includes a subcommand, like this...
package cmd
import (
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var cfgFile string
func NewCmdSubcommand() *cobra.Command {
var cmd = &cobra.Command{
Use: "subcommand",
Short: "An example subcommand",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("This is an example subcommand\n")
},
}
return cmd
}
func NewCmdRoot() *cobra.Command {
config := viper.New()
var cmd = &cobra.Command{
Use: "example",
Short: "A brief description of your application",
PersistentPreRun: func(cmd *cobra.Command, args []string) {
initConfig(cmd, config)
},
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("Hello, world\n")
},
}
cmd.PersistentFlags().StringVar(
&cfgFile, "config", "", "config file (default is $HOME/.example.yaml)")
cmd.PersistentFlags().String("name", "", "a name")
cmd.AddCommand(NewCmdSubcommand())
err := config.BindPFlag("name", cmd.PersistentFlags().Lookup("name"))
if err != nil {
panic(err)
}
return cmd
}
func initConfig(cmd *cobra.Command, config *viper.Viper) {
name, err := cmd.PersistentFlags().GetString("name")
if err != nil {
panic(err)
}
fmt.Printf("name = %s\n", name)
if cfgFile != "" {
// Use config file from the flag.
config.SetConfigFile(cfgFile)
} else {
config.AddConfigPath(".")
config.SetConfigName(".example")
}
config.AutomaticEnv() // read in environment variables that match
// If a config file is found, read it in.
if err := config.ReadInConfig(); err == nil {
fmt.Fprintln(os.Stderr, "Using config file:", config.ConfigFileUsed())
}
// *** This line triggers a nil pointer reference.
fmt.Printf("name is %s\n", config.GetString("name"))
}
...we would find that it works when executing the root command:
$ ./example
name =
name is
Hello, world
But it fails when we run the subcommand:
[lars#madhatter go]$ ./example subcommand
panic: flag accessed but not defined: name
goroutine 1 [running]:
example/cmd.initConfig(0xc000172000, 0xc0001227e0)
/home/lars/tmp/go/cmd/root.go:55 +0x368
example/cmd.NewCmdRoot.func1(0xc000172000, 0x96eca0, 0x0, 0x0)
/home/lars/tmp/go/cmd/root.go:32 +0x34
github.com/spf13/cobra.(*Command).execute(0xc000172000, 0x96eca0, 0x0, 0x0, 0xc000172000, 0x96eca0)
/home/lars/go/pkg/mod/github.com/spf13/cobra#v1.1.3/command.go:836 +0x231
github.com/spf13/cobra.(*Command).ExecuteC(0xc00011db80, 0x0, 0xffffffff, 0xc0000240b8)
/home/lars/go/pkg/mod/github.com/spf13/cobra#v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/home/lars/go/pkg/mod/github.com/spf13/cobra#v1.1.3/command.go:897
main.main()
/home/lars/tmp/go/main.go:11 +0x2a
This is because the subcommand inherits the PersistentPreRun command from the root (this is what the Persistent part means), but when this method runs, the cmd argument passwd to PersistentPreRun is no longer the root command; it's the subcommand command. When we try to call cmd.PersistentFlags(), it fails because the current command doesn't have any persistent flags associated with it.
In this case, we need to instead use the Flags method:
Flags returns the complete FlagSet that applies to this command (local and persistent declared here and by all parents).
This gives us access to persistent flags declared by parents.
An additional issue, that doesn't appear to be called out explicitly in the documentation, is that Flags() is only available after command processing has been run (that is, after you call cmd.Execute() on the command or a parent). That means we can use it in PersistentPreRun, but we can't use it in NewCmdRoot (because that method finishes before we process the command line).
TL;DR
We have to use cmd.PersistentFlags() in NewCmdRoot because we're looking for persistent flags applied to the current command, and the value from Flags() won't be available yet.
We need to use cmd.Flags() in PersistentPreRun (and other persistent commands methods) because when processing a subcommand, PersistentFlags will only look for persistent flags on the current command, but won't traverse parents. We need to use cmd.Flags() instead, which will roll up persistent flags declared by parents.

GO: Run cli command with wrong args

I use cobra to create CLI command tool.
everything is looking OK except the error handling
what I want that if command was sent by mistake (wrong args or wrong input) return std.err instead of std.out
to simplify the secnario I've created this which demonstrate my use-case
package main
import (
"errors"
"fmt"
"os"
"github.com/spf13/cobra"
)
var (
RootCmd = &cobra.Command{
Use: "myApp",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("ROOT verbose = %d, args = %v\n", args)
},
}
provideCmd = &cobra.Command{
Use: "provide",
Run: nil,
}
appCmd = &cobra.Command{
Use: "apps",
RunE: func(cmd *cobra.Command, args []string) error {
name := args[0]
if name != "myapp" {
err := errors.New("app name doesnt exist")
return err
}
return nil
},
SilenceUsage: true,
}
)
func init() {
// Add the application command to app command
provideCmd.AddCommand(appCmd)
// add provide command to root command
RootCmd.AddCommand(provideCmd)
}
func main() {
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(-1)
}
}
Now if I compile the binary and run exec.Command against the binary everything is working as expected. but if I want to test the error scenario like mycli provide apps apps1
I want to see that returned in std.err and not at std.out
When I execute mycli provide apps myapp everything should be OK
but if I run mycli provide apps myapp2 I want to get std.err and not std.out , which is not the case here ...what am I missing here ?
https://play.golang.org/p/B00z4eZ7Sj-
Your sample already prints the error both to stdout and stderr.
By default the cobra package prints any errors it encounters to stderr, unless you specifically change that.
So running
./main provide apps something 2> ./stderr.txt creates a text file with the following content (this is what cobra writes to stderr without your intervention):
Error: app name doesnt exist
And running ./main provide apps something > ./stdout.txt - creates a text file with the following content (you printed that yourself with fmt.Println(err), the second line from the bottom in your code):
app name doesnt exist
Which means default behaviour prints errors both to stdout and stderr.
As Devin has advised you, changing the last line to os.Stderr.WriteString(err) or
fmt.Fprintln(os.Stderr, err) (the one I would use) will make your project to print everything to stderr only, which means printing errors twice:
Error: app name doesnt exist
app name doesnt exist
It might be useful to know that cobra allows you some control of error printing behaviour. For example, you can tell a cobra command which stream to print to:
command.SetOutput(os.Stdout) // Defaults to os.Stderr
you could also prevent printing of errors:
command.SilenceErrors = true
or prevent printing of usage text:
command.SilenceUsage = true

How to pass a flag to a command in go lang?

I have been trying to run a command and parse the output in golang. Here is a sample of what I am trying to do:
package main
import (
"fmt"
"os/exec"
)
func main() {
out,err := exec.Command("ls -ltr").Output()
if err != nil {
fmt.Println("Error: %s", err)
}
fmt.Printf("%s",out)
}
Now, when I am trying to run "ls -ltr", I get this error:
Error: %s exec: "ls -ltr": executable file not found in $PATH
So, basically go is looking for whole "ls -ltr" in PATH. And it's not there obviously. Is there any way I can pass a flag to any argument?TIA.
You pass arguments to the program by passing more arguments to the function - it's variadic:
out,err := exec.Command("ls","-ltr").Output()
https://golang.org/pkg/os/exec/#Command
This is a pretty common convention with exec-style functions which you will see in most languages. The other common pattern is builders.
Sometimes the layout of arguments you need to pass won't be known at compile-time (though it's not a good idea to send arbitrary commands to the system - stay safe!). If you want to pass an unknown number of arguments, you can use an array with some special syntax:
// Populate myArguments however you like
myArguments := []string{"bar","baz"}
// Pass myArguments with "..." to use variadic behaviour
out,err := exec.Command("foo", myArguments...).Output()

How to execute shell command

I have some trouble with execute shell commands from a Go program:
var command = pwd + "/build " + file_name
dateCmd := exec.Command(string(command))
dateOut, err := dateCmd.Output()
check(err)
If command variable equals a single word like /home/slavik/project/build (build is shell script) it works, but if I try to pass some arg i.e. /home/slavik/project/build xxx or /home/slavik/project/build -v=1 the Go program raises an exception like file /home/slavik/project/build not found
What's wrong with my code?
You have to pass the program and the arguments separately. See the signature of exec.Command:
func Command(name string, arg ...string) *Cmd
So if you want to pass e.g. -v=1, your call probably should look something like:
dateCmd := exec.Command(pwd + "/build", "-v=1")
Use
exec.Command(pwd + "/build", fileName)

Resources