I have been trying to run a command and parse the output in golang. Here is a sample of what I am trying to do:
package main
import (
"fmt"
"os/exec"
)
func main() {
out,err := exec.Command("ls -ltr").Output()
if err != nil {
fmt.Println("Error: %s", err)
}
fmt.Printf("%s",out)
}
Now, when I am trying to run "ls -ltr", I get this error:
Error: %s exec: "ls -ltr": executable file not found in $PATH
So, basically go is looking for whole "ls -ltr" in PATH. And it's not there obviously. Is there any way I can pass a flag to any argument?TIA.
You pass arguments to the program by passing more arguments to the function - it's variadic:
out,err := exec.Command("ls","-ltr").Output()
https://golang.org/pkg/os/exec/#Command
This is a pretty common convention with exec-style functions which you will see in most languages. The other common pattern is builders.
Sometimes the layout of arguments you need to pass won't be known at compile-time (though it's not a good idea to send arbitrary commands to the system - stay safe!). If you want to pass an unknown number of arguments, you can use an array with some special syntax:
// Populate myArguments however you like
myArguments := []string{"bar","baz"}
// Pass myArguments with "..." to use variadic behaviour
out,err := exec.Command("foo", myArguments...).Output()
Related
This question already has answers here:
Multiple variables of different types in one line in Go (without short variable declaration syntax)
(2 answers)
Closed 1 year ago.
I want to declare two variables of different datatypes(string and error) in a single statement in Go. I do not want to use the short declaration(:=) operator because I like specifying the type of the variable at declaration.
I am following a Go tutorial from the Go docs. I have a function called greetings.Hello() that I am calling from another module. The greetings.Hello() function looks like this:
package greetings
import (
"errors"
"fmt"
)
func Hello(name string) (string, error) {
// If no name was given, return an error with a message
if name == "" {
return "", errors.New("empty name")
}
// If a name was received, return a value
var message string = fmt.Sprintf("Welcome %v!", name)
return message, nil
}
So as you can see, this function returns two values(a string and an error). So ultimately, I would have to assign the result of this function to two variables in the caller. I am calling the greetings.Hello() function from a module named hello. The main function of the hello module's main package looks like this:
package main
import (
"fmt"
"log"
"creating_modules/greetings"
)
func main() {
log.SetPrefix("greetings: ")
log.SetFlags(0)
var message string, err error = greetings.Hello("")
if err !=nil {
log.Fatal(err)
}
fmt.Println(message)
}
The creating_modules/greetings is the greetings module that contains the function Hello(). Most of the gophers tackle it like this:
message, error := greetings.Hello()
But I want to declare the variables along with their datatypes in a single statement. Also the two variables should be assigned the return values of greetings.Hello(). The above mentioned main function of the hello module returns an error when it is run because of the incorrect assignment, referring to this line:
var message string, err error = greetings.Hello("")
The Go compiler returns this error when this code is run using go run:
.\hello.go:14:20: syntax error: unexpected comma at end of statement
This issue can simply be reproduced by copy-pasting the code above(note that the greetings module is a local module so you will need to set the reference path for go tools using go edit -replace)
Another thing to be noted is that my question is different from this question because that question is about declaring variables with the same data type in a single statement whereas mine is about declaring multiple variables with different data types in a single statement.
P.S i won't be surprised to know that Golang does not have this feature
declare the variables along with their datatypes in a single statement
Not possible
Supporting clause from the language spec under Variable declarations
If a type is present, each variable is given that type. Otherwise, each variable is given the type of the corresponding initialization value in the assignment. If that value is an untyped constant, it is first implicitly converted to its default type;
So something like below could work by not specifying either of the types, but you could very well use short variable declarations using := instead
var message, error = greetings.Hello()
But you can declare the variables explicitly with their type information and use the = assignment.
var message string
var err error
if message, err = greetings.Hello(""); err != nil {
log.Fatal(err)
}
go version go1.15.6 windows/amd64
dev os Windows [Version 10.0.19041.630]
I have a Go app in which I am running the AWS CLI using exec.Cmd.Run(). I build out the Cmd class and populate the arguments.
Before I run the Cmd, I use the .String() method to view the command to be ran. If I take this value, copy it to a shell, the command executes with no modifications to the output given to me with no issues reported.
However, when I run the command, it fails returning an error. When I debug the script, it is failing because it says the AWS CLI is saying a parameter is incorrect.
Questions:
Is it possible to see the 100% raw representation of what is being ran? It does not match the return value of .String()
Is there a better way to call an os level command that I am missing?
Real Example:
cmd := &exec.Cmd{
Path: awsPath,
Args: args,
Stdout: &stdout,
Stderr: &stderr,
}
fmt.Printf("Command: %s\n", cmd.String())
// c:\PROGRA~1\Amazon\AWSCLIV2\aws.exe --profile testprofile --region us-east-1 --output json ec2 describe-network-interfaces --filters Name=group-id,Values=sg-abc123
// Running above works 100% of the time if ran from a shell window
err := cmd.Run()
// always errors out saying the format is incorrect
GoPlayground Replication of Issue
https://play.golang.org/p/mvV9VG8F0oz
From cmd.String source:
// String returns a human-readable description of c.
// It is intended only for debugging.
// In particular, it is not suitable for use as input to a shell.
// The output of String may vary across Go releases.
You are seeing the reverse, but the problem is the same: eye-balling a printed command string does not show the exact executable path (is there a rogue space or unprintable character?), same with the arguments (rogue characters?).
Use fmt.Printf("cmd : %q\n", cmd.Path) to show any hidden unicode characters etc. And use the same technique with each of the arguments.
EDIT:
I have found the root cause of your problem you met: os/exec
// Path is the path of the command to run.
//
// This is the only field that must be set to a non-zero
// value. If Path is relative, it is evaluated relative
// to Dir.
Path string
// Args holds command line arguments, including the command as **Args[0]**.
// If the Args field is empty or nil, Run uses {Path}.
//
// In typical use, both Path and Args are set by calling Command.
Args []string
So if you have declare the Cmd.Path := "/usr/local/bin/aws", you have to declare Cmd. Args like this: Args: []string{"", "s3", "help"}, because the Args including the command as Args[0] in above document link.
Final, I think you can exec command like this for simple and effectively:
package main
import (
"bytes"
"fmt"
"os/exec"
)
func main() {
stdout := &bytes.Buffer{}
stderr := &bytes.Buffer{}
name := "/usr/local/bin/aws"
arg := []string{"s3", "help"}
cmd := exec.Command(name, arg...)
cmd.Stderr = stderr
cmd.Stdout = stdout
fmt.Printf("Command: %q\n", cmd.String())
err := cmd.Run()
if err != nil {
fmt.Println("Error: ", stderr.String())
}
fmt.Println("Output: ", stdout.String())
}
=========
$ go run main.go
Command: "/usr/local/bin/aws s3 help"
Done.
I want to test different (correct/incorrect) command line arguments passed to my CLI program, but I am not sure how to achieve this with go/testing package because I am getting flag redefined error. Looks like it happens because flag.Parse() can be called only once. What is the proper approach to test different command line arguments passed into the go program? Is there is any way to define something like setup()/teardown() or run every case in isolation (but in the same file)?
Here is my code:
Function to test:
func (p *Params) Parse() (*Params, error) {
param1Ptr := flag.String("param1", "default", "param1 desc")
param2Ptr := flag.String("param2", "default", "param1 desc")
...
...
flag.Parse()
...
}
Test file:
package main
import (
"os"
"testing"
)
func TestParam1(t *testing.T) {
os.Args = []string{"cmd", "-param1", "incorrect", "-param2", "correct"}
params := Params{}
_, err := params.Parse()
...
...
}
func TestParam2(t *testing.T) {
os.Args = []string{"cmd", "-param1", "correct", "-param2", "incorrect"}
params := Params{}
_, err := params.Parse()
...
...
}
Don't use the global FlagSet object in the flags package. Create your own FlagSet as a field of Params: https://golang.org/pkg/flag/#FlagSet
All that flag.String et al do is pass through the function call to a global FlagSet object in the flag package (specifically flag.CommandLine is the variable). This is easy to use but not a generally good practice. Using your own flagset would avoid the issues you described as well as other potential side effects from using global variables.
Clear the global FlagSet before each test using:
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ExitOnError)
See How to unset flags Visited on command line in GoLang for Tests
I'm facing with a weird golang issue. The following code will clarify:
package main
import (
"os/exec"
"io"
"fmt"
"os"
)
var (
pw io.WriteCloser
pr io.ReadCloser
)
func main() {
term := exec.Command("/bin/sh")
// Get stdin writer pipe
pw, _ = term.StdinPipe()
pr, _ = term.StdoutPipe()
term.Start()
run("cd ~")
pwd := run("pwd");
// Do something with pwd output
...
term.Wait()
}
func run(c string) string {
io.WriteString(pw, fmt.Sprintln(c))
buf := make([]byte, 32 * 1024)
pr.Read(buf)
return string(buf)
}
I'd like to run some commands in a shell env and read their output. There's no problem on write/run command but it seems that there're some limitations while reading:
you can't know if a command doesn't output anything or not;
there's no way to check if stdout is ready to be read or not.
The pr.Read(dest) method will block the code flow until something is read from stdout. As said, the goal is to read sequentially (without using a go routine and/or an infinite loop). This means that if we send a cd command the func end is never reached.
Setting the non-block flag through unix.SetNonblock on stdout file descriptor seems to solve the above issue but you can't know prior if it's ready or not and an error saying "resource temporary not available" is returned from .Read call.
As Cerise Limón mentioned go functions whould be the way to go here, since these sorts of interactive scripting exercises are traditionally done with expect.
You can wrap the the parrellel execution into a library to it might still look like sequencial code, so this might be helpful: https://github.com/ThomasRooney/gexpect
From the readme:
child, err := gexpect.Spawn("python")
if err != nil { panic(err) }
child.Expect(">>>")
child.SendLine("print 'Hello World'")
child.Interact()
child.Close()
I would like to run my program like this:
go run launch.go http://example.com --m=2 --strat=par
"http://example.com" gets interpreted as the first command line argument, which is ok, but the flags are not parsed after that and stay at the default value. If I put it like this:
go run launch.go --m=2 --strat=par http://example.com
then "--m=2" is interpreted as the first argument (which should be the URL).
I could also just remove the os.Args completely, but then I would have only optional flags and I want one (the URL) to be mandatory.
Here's my code:
package main
import (
"fmt"
"webcrawler/crawler"
"webcrawler/model"
"webcrawler/urlutils"
"os"
"flag"
)
func main() {
if len(os.Args) < 2 {
log.Fatal("Url must be provided as first argument")
}
strategy := flag.String("strat", "par", "par for parallel OR seq for sequential crawling strategy")
routineMultiplier := flag.Int("m", 1, "Goroutine multiplier. Default 1x logical CPUs. Only works in parallel strategy")
page := model.NewBasePage(os.Args[1])
urlutils.BASE_URL = os.Args[1]
flag.Parse()
pages := crawler.Crawl(&page, *strategy, *routineMultiplier)
fmt.Printf("Crawled: %d\n", len(pages))
}
I am pretty sure that this should be possible, but I can't figure out how.
EDIT:
Thanks justinas for the hint with the flag.Args(). I now adapted it like this and it works:
...
flag.Parse()
args := flag.Args()
if len(args) != 1 {
log.Fatal("Only one argument (URL) allowed.")
}
page := model.NewBasePage(args[0])
...
os.Args doesn't really know anything about the flag package and contains all command-line arguments. Try flag.Args() (after calling flag.Parse(), of course).
As a followup, to parse flags that follow a command like
runme init -m thisis
You can create your own flagset to skip the first value like
var myValue string
mySet := flag.NewFlagSet("",flag.ExitOnError)
mySet.StringVar(&myValue,"m","mmmmm","something")
mySet.Parse(os.Args[2:])
This tripped me up too, and since I call flag.String/flag.Int64/etc in a couple of places in my app, I didn't want to have to pass around a new flag.FlagSet all over the place.
// If a commandline app works like this: ./app subcommand -flag -flag2
// `flag.Parse` won't parse anything after `subcommand`.
// To still be able to use `flag.String/flag.Int64` etc without creating
// a new `flag.FlagSet`, we need this hack to find the first arg that has a dash
// so we know when to start parsing
firstArgWithDash := 1
for i := 1; i < len(os.Args); i++ {
firstArgWithDash = i
if len(os.Args[i]) > 0 && os.Args[i][0] == '-' {
break
}
}
flag.CommandLine.Parse(os.Args[firstArgWithDash:])
The reason I went with this is because flag.Parse just calls flag.CommandLine.Parse(os.Args[1:]) under the hood anyway.
You can check if the Arg starts with "--" or "-" and avoid using that Arg in a loop.
For example:
for _, file := range os.Args[1:] {
if strings.HasPrefix(file, "--") {
continue
}
//do stuff
}