This question already has answers here:
Executing a Bash Script from Golang
(4 answers)
Closed 3 months ago.
I want to create a wrapper program that can wrapper whatever shell commands user provides, like:
./wrapper "cmd1 && cmd2"
In Python, I can call os.system("cmd1 && cmd2"). But Golang's exec.Command needs a list for command and args. Is there way in Golang to archive the same as Python's os.system()?
os/exec https://pkg.go.dev/os/exec
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("/usr/bin/bash", "-c", os.Args[1])
output, err := cmd.CombinedOutput()
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
$ go run main.go "ls -alh && pwd"
total 4.0K
drwxr-xr-x 2 xxxx xxxx 120 Nov 14 11:12 .
drwxrwxrwt 17 root root 420 Nov 14 11:42 ..
-rw-r--r-- 1 xxxx xxxx 217 Nov 14 11:42 main.go
/tmp/stk
Related
I am following : Containers From Scratch • Liz Rice • GOTO 2018
and test the source code lizrice/containers-from-scratch locally to learn containers.
But with code below , I am not able to fork a child process on Ubuntu 1804, below are my main.go
package main
import (
"fmt"
"os"
"os/exec"
"syscall"
)
// go run main.go run <cmd> <args>
func main() {
switch os.Args[1] {
case "run":
run()
case "child":
child()
default:
panic("help")
}
}
func run() {
fmt.Printf("In <Run> Running %v \n", os.Args[2:])
cmd := exec.Command("/proc/self/exe", append([]string{"child"}, os.Args[2:]...)...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.SysProcAttr = &syscall.SysProcAttr{
Cloneflags: syscall.CLONE_NEWUTS,
}
must(cmd.Run())
}
func child() {
fmt.Printf("In <child> Running %v \n", os.Args[2:])
cmd := exec.Command(os.Args[2], os.Args[3:]...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
syscall.Sethostname([]byte("container"))
must(cmd.Run())
}
func must(err error) {
if err != nil {
panic(err)
}
}
I tried to run it :
go run main1.go run /bin/bash
Error below occurred:
In <Run> Running [/bin/bash]
panic: fork/exec /proc/self/exe: operation not permitted
goroutine 1 [running]:
main.must(...)
/home/jia/cs_study_plan/docker/containers-from-scratch/main1.go:72
main.run()
/home/jia/cs_study_plan/docker/containers-from-scratch/main1.go:44 +0x27f
main.main()
/home/jia/cs_study_plan/docker/containers-from-scratch/main1.go:17 +0x4d
exit status 2
Any suggestion how can I fix this permission issue in my main.go ?
Thank you #Peter for suggestions.
I could run my app use 'go build' and 'sudo'
Here is what I do
1. go to directory of main1.go
2. go mod init main1 <it must be the same name as main1.go>
3. go mod tidy <it is optional step>
4. gp build
5. main1 is build in current directory
6. run command :
sudo ./main1 run /bin/bash
7. child forked successfully , as I can tell from my output
In <Run> Running [/bin/bash]
In <child> Running [/bin/bash]
ps
PID TTY TIME CMD
15774 pts/0 00:00:00 sudo
15775 pts/0 00:00:00 main1
15781 pts/0 00:00:00 exe
15786 pts/0 00:00:00 bash
16047 pts/0 00:00:00 ps
Also I tried to disable 'AppArmor' in my Ubunut , see if that is what blocks main1.go from fork child process
$sudo systemctl status apparmor
[sudo] password for jia:
● apparmor.service - AppArmor initialization
Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2022-06-09 08:31:21 CST; 2h 3min ago
Docs: man:apparmor(7)
http://wiki.apparmor.net/
Main PID: 660 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/apparmor.service
Jun 09 08:31:21 ub1804 apparmor[660]: * Starting AppArmor profiles
Jun 09 08:31:21 ub1804 apparmor[660]: Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox
Jun 09 08:31:21 ub1804 apparmor[660]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Jun 09 08:31:21 ub1804 systemd[1]: Starting AppArmor initialization...
Jun 09 08:31:21 ub1804 apparmor[660]: ...done.
Jun 09 08:31:21 ub1804 systemd[1]: Started AppArmor initialization.
$sudo systemctl stop apparmor
$go run main1.go run echo hello
In <Run> Running [echo hello]
panic: fork/exec /proc/self/exe: operation not permitted
NO luck , if anyone knows what stops my main1.go to fork child , please let me know , thanks in advance
Your problem appears because Ubuntu has sudo preinstalled so your user has restricted rights. According to proc documentation, it provides an interface for kernel data structures and represents a pseudo-filesystem. The /proc is mounted by the system itself as read-only, but you are trying to create a new process inside this file system which is read-only for your current user.
You can read more about it in the man https://man7.org/linux/man-pages/man5/proc.5.html
TL;DR: you need to run your program as a root.
I am connecting to pod via client-Go and I want to get the properties of the file directory
func GetPodFiles(c *gin.Context) {
client, _ := Init.ClusterID(c)
path := c.DefaultQuery("path", "/")
cmd := []string{
"sh",
"-c",
fmt.Sprintf("ls -l %s", path),
}
config, _ := Init.ClusterCfg(c)
req := client.CoreV1().RESTClient().Post().
Resource("pods").
Name("nacos-0").
Namespace("default").SubResource("exec").Param("container", "nacos")
req.VersionedParams(
&v1.PodExecOptions{
Command: cmd,
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
},
scheme.ParameterCodec,
)
var stdout, stderr bytes.Buffer
exec, err := remotecommand.NewSPDYExecutor(config, "POST", req.URL())
if err != nil {
response.FailWithMessage(response.InternalServerError, err.Error(), c)
return
}
err = exec.Stream(remotecommand.StreamOptions{
Stdin: nil,
Stdout: &stdout,
Stderr: &stderr,
})
if err != nil {
response.FailWithMessage(response.InternalServerError, "Error obtaining file", c)
return
}
fmt.Println(stdout.String())
}
Execution Result Output
total 0
lrwxrwxrwx 1 root root 7 Jun 1 2018 bin -> usr/bin
drwxr-xr-x 5 root root 360 Feb 16 16:39 dev
lrwxrwxrwx 1 root root 8 Jun 1 2018 sbin -> usr/sbin
drwxr-xr-x 2 root root 6 Apr 11 2018 srv
Expect the result
"data": [
{
"perm": "drwxr-xr-x",
"mod_time": "2022-03-02 15:02:15",
"kind": "d",
"name": "temp",
"size": ""
},
]
Is there a good way or a golang third-party library to handle it. Please let me know. Thank you
In a Kubernetes pod you can execute the stat linux command instead of ls command.
$ stat yourFileOrDirName
The output of this command by default is like this:
File: yourFileOrDirName
Size: 346 Blocks: 0 IO Block: 4096 directory
Device: 51h/82d Inode: 40431 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 1000/ username) Gid: ( 1000/ groupname)
Access: 2022-03-02 11:59:07.384821351 +0100
Modify: 2022-03-02 11:58:48.733821177 +0100
Change: 2022-03-02 11:58:48.733821177 +0100
Birth: 2021-12-21 11:12:05.571841723 +0100
But you can tweak its output like this:
$ stat --printf="%n,%A,%y,%s" yourFileOrDirName
where %n - file name, %A - permission bits and file type in human readable form, %y - time of last data modification human-readable, %s - total size, in bytes. You can also choose any character as a delimiter instead of comma.
the output will be:
yourFileOrDirName,drwxr-xr-x,2022-03-02 11:58:48.733821177 +0100,346
See more info about the stat command here.
After you get such output, I believe you can easily 'convert' it to json format if you really need it.
Furthermore, you can run the stat command like this:
$ stat --printf="{\"data\":[{\"name\":\"%n\",\"perm\":\"%A\",\"mod_time\":\"%y\",\"size\":\"%s\"}]}" yourFileOrDirName
Or as #mdaniel suggested, since the command does not contain any shell variables, nor a ', the cleaner command is:
stat --printf='{"data":[{"name":"%n","perm":"%A","mod_time":"%y","size":"%s"}]}' yourFileOrDirName
and get the DIY json output:
{"data":[{"name":"yourFileOrDirName","perm":"drwxrwxr-x","mod_time":"2022-02-04 15:17:27.000000000 +0000","size":"4096"}]}
So I've tried to execute a chain i.e. multiple commands on a Pod's container using client-go, and it seems to only work for some commands like ls.
Here is what I've tried:
req := client.CoreV1().RESTClient().Post().Resource("pods").Name(pod.Name).Namespace(pod.ObjectMeta.Namespace).SubResource("exec") // .Param("container", containerName)
scheme := runtime.NewScheme()
if err := _v1.AddToScheme(scheme); err != nil {
panic(err.Error())
}
parameterCodec := runtime.NewParameterCodec(scheme)
req.VersionedParams(&_v1.PodExecOptions{
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
Container: containerName,
Command: strings.Fields("/bin/sh -c " + command),
}, parameterCodec)
exec, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
if err != nil {
panic(err.Error())
}
var stdout, stderr bytes.Buffer
err = exec.Stream(remotecommand.StreamOptions{
Stdin: nil,
Stdout: &stdout,
Stderr: &stderr,
Tty: false,
})
if err != nil {
panic(err.Error())
}
log.Printf("Output from pod: %v", stdout.String())
log.Printf("Error from pod: %v", stderr.String())
When the command variable is just a simple ls -l, I get the desired output. But when I try to do something like 'ls -l && echo hello' it produces an error command terminated with exit code 2.
It doesn't output anything if I only put echo hello. However, it does produce the desired output hello if I remove the Bourne Shell prefix /bin/sh -c and the Command attribute equals to string.Fields("echo hello"), but this approach doesn't let me chain commands.
All in all, what I am trying to do is to execute a chain of commands on a Pod's container.
The corev1.PodExecOptions.Command accept value of []string type.
req.VersionedParams(&_v1.PodExecOptions{
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
Container: containerName,
Command: cmds,
}, parameterCodec)
where, cmds can be:
cmds := []string{
"sh",
"-c",
"echo $HOME; ls -l && echo hello",
}
Output:
/root
total 68
drwxr-xr-x 2 root root 4096 Feb 24 00:00 bin
drwxr-xr-x 2 root root 4096 Feb 1 17:09 boot
drwxr-xr-x 2 root root 4096 Feb 24 00:00 mnt
drwxr-xr-x 2 root root 4096 Feb 24 00:00 opt
dr-xr-xr-x 396 root root 0 Mar 19 11:47 proc
drwx------ 2 root root 4096 Feb 24 00:00 root
.
.
hello
Explanation: Tim's answer
command: ["/bin/sh","-c"]
args: ["command one; command two && command three"]
The command ["/bin/sh", "-c"] says "run a shell, and execute the following instructions". The args are then passed as commands to the shell. In shell scripting a semicolon separates commands, and && conditionally runs the following command if the first succeed. In the above example, it always runs command one followed by command two, and only runs command three if command two succeeded.
N.B.:
For bash, it will be similar to something like below:
cmds := []string{
"bash",
"-c",
`export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}
mysql -h localhost -nsLNE -e "select 1;" 2>/dev/null | grep -v "*"`,
},
I have a file that was generated using the Go compress/gzip package with code like
payload := bytes.NewBuffer(nil)
gw := gzip.NewWriter(payload)
tw := tar.NewWriter(gw)
...
tw.Close()
gw.Close()
How can I unzip this file from the command line on Mac? I tried gunzip but it fails
$ gunzip test.gz
gunzip: test.gz: not in gzip format
Also tried following without luck
$ tar -xvf test.gz
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Your issue is that you are never writing to your tar writer, in your example code. In order to produce a valid targz file with content, you need to keep in mind that:
Add at least one file needs to be in the tarball
Each file requires headers
You'll need to use tar.WriteHeader to create the header for each file, and you can then simply write the content of the files as bytes through a call to tar.Write.
You can then untar it using tar -xvf test.tgz like you mentioned in your previous example.
Here is a sample code that I quickly wrote on my machine for the sake of demonstration:
package main
import (
"compress/gzip"
"archive/tar"
"os"
"fmt"
"time"
)
func main() {
// Create targz file which will contain other files.
file, err := os.Create("test.tgz")
if err != nil {
panic(err)
}
gw := gzip.NewWriter(file)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// Create file(s) in targz
if err := addFile(tw, "myfile.test", "example content"); err != nil {
panic(err)
}
}
func addFile(tw *tar.Writer, fileName, content string) error {
header := &tar.Header{
Name: fileName,
Size: int64(len(content)),
Mode: 0655,
ModTime: time.Now(),
}
err := tw.WriteHeader(header)
if err != nil {
return fmt.Errorf("could not write header for file %q: %w", fileName, err)
}
_, err = tw.Write([]byte(content))
if err != nil {
return fmt.Errorf("could not write content for file %q: %w", fileName, err)
}
return nil
}
And here is the result:
$> go run main.go
$> ls -la
total 5
drwxr-xr-x 12 ullaakut staff 384 Feb 12 05:34 ./
drwxr-xr-x 29 ullaakut staff 928 Jan 28 14:56 ../
-rw-r--r--# 1 ullaakut staff 6148 Dec 25 13:01 .DS_Store
-rw-r--r-- 1 ullaakut staff 888 Feb 12 05:34 main.go
-rw-r--r-- 1 ullaakut staff 121 Feb 12 05:34 test.tgz
$> tar -zxvf test.tgz
x myfile.test
$> cat myfile.test
example content
As you can see, the archive is extracted and uncompressed just fine :)
I tried to run the following Go code:
package main
import (
"fmt"
"io/ioutil"
)
func main() {
items, err := ioutil.ReadDir("/dev/fd")
if err != nil {
panic(err)
}
fmt.Println(items)
}
I just get this error:
panic: lstat /dev/fd/4: bad file descriptor
goroutine 1 [running]:
main.main()
/Users/andy/Desktop/demo.go:11 +0xe8
exit status 2
The /dev/fd folder definitely exists, and there is a /dev/fd/4 inside there when I ls it.
$ ls -Al /dev/fd
total 9
crw--w---- 1 andy tty 16, 4 Jan 25 00:16 0
crw--w---- 1 andy tty 16, 4 Jan 25 00:16 1
crw--w---- 1 andy tty 16, 4 Jan 25 00:16 2
dr--r--r-- 3 root wheel 4419 Jan 23 20:42 3/
dr--r--r-- 1 root wheel 0 Jan 23 20:42 4/
What's going on? Why can't I read this directory? I'm trying to port the ls command to Go here, so I would like to be able to read this directory in order to produce similar output to ls.
EDIT: I am running everything as non-root user. The executable bit on /dev/fd is set.
$ ls -al /dev | grep fd
dr-xr-xr-x 1 root wheel 0 Jan 23 20:42 fd/
$ stat /dev/fd/4 # same result with -L flag
stat: /dev/fd/4: stat: Bad file descriptor
First, let's remember that /dev/fd is special. Its contents are different for every process (since they reflect the file descriptors of that process), so it doesn't really mean anything that ls can list it because its contents will be different for ls and your program.
Anyway, here's a slightly updated version of your program where instead of letting ioutil do things behind our back, we do it ourselves to see what's going on:
package main
import (
"fmt"
"time"
"os"
"log"
)
func main() {
d, err := os.Open("/dev/fd")
if err != nil {
log.Fatal(err)
}
names, err := d.Readdirnames(0)
if err != nil {
log.Fatal(err)
}
for _, n := range names {
n = "/dev/fd/" + n
fmt.Printf("file: %s\n", n)
_, err := os.Lstat(n)
if err != nil {
fmt.Printf("lstat error: %s - %v\n", n, err)
}
}
time.Sleep(time.Second * 200)
}
And then when run it gives me:
file: /dev/fd/0
file: /dev/fd/1
file: /dev/fd/2
file: /dev/fd/3
file: /dev/fd/4
lstat error: /dev/fd/4 - lstat /dev/fd/4: bad file descriptor
So this is indeed the same problem. I added the sleep at the end so that the process doesn't die so that we can debug which file descriptors it has. In another terminal:
$ lsof -p 7861
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
foo 7861 art cwd DIR 1,4 2272 731702 /Users/art/src/go/src
foo 7861 art txt REG 1,4 1450576 8591078117 /private/var/folders/m7/d614cd9x61s0l3thb7cf3rkh0000gn/T/go-build268777304/command-line-arguments/_obj/exe/foo
foo 7861 art txt REG 1,4 837248 8590944844 /usr/lib/dyld
foo 7861 art 0u CHR 16,4 0t8129 645 /dev/ttys004
foo 7861 art 1u CHR 16,4 0t8129 645 /dev/ttys004
foo 7861 art 2u CHR 16,4 0t8129 645 /dev/ttys004
foo 7861 art 3r DIR 37,7153808 0 316 /dev/fd
foo 7861 art 4u KQUEUE count=0, state=0x8
We can see that fd 4 is a KQUEUE. Kqueue file descriptors are used for managing events on OSX, similar to epoll on Linux if you know that mechanism.
It appears that OSX does not allow stat on kqueue file descriptors in /dev/fd which we can verify with:
$ cat > foo.c
#include <sys/types.h>
#include <sys/event.h>
#include <sys/stat.h>
#include <stdio.h>
#include <err.h>
int
main(int argc, char **argv)
{
int fd = kqueue();
char path[16];
struct stat st;
snprintf(path, sizeof(path), "/dev/fd/%d", fd);
if (lstat(path, &st) == -1)
err(1, "lstat");
return 0;
}
$ cc -o foo foo.c && ./foo
foo: lstat: Bad file descriptor
$
Go programs on OSX need a kqueue to handle various events (not exactly sure which, but probably signals, timers and various file events).
You have four options here: don't stat, ignore the error, don't touch /dev/fd because it's weird, convince Apple that this is a bug.
Assuming that you're running your code as a non-root user, I'd guess that the problem is that the directory doesn't have the execute bit set, which prevents chdir-ing to that directory (which ReadDir might do before attempting to read the contents).