I am connecting to pod via client-Go and I want to get the properties of the file directory
func GetPodFiles(c *gin.Context) {
client, _ := Init.ClusterID(c)
path := c.DefaultQuery("path", "/")
cmd := []string{
"sh",
"-c",
fmt.Sprintf("ls -l %s", path),
}
config, _ := Init.ClusterCfg(c)
req := client.CoreV1().RESTClient().Post().
Resource("pods").
Name("nacos-0").
Namespace("default").SubResource("exec").Param("container", "nacos")
req.VersionedParams(
&v1.PodExecOptions{
Command: cmd,
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
},
scheme.ParameterCodec,
)
var stdout, stderr bytes.Buffer
exec, err := remotecommand.NewSPDYExecutor(config, "POST", req.URL())
if err != nil {
response.FailWithMessage(response.InternalServerError, err.Error(), c)
return
}
err = exec.Stream(remotecommand.StreamOptions{
Stdin: nil,
Stdout: &stdout,
Stderr: &stderr,
})
if err != nil {
response.FailWithMessage(response.InternalServerError, "Error obtaining file", c)
return
}
fmt.Println(stdout.String())
}
Execution Result Output
total 0
lrwxrwxrwx 1 root root 7 Jun 1 2018 bin -> usr/bin
drwxr-xr-x 5 root root 360 Feb 16 16:39 dev
lrwxrwxrwx 1 root root 8 Jun 1 2018 sbin -> usr/sbin
drwxr-xr-x 2 root root 6 Apr 11 2018 srv
Expect the result
"data": [
{
"perm": "drwxr-xr-x",
"mod_time": "2022-03-02 15:02:15",
"kind": "d",
"name": "temp",
"size": ""
},
]
Is there a good way or a golang third-party library to handle it. Please let me know. Thank you
In a Kubernetes pod you can execute the stat linux command instead of ls command.
$ stat yourFileOrDirName
The output of this command by default is like this:
File: yourFileOrDirName
Size: 346 Blocks: 0 IO Block: 4096 directory
Device: 51h/82d Inode: 40431 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 1000/ username) Gid: ( 1000/ groupname)
Access: 2022-03-02 11:59:07.384821351 +0100
Modify: 2022-03-02 11:58:48.733821177 +0100
Change: 2022-03-02 11:58:48.733821177 +0100
Birth: 2021-12-21 11:12:05.571841723 +0100
But you can tweak its output like this:
$ stat --printf="%n,%A,%y,%s" yourFileOrDirName
where %n - file name, %A - permission bits and file type in human readable form, %y - time of last data modification human-readable, %s - total size, in bytes. You can also choose any character as a delimiter instead of comma.
the output will be:
yourFileOrDirName,drwxr-xr-x,2022-03-02 11:58:48.733821177 +0100,346
See more info about the stat command here.
After you get such output, I believe you can easily 'convert' it to json format if you really need it.
Furthermore, you can run the stat command like this:
$ stat --printf="{\"data\":[{\"name\":\"%n\",\"perm\":\"%A\",\"mod_time\":\"%y\",\"size\":\"%s\"}]}" yourFileOrDirName
Or as #mdaniel suggested, since the command does not contain any shell variables, nor a ', the cleaner command is:
stat --printf='{"data":[{"name":"%n","perm":"%A","mod_time":"%y","size":"%s"}]}' yourFileOrDirName
and get the DIY json output:
{"data":[{"name":"yourFileOrDirName","perm":"drwxrwxr-x","mod_time":"2022-02-04 15:17:27.000000000 +0000","size":"4096"}]}
Related
This question already has answers here:
Executing a Bash Script from Golang
(4 answers)
Closed 3 months ago.
I want to create a wrapper program that can wrapper whatever shell commands user provides, like:
./wrapper "cmd1 && cmd2"
In Python, I can call os.system("cmd1 && cmd2"). But Golang's exec.Command needs a list for command and args. Is there way in Golang to archive the same as Python's os.system()?
os/exec https://pkg.go.dev/os/exec
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("/usr/bin/bash", "-c", os.Args[1])
output, err := cmd.CombinedOutput()
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
$ go run main.go "ls -alh && pwd"
total 4.0K
drwxr-xr-x 2 xxxx xxxx 120 Nov 14 11:12 .
drwxrwxrwt 17 root root 420 Nov 14 11:42 ..
-rw-r--r-- 1 xxxx xxxx 217 Nov 14 11:42 main.go
/tmp/stk
How can I update a file in tarball with Go?
I can only find some way to append file into a tar file like this:
file, err := os.OpenFile(path, os.O_WRONLY, 0644)
if _, err = file.Seek(-1024, os.SEEK_END); err != nil {
log.Fatalln(err)
}
if err != nil { return err }
defer file.Close()
tw := tar.NewWriter(file)
hdr := &tar.Header{
Name: "manifest.json",
Mode: 0644,
Size: int64(len(content)),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatal(err)
}
if _, err := tw.Write(content); err != nil {
log.Fatal(err)
}
if err := tw.Close(); err != nil {
log.Fatal(err)
}
After finding solutions by google ,it seemed I have some commands like tar uvf ... or try to extract the tarball and make a new one by golang or command tar.
As tar document says , command tar uf can update a file in tar ball directly, but after using tar uvf , I found a very interesting thing like below:
a.tar is a docker image made by docker image save
I want to change manifest.json file in thie docker image tarball by tar uvf a.tar manifest.json
before update, the content of tarball seems OK.
[root#master1 a]# ls
a.tar
[root#master1 a]# tar xf a.tar
tar: manifest.json: implausibly old time stamp 1970-01-01 08:00:00
tar: repositories: implausibly old time stamp 1970-01-01 08:00:00
[root#master1 a]# ls
2080c332033e330242f9ea2043e18c7ea98de2ee1b1b3921b1a33bcb84ac4cd5
2acf0fc34f8cc199bde6f74a7d85382461ff69cf4618e5964e7e1d19d0fe1d5c
3962220e6895f5a8c88034ae1bc7926b9bb40ecdc441196cd01c7cf36e36b7db.json
a7d478578679c443a1d4e695154dbbe8cad5525ac46fa80e925c2a080d0b620f
a.tar
c2103589e99f907333422ae78702360ad258a8f0366c20e341c9e0c53743e78a.json
da70bf0a87b384edc6cfc5cff3a33ba087de7b81f914df73181bfc2869314ebb
manifest.json
repositories
then I try to update manifest.json by tar uvf a.tar manifest.json
[root#master1 b]# ls
a.tar
[root#master1 b]# tar xf a.tar
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
[root#master1 b]# ls -al
total 65972
drwxr-xr-x 6 root root 4096 Apr 26 15:28 .
drwxr-xr-x 4 root root 4096 Apr 26 15:28 ..
drwxr-xr-x 2 root root 4096 Sep 10 2019 2080c332033e330242f9ea2043e18c7ea98de2ee1b1b3921b1a33bcb84ac4cd5
drwxr-xr-x 2 root root 4096 Dec 31 2019 2acf0fc34f8cc199bde6f74a7d85382461ff69cf4618e5964e7e1d19d0fe1d5c
-rw-r--r-- 1 root root 2429 Dec 31 2019 3962220e6895f5a8c88034ae1bc7926b9bb40ecdc441196cd01c7cf36e36b7db.json
drwxr-xr-x 2 root root 4096 Dec 31 2019 a7d478578679c443a1d4e695154dbbe8cad5525ac46fa80e925c2a080d0b620f
-rw-r--r-- 1 root root 67517440 Apr 26 15:28 a.tar
-rw-r--r-- 1 root root 2467 Sep 10 2019 c2103589e99f907333422ae78702360ad258a8f0366c20e341c9e0c53743e78a.json
drwxr-xr-x 2 root root 4096 Sep 10 2019 da70bf0a87b384edc6cfc5cff3a33ba087de7b81f914df73181bfc2869314ebb
As you can see, manifest.json and repositories disappeared . And sometimes , there will be a var directory. This is very strange .
Thanks a lot if someone can tell me why.
After trying use command 7za d and 7za a, I change a file in a tarball successfully .
So I've tried to execute a chain i.e. multiple commands on a Pod's container using client-go, and it seems to only work for some commands like ls.
Here is what I've tried:
req := client.CoreV1().RESTClient().Post().Resource("pods").Name(pod.Name).Namespace(pod.ObjectMeta.Namespace).SubResource("exec") // .Param("container", containerName)
scheme := runtime.NewScheme()
if err := _v1.AddToScheme(scheme); err != nil {
panic(err.Error())
}
parameterCodec := runtime.NewParameterCodec(scheme)
req.VersionedParams(&_v1.PodExecOptions{
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
Container: containerName,
Command: strings.Fields("/bin/sh -c " + command),
}, parameterCodec)
exec, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
if err != nil {
panic(err.Error())
}
var stdout, stderr bytes.Buffer
err = exec.Stream(remotecommand.StreamOptions{
Stdin: nil,
Stdout: &stdout,
Stderr: &stderr,
Tty: false,
})
if err != nil {
panic(err.Error())
}
log.Printf("Output from pod: %v", stdout.String())
log.Printf("Error from pod: %v", stderr.String())
When the command variable is just a simple ls -l, I get the desired output. But when I try to do something like 'ls -l && echo hello' it produces an error command terminated with exit code 2.
It doesn't output anything if I only put echo hello. However, it does produce the desired output hello if I remove the Bourne Shell prefix /bin/sh -c and the Command attribute equals to string.Fields("echo hello"), but this approach doesn't let me chain commands.
All in all, what I am trying to do is to execute a chain of commands on a Pod's container.
The corev1.PodExecOptions.Command accept value of []string type.
req.VersionedParams(&_v1.PodExecOptions{
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
Container: containerName,
Command: cmds,
}, parameterCodec)
where, cmds can be:
cmds := []string{
"sh",
"-c",
"echo $HOME; ls -l && echo hello",
}
Output:
/root
total 68
drwxr-xr-x 2 root root 4096 Feb 24 00:00 bin
drwxr-xr-x 2 root root 4096 Feb 1 17:09 boot
drwxr-xr-x 2 root root 4096 Feb 24 00:00 mnt
drwxr-xr-x 2 root root 4096 Feb 24 00:00 opt
dr-xr-xr-x 396 root root 0 Mar 19 11:47 proc
drwx------ 2 root root 4096 Feb 24 00:00 root
.
.
hello
Explanation: Tim's answer
command: ["/bin/sh","-c"]
args: ["command one; command two && command three"]
The command ["/bin/sh", "-c"] says "run a shell, and execute the following instructions". The args are then passed as commands to the shell. In shell scripting a semicolon separates commands, and && conditionally runs the following command if the first succeed. In the above example, it always runs command one followed by command two, and only runs command three if command two succeeded.
N.B.:
For bash, it will be similar to something like below:
cmds := []string{
"bash",
"-c",
`export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}
mysql -h localhost -nsLNE -e "select 1;" 2>/dev/null | grep -v "*"`,
},
I have a file that was generated using the Go compress/gzip package with code like
payload := bytes.NewBuffer(nil)
gw := gzip.NewWriter(payload)
tw := tar.NewWriter(gw)
...
tw.Close()
gw.Close()
How can I unzip this file from the command line on Mac? I tried gunzip but it fails
$ gunzip test.gz
gunzip: test.gz: not in gzip format
Also tried following without luck
$ tar -xvf test.gz
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Your issue is that you are never writing to your tar writer, in your example code. In order to produce a valid targz file with content, you need to keep in mind that:
Add at least one file needs to be in the tarball
Each file requires headers
You'll need to use tar.WriteHeader to create the header for each file, and you can then simply write the content of the files as bytes through a call to tar.Write.
You can then untar it using tar -xvf test.tgz like you mentioned in your previous example.
Here is a sample code that I quickly wrote on my machine for the sake of demonstration:
package main
import (
"compress/gzip"
"archive/tar"
"os"
"fmt"
"time"
)
func main() {
// Create targz file which will contain other files.
file, err := os.Create("test.tgz")
if err != nil {
panic(err)
}
gw := gzip.NewWriter(file)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// Create file(s) in targz
if err := addFile(tw, "myfile.test", "example content"); err != nil {
panic(err)
}
}
func addFile(tw *tar.Writer, fileName, content string) error {
header := &tar.Header{
Name: fileName,
Size: int64(len(content)),
Mode: 0655,
ModTime: time.Now(),
}
err := tw.WriteHeader(header)
if err != nil {
return fmt.Errorf("could not write header for file %q: %w", fileName, err)
}
_, err = tw.Write([]byte(content))
if err != nil {
return fmt.Errorf("could not write content for file %q: %w", fileName, err)
}
return nil
}
And here is the result:
$> go run main.go
$> ls -la
total 5
drwxr-xr-x 12 ullaakut staff 384 Feb 12 05:34 ./
drwxr-xr-x 29 ullaakut staff 928 Jan 28 14:56 ../
-rw-r--r--# 1 ullaakut staff 6148 Dec 25 13:01 .DS_Store
-rw-r--r-- 1 ullaakut staff 888 Feb 12 05:34 main.go
-rw-r--r-- 1 ullaakut staff 121 Feb 12 05:34 test.tgz
$> tar -zxvf test.tgz
x myfile.test
$> cat myfile.test
example content
As you can see, the archive is extracted and uncompressed just fine :)
I tried to run the following Go code:
package main
import (
"fmt"
"io/ioutil"
)
func main() {
items, err := ioutil.ReadDir("/dev/fd")
if err != nil {
panic(err)
}
fmt.Println(items)
}
I just get this error:
panic: lstat /dev/fd/4: bad file descriptor
goroutine 1 [running]:
main.main()
/Users/andy/Desktop/demo.go:11 +0xe8
exit status 2
The /dev/fd folder definitely exists, and there is a /dev/fd/4 inside there when I ls it.
$ ls -Al /dev/fd
total 9
crw--w---- 1 andy tty 16, 4 Jan 25 00:16 0
crw--w---- 1 andy tty 16, 4 Jan 25 00:16 1
crw--w---- 1 andy tty 16, 4 Jan 25 00:16 2
dr--r--r-- 3 root wheel 4419 Jan 23 20:42 3/
dr--r--r-- 1 root wheel 0 Jan 23 20:42 4/
What's going on? Why can't I read this directory? I'm trying to port the ls command to Go here, so I would like to be able to read this directory in order to produce similar output to ls.
EDIT: I am running everything as non-root user. The executable bit on /dev/fd is set.
$ ls -al /dev | grep fd
dr-xr-xr-x 1 root wheel 0 Jan 23 20:42 fd/
$ stat /dev/fd/4 # same result with -L flag
stat: /dev/fd/4: stat: Bad file descriptor
First, let's remember that /dev/fd is special. Its contents are different for every process (since they reflect the file descriptors of that process), so it doesn't really mean anything that ls can list it because its contents will be different for ls and your program.
Anyway, here's a slightly updated version of your program where instead of letting ioutil do things behind our back, we do it ourselves to see what's going on:
package main
import (
"fmt"
"time"
"os"
"log"
)
func main() {
d, err := os.Open("/dev/fd")
if err != nil {
log.Fatal(err)
}
names, err := d.Readdirnames(0)
if err != nil {
log.Fatal(err)
}
for _, n := range names {
n = "/dev/fd/" + n
fmt.Printf("file: %s\n", n)
_, err := os.Lstat(n)
if err != nil {
fmt.Printf("lstat error: %s - %v\n", n, err)
}
}
time.Sleep(time.Second * 200)
}
And then when run it gives me:
file: /dev/fd/0
file: /dev/fd/1
file: /dev/fd/2
file: /dev/fd/3
file: /dev/fd/4
lstat error: /dev/fd/4 - lstat /dev/fd/4: bad file descriptor
So this is indeed the same problem. I added the sleep at the end so that the process doesn't die so that we can debug which file descriptors it has. In another terminal:
$ lsof -p 7861
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
foo 7861 art cwd DIR 1,4 2272 731702 /Users/art/src/go/src
foo 7861 art txt REG 1,4 1450576 8591078117 /private/var/folders/m7/d614cd9x61s0l3thb7cf3rkh0000gn/T/go-build268777304/command-line-arguments/_obj/exe/foo
foo 7861 art txt REG 1,4 837248 8590944844 /usr/lib/dyld
foo 7861 art 0u CHR 16,4 0t8129 645 /dev/ttys004
foo 7861 art 1u CHR 16,4 0t8129 645 /dev/ttys004
foo 7861 art 2u CHR 16,4 0t8129 645 /dev/ttys004
foo 7861 art 3r DIR 37,7153808 0 316 /dev/fd
foo 7861 art 4u KQUEUE count=0, state=0x8
We can see that fd 4 is a KQUEUE. Kqueue file descriptors are used for managing events on OSX, similar to epoll on Linux if you know that mechanism.
It appears that OSX does not allow stat on kqueue file descriptors in /dev/fd which we can verify with:
$ cat > foo.c
#include <sys/types.h>
#include <sys/event.h>
#include <sys/stat.h>
#include <stdio.h>
#include <err.h>
int
main(int argc, char **argv)
{
int fd = kqueue();
char path[16];
struct stat st;
snprintf(path, sizeof(path), "/dev/fd/%d", fd);
if (lstat(path, &st) == -1)
err(1, "lstat");
return 0;
}
$ cc -o foo foo.c && ./foo
foo: lstat: Bad file descriptor
$
Go programs on OSX need a kqueue to handle various events (not exactly sure which, but probably signals, timers and various file events).
You have four options here: don't stat, ignore the error, don't touch /dev/fd because it's weird, convince Apple that this is a bug.
Assuming that you're running your code as a non-root user, I'd guess that the problem is that the directory doesn't have the execute bit set, which prevents chdir-ing to that directory (which ReadDir might do before attempting to read the contents).