How can I update a file in tarball with Go?
I can only find some way to append file into a tar file like this:
file, err := os.OpenFile(path, os.O_WRONLY, 0644)
if _, err = file.Seek(-1024, os.SEEK_END); err != nil {
log.Fatalln(err)
}
if err != nil { return err }
defer file.Close()
tw := tar.NewWriter(file)
hdr := &tar.Header{
Name: "manifest.json",
Mode: 0644,
Size: int64(len(content)),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatal(err)
}
if _, err := tw.Write(content); err != nil {
log.Fatal(err)
}
if err := tw.Close(); err != nil {
log.Fatal(err)
}
After finding solutions by google ,it seemed I have some commands like tar uvf ... or try to extract the tarball and make a new one by golang or command tar.
As tar document says , command tar uf can update a file in tar ball directly, but after using tar uvf , I found a very interesting thing like below:
a.tar is a docker image made by docker image save
I want to change manifest.json file in thie docker image tarball by tar uvf a.tar manifest.json
before update, the content of tarball seems OK.
[root#master1 a]# ls
a.tar
[root#master1 a]# tar xf a.tar
tar: manifest.json: implausibly old time stamp 1970-01-01 08:00:00
tar: repositories: implausibly old time stamp 1970-01-01 08:00:00
[root#master1 a]# ls
2080c332033e330242f9ea2043e18c7ea98de2ee1b1b3921b1a33bcb84ac4cd5
2acf0fc34f8cc199bde6f74a7d85382461ff69cf4618e5964e7e1d19d0fe1d5c
3962220e6895f5a8c88034ae1bc7926b9bb40ecdc441196cd01c7cf36e36b7db.json
a7d478578679c443a1d4e695154dbbe8cad5525ac46fa80e925c2a080d0b620f
a.tar
c2103589e99f907333422ae78702360ad258a8f0366c20e341c9e0c53743e78a.json
da70bf0a87b384edc6cfc5cff3a33ba087de7b81f914df73181bfc2869314ebb
manifest.json
repositories
then I try to update manifest.json by tar uvf a.tar manifest.json
[root#master1 b]# ls
a.tar
[root#master1 b]# tar xf a.tar
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
[root#master1 b]# ls -al
total 65972
drwxr-xr-x 6 root root 4096 Apr 26 15:28 .
drwxr-xr-x 4 root root 4096 Apr 26 15:28 ..
drwxr-xr-x 2 root root 4096 Sep 10 2019 2080c332033e330242f9ea2043e18c7ea98de2ee1b1b3921b1a33bcb84ac4cd5
drwxr-xr-x 2 root root 4096 Dec 31 2019 2acf0fc34f8cc199bde6f74a7d85382461ff69cf4618e5964e7e1d19d0fe1d5c
-rw-r--r-- 1 root root 2429 Dec 31 2019 3962220e6895f5a8c88034ae1bc7926b9bb40ecdc441196cd01c7cf36e36b7db.json
drwxr-xr-x 2 root root 4096 Dec 31 2019 a7d478578679c443a1d4e695154dbbe8cad5525ac46fa80e925c2a080d0b620f
-rw-r--r-- 1 root root 67517440 Apr 26 15:28 a.tar
-rw-r--r-- 1 root root 2467 Sep 10 2019 c2103589e99f907333422ae78702360ad258a8f0366c20e341c9e0c53743e78a.json
drwxr-xr-x 2 root root 4096 Sep 10 2019 da70bf0a87b384edc6cfc5cff3a33ba087de7b81f914df73181bfc2869314ebb
As you can see, manifest.json and repositories disappeared . And sometimes , there will be a var directory. This is very strange .
Thanks a lot if someone can tell me why.
After trying use command 7za d and 7za a, I change a file in a tarball successfully .
Related
This question already has answers here:
Executing a Bash Script from Golang
(4 answers)
Closed 3 months ago.
I want to create a wrapper program that can wrapper whatever shell commands user provides, like:
./wrapper "cmd1 && cmd2"
In Python, I can call os.system("cmd1 && cmd2"). But Golang's exec.Command needs a list for command and args. Is there way in Golang to archive the same as Python's os.system()?
os/exec https://pkg.go.dev/os/exec
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("/usr/bin/bash", "-c", os.Args[1])
output, err := cmd.CombinedOutput()
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
$ go run main.go "ls -alh && pwd"
total 4.0K
drwxr-xr-x 2 xxxx xxxx 120 Nov 14 11:12 .
drwxrwxrwt 17 root root 420 Nov 14 11:42 ..
-rw-r--r-- 1 xxxx xxxx 217 Nov 14 11:42 main.go
/tmp/stk
I am connecting to pod via client-Go and I want to get the properties of the file directory
func GetPodFiles(c *gin.Context) {
client, _ := Init.ClusterID(c)
path := c.DefaultQuery("path", "/")
cmd := []string{
"sh",
"-c",
fmt.Sprintf("ls -l %s", path),
}
config, _ := Init.ClusterCfg(c)
req := client.CoreV1().RESTClient().Post().
Resource("pods").
Name("nacos-0").
Namespace("default").SubResource("exec").Param("container", "nacos")
req.VersionedParams(
&v1.PodExecOptions{
Command: cmd,
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
},
scheme.ParameterCodec,
)
var stdout, stderr bytes.Buffer
exec, err := remotecommand.NewSPDYExecutor(config, "POST", req.URL())
if err != nil {
response.FailWithMessage(response.InternalServerError, err.Error(), c)
return
}
err = exec.Stream(remotecommand.StreamOptions{
Stdin: nil,
Stdout: &stdout,
Stderr: &stderr,
})
if err != nil {
response.FailWithMessage(response.InternalServerError, "Error obtaining file", c)
return
}
fmt.Println(stdout.String())
}
Execution Result Output
total 0
lrwxrwxrwx 1 root root 7 Jun 1 2018 bin -> usr/bin
drwxr-xr-x 5 root root 360 Feb 16 16:39 dev
lrwxrwxrwx 1 root root 8 Jun 1 2018 sbin -> usr/sbin
drwxr-xr-x 2 root root 6 Apr 11 2018 srv
Expect the result
"data": [
{
"perm": "drwxr-xr-x",
"mod_time": "2022-03-02 15:02:15",
"kind": "d",
"name": "temp",
"size": ""
},
]
Is there a good way or a golang third-party library to handle it. Please let me know. Thank you
In a Kubernetes pod you can execute the stat linux command instead of ls command.
$ stat yourFileOrDirName
The output of this command by default is like this:
File: yourFileOrDirName
Size: 346 Blocks: 0 IO Block: 4096 directory
Device: 51h/82d Inode: 40431 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 1000/ username) Gid: ( 1000/ groupname)
Access: 2022-03-02 11:59:07.384821351 +0100
Modify: 2022-03-02 11:58:48.733821177 +0100
Change: 2022-03-02 11:58:48.733821177 +0100
Birth: 2021-12-21 11:12:05.571841723 +0100
But you can tweak its output like this:
$ stat --printf="%n,%A,%y,%s" yourFileOrDirName
where %n - file name, %A - permission bits and file type in human readable form, %y - time of last data modification human-readable, %s - total size, in bytes. You can also choose any character as a delimiter instead of comma.
the output will be:
yourFileOrDirName,drwxr-xr-x,2022-03-02 11:58:48.733821177 +0100,346
See more info about the stat command here.
After you get such output, I believe you can easily 'convert' it to json format if you really need it.
Furthermore, you can run the stat command like this:
$ stat --printf="{\"data\":[{\"name\":\"%n\",\"perm\":\"%A\",\"mod_time\":\"%y\",\"size\":\"%s\"}]}" yourFileOrDirName
Or as #mdaniel suggested, since the command does not contain any shell variables, nor a ', the cleaner command is:
stat --printf='{"data":[{"name":"%n","perm":"%A","mod_time":"%y","size":"%s"}]}' yourFileOrDirName
and get the DIY json output:
{"data":[{"name":"yourFileOrDirName","perm":"drwxrwxr-x","mod_time":"2022-02-04 15:17:27.000000000 +0000","size":"4096"}]}
So I've tried to execute a chain i.e. multiple commands on a Pod's container using client-go, and it seems to only work for some commands like ls.
Here is what I've tried:
req := client.CoreV1().RESTClient().Post().Resource("pods").Name(pod.Name).Namespace(pod.ObjectMeta.Namespace).SubResource("exec") // .Param("container", containerName)
scheme := runtime.NewScheme()
if err := _v1.AddToScheme(scheme); err != nil {
panic(err.Error())
}
parameterCodec := runtime.NewParameterCodec(scheme)
req.VersionedParams(&_v1.PodExecOptions{
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
Container: containerName,
Command: strings.Fields("/bin/sh -c " + command),
}, parameterCodec)
exec, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
if err != nil {
panic(err.Error())
}
var stdout, stderr bytes.Buffer
err = exec.Stream(remotecommand.StreamOptions{
Stdin: nil,
Stdout: &stdout,
Stderr: &stderr,
Tty: false,
})
if err != nil {
panic(err.Error())
}
log.Printf("Output from pod: %v", stdout.String())
log.Printf("Error from pod: %v", stderr.String())
When the command variable is just a simple ls -l, I get the desired output. But when I try to do something like 'ls -l && echo hello' it produces an error command terminated with exit code 2.
It doesn't output anything if I only put echo hello. However, it does produce the desired output hello if I remove the Bourne Shell prefix /bin/sh -c and the Command attribute equals to string.Fields("echo hello"), but this approach doesn't let me chain commands.
All in all, what I am trying to do is to execute a chain of commands on a Pod's container.
The corev1.PodExecOptions.Command accept value of []string type.
req.VersionedParams(&_v1.PodExecOptions{
Stdin: false,
Stdout: true,
Stderr: true,
TTY: false,
Container: containerName,
Command: cmds,
}, parameterCodec)
where, cmds can be:
cmds := []string{
"sh",
"-c",
"echo $HOME; ls -l && echo hello",
}
Output:
/root
total 68
drwxr-xr-x 2 root root 4096 Feb 24 00:00 bin
drwxr-xr-x 2 root root 4096 Feb 1 17:09 boot
drwxr-xr-x 2 root root 4096 Feb 24 00:00 mnt
drwxr-xr-x 2 root root 4096 Feb 24 00:00 opt
dr-xr-xr-x 396 root root 0 Mar 19 11:47 proc
drwx------ 2 root root 4096 Feb 24 00:00 root
.
.
hello
Explanation: Tim's answer
command: ["/bin/sh","-c"]
args: ["command one; command two && command three"]
The command ["/bin/sh", "-c"] says "run a shell, and execute the following instructions". The args are then passed as commands to the shell. In shell scripting a semicolon separates commands, and && conditionally runs the following command if the first succeed. In the above example, it always runs command one followed by command two, and only runs command three if command two succeeded.
N.B.:
For bash, it will be similar to something like below:
cmds := []string{
"bash",
"-c",
`export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}
mysql -h localhost -nsLNE -e "select 1;" 2>/dev/null | grep -v "*"`,
},
I have a file that was generated using the Go compress/gzip package with code like
payload := bytes.NewBuffer(nil)
gw := gzip.NewWriter(payload)
tw := tar.NewWriter(gw)
...
tw.Close()
gw.Close()
How can I unzip this file from the command line on Mac? I tried gunzip but it fails
$ gunzip test.gz
gunzip: test.gz: not in gzip format
Also tried following without luck
$ tar -xvf test.gz
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Your issue is that you are never writing to your tar writer, in your example code. In order to produce a valid targz file with content, you need to keep in mind that:
Add at least one file needs to be in the tarball
Each file requires headers
You'll need to use tar.WriteHeader to create the header for each file, and you can then simply write the content of the files as bytes through a call to tar.Write.
You can then untar it using tar -xvf test.tgz like you mentioned in your previous example.
Here is a sample code that I quickly wrote on my machine for the sake of demonstration:
package main
import (
"compress/gzip"
"archive/tar"
"os"
"fmt"
"time"
)
func main() {
// Create targz file which will contain other files.
file, err := os.Create("test.tgz")
if err != nil {
panic(err)
}
gw := gzip.NewWriter(file)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// Create file(s) in targz
if err := addFile(tw, "myfile.test", "example content"); err != nil {
panic(err)
}
}
func addFile(tw *tar.Writer, fileName, content string) error {
header := &tar.Header{
Name: fileName,
Size: int64(len(content)),
Mode: 0655,
ModTime: time.Now(),
}
err := tw.WriteHeader(header)
if err != nil {
return fmt.Errorf("could not write header for file %q: %w", fileName, err)
}
_, err = tw.Write([]byte(content))
if err != nil {
return fmt.Errorf("could not write content for file %q: %w", fileName, err)
}
return nil
}
And here is the result:
$> go run main.go
$> ls -la
total 5
drwxr-xr-x 12 ullaakut staff 384 Feb 12 05:34 ./
drwxr-xr-x 29 ullaakut staff 928 Jan 28 14:56 ../
-rw-r--r--# 1 ullaakut staff 6148 Dec 25 13:01 .DS_Store
-rw-r--r-- 1 ullaakut staff 888 Feb 12 05:34 main.go
-rw-r--r-- 1 ullaakut staff 121 Feb 12 05:34 test.tgz
$> tar -zxvf test.tgz
x myfile.test
$> cat myfile.test
example content
As you can see, the archive is extracted and uncompressed just fine :)
I don't see any options for the FSPathMoveObjectToTrashSync() function for not following links.
Here is what I have tried
Create a link and a file
[ 21:32:41 /tmp ] $ touch my_file
[ 21:32:45 /tmp ] $ ln -s my_file my_link
[ 21:32:52 /tmp ] $ la
total 8
drwxrwxrwt 12 root wheel 408 17 Maj 21:32 .
drwxr-xr-x# 6 root wheel 204 9 Sep 2009 ..
-rw-r--r-- 1 neoneye wheel 0 17 Maj 21:32 my_file
lrwxr-xr-x 1 neoneye wheel 7 17 Maj 21:32 my_link -> my_file
Move the link to the trash
OSStatus status = FSPathMoveObjectToTrashSync(
"/tmp/my_link",
NULL,
kFSFileOperationDefaultOptions
);
NSLog(#"status: %i", (int)status);
Output is
status: 0
However the file got removed and not the link
[ 21:32:55 /tmp ] $ la
total 8
drwxrwxrwt 11 root wheel 374 17 Maj 21:33 .
drwxr-xr-x# 6 root wheel 204 9 Sep 2009 ..
lrwxr-xr-x 1 neoneye wheel 7 17 Maj 21:32 my_link -> my_file
[ 21:33:05 /tmp ] $
How can I move move symlinks to the trash?
The Solution.. thanks to Rob Napier
NSString* path = #"/tmp/my_link";
OSStatus status = 0;
FSRef ref;
status = FSPathMakeRefWithOptions(
(const UInt8 *)[path fileSystemRepresentation],
kFSPathMakeRefDoNotFollowLeafSymlink,
&ref,
NULL
);
NSAssert((status == 0), #"failed to make FSRef");
status = FSMoveObjectToTrashSync(
&ref,
NULL,
kFSFileOperationDefaultOptions
);
NSLog(#"status: %i", (int)status);
Use FSPathMakeRefWithOptions() to generate an FSRef to the link. Then use FSMoveObjectToTrashSync() to delete it.
The other way would be to tell the NSWorkspace to “recycle” it, by sending it either a performFileOperation:source:destination:files:tag: message with the NSWorkspaceRecycleOperation operation or a recycleURLs:completionHandler: message.
I don't know how well either one of these would work on symlinks, but it's worth trying if you'd rather not deal with FSRefs.
my retro-futuristic approach
https://github.com/reklis/recycle
//
// main.swift
// recycle
//
// usage: recycle <files or directories to throw out>
//
import Foundation
import AppKit
var args = NSProcessInfo.processInfo().arguments
args.removeAtIndex(0) // first item in list is the program itself
var w = NSWorkspace.sharedWorkspace()
var fm = NSFileManager.defaultManager()
for arg in args {
let path = arg.stringByStandardizingPath;
let file = path.lastPathComponent
let source = path.stringByDeletingLastPathComponent
w.performFileOperation(NSWorkspaceRecycleOperation,
source:source,
destination: "",
files: [file],
tag: nil)
}