Is it possible to extract a tar.xz package in golang? My understanding is it's possible to use the library for tar and sending it to an xz go library.
I recently created an XZ decompression package so it is now
possible to extract a tar.xz using only Go code.
The following code extracts the file myfile.tar.xz to the current
directory:
package main
import (
"archive/tar"
"fmt"
"io"
"log"
"os"
"github.com/xi2/xz"
)
func main() {
// Open a file
f, err := os.Open("myfile.tar.xz")
if err != nil {
log.Fatal(err)
}
// Create an xz Reader
r, err := xz.NewReader(f, 0)
if err != nil {
log.Fatal(err)
}
// Create a tar Reader
tr := tar.NewReader(r)
// Iterate through the files in the archive.
for {
hdr, err := tr.Next()
if err == io.EOF {
// end of tar archive
break
}
if err != nil {
log.Fatal(err)
}
switch hdr.Typeflag {
case tar.TypeDir:
// create a directory
fmt.Println("creating: " + hdr.Name)
err = os.MkdirAll(hdr.Name, 0777)
if err != nil {
log.Fatal(err)
}
case tar.TypeReg, tar.TypeRegA:
// write a file
fmt.Println("extracting: " + hdr.Name)
w, err := os.Create(hdr.Name)
if err != nil {
log.Fatal(err)
}
_, err = io.Copy(w, tr)
if err != nil {
log.Fatal(err)
}
w.Close()
}
}
f.Close()
}
http://golang.org/pkg/archive/tar/#example_
also you can do
import "os/exec"
cmd := exec.Command("tar", "-x", "/your/archive.tar.xz")
err := cmd.Run()
There is no Lempel-Ziv-Markow encoder or decoder in the Go standard library. If you are allowed to assume that the platform your code runs on provides the xz utility, you could use stub functions like these:
import "os/exec"
// decompress xz compressed data stream r.
func UnxzReader(r io.Reader) (io.ReadCloser, error) {
unxz := exec.Command("xz", "-d")
unxz.Stdin = r
out, err := unxz.StdoutPipe()
if err != nil {
return nil, err
}
err = unxz.Start()
if err != nil {
return nil, err
}
// we are not interested in the exit status, but we should really collect
// that zombie process
go unxz.Wait()
return out, nil
}
Related
I am trying to create a .tar.gz file from folder that contains multiple files / folders. Once the .tar.gz file gets created, while extracting, the files are not not properly extracted. Mostly I think its because of large names or path exceeding some n characters, because same thing works when the filename/path is small. I referred this https://github.com/golang/go/issues/17630 and tried to add below code but it did not help.
header.Uid = 0
header.Gid = 0
I am using simple code seen below to create .tar.gz. The approach is, I create a temp folder, do some processing on the files and from that temp path, I create the .tar.gz file hence in the path below I am using pre-defined temp folder path.
package main
import (
"archive/tar"
"compress/gzip"
"fmt"
"io"
"log"
"os"
fp "path/filepath"
)
func main() {
// Create output file
out, err := os.Create("output.tar.gz")
if err != nil {
log.Fatalln("Error writing archive:", err)
}
defer out.Close()
// Create the archive and write the output to the "out" Writer
tmpDir := "C:/Users/USERNAME~1/AppData/Local/Temp/temp-241232063"
err = createArchive1(tmpDir, out)
if err != nil {
log.Fatalln("Error creating archive:", err)
}
fmt.Println("Archive created successfully")
}
func createArchive1(path string, targetFile *os.File) error {
gw := gzip.NewWriter(targetFile)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// walk through every file in the folder
err := fp.Walk(path, func(filePath string, info os.FileInfo, err error) error {
// ensure the src actually exists before trying to tar it
if _, err := os.Stat(filePath); err != nil {
return err
}
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
// generate tar header
header, err := tar.FileInfoHeader(info, info.Name())
header.Uid = 0
header.Gid = 0
if err != nil {
return err
}
header.Name = filePath //strings.TrimPrefix(filePath, fmt.Sprintf("%s/", fp.Dir(path))) //info.Name()
// write header
if err := tw.WriteHeader(header); err != nil {
return err
}
if _, err := io.Copy(tw, file); err != nil {
return err
}
return nil
})
return err
}
Please let me know what wrong I am doing.
I'm studying networks, and I'm doing a tcp server with Go. One of the challenges I'm studying is to send binaries or strings by a socket client to a server, save the server response to a txt, and compare it to the original data that was sent.
The problem is that the binaries do not arrive completely on the server.
Server
package main
import (
"fmt"
"io"
"log"
"net"
)
func main() {
l, err := net.Listen("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if nil != err {
log.Println(err)
continue
}
defer conn.Close()
go ConnHandler(conn)
}
}
func ConnHandler(conn net.Conn) {
recvBuf := make([]byte, 4096)
for {
n, err := conn.Read(recvBuf)
if nil != err {
if io.EOF == err {
log.Println(err)
return
}
log.Println(err)
return
}
if 0 < n {
data := recvBuf[:n]
fmt.Println(string(data))
}
}
}
Client
package main
import (
"fmt"
"log"
"net"
)
func main() {
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
var s string
fmt.Scanln(&s)
conn.Write([]byte(s))
conn.Close()
}
I'm generating the binaries using the command on linux:
head -c100000 /dev/urandom > binary_message.txt
I run the server:
./server > result.txt
And I send this data by the client using:
./client < binary_data.txt
In the end the file binary_data.txt have 98KB but the result .txt only has 0KB.
The problem is with scanning the binary from input. You didn't see it because the errors were ignored and not printed or otherwise handled. fmt.Scanln returns an error (so does the Write function). You should always check for possible errors happening.
I rewrote the client to load the file from disk itself as I don't think using stdin is a good fit for binary data.
package main
import (
"flag"
"io"
"log"
"net"
"os"
)
var fileName = flag.String("file", "", "file to send")
func main() {
flag.Parse()
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer conn.Close()
f, err := os.Open(*fileName)
if err != nil {
log.Println(err)
return
}
defer f.Close()
b := make([]byte, 1024)
for {
n, err := f.Read(b)
if err != nil {
if err == io.EOF {
log.Println("Done sending")
return
}
log.Println(err)
return
}
if _, err := conn.Write(b[:n]); err != nil {
log.Println(err)
return
}
}
}
You can use it with:
go run . -file=binary_message.txt
or if you have built the binary:
./client -file=binary_message.txt
I suggest you do the same for the server. Open a file for writing and write the binary data into that file. Use a flag to pass in the filename to write to. That will be cleaner than piping stdout to a file.
I just tried to download webp image from url, but I found something different when I try to process the stored image.
If I download the image from the browser, it can be decoded using x/image/webp package, but if I store the image using http.Get() then create a new file then io.Copy() the image, it says:
"missing RIFF chunk header"
I assume that I need to write some RIFF chunk header when I store it using golang code.
func main(){
response, e := http.Get(URL)
if e != nil {
log.Fatal(e)
}
defer response.Body.Close()
//open a file for writing
file, err := os.Create('tv.webp')
if err != nil {
log.Fatal(err)
}
defer file.Close()
// Use io.Copy to just dump the response body to the file. This supports huge files
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println("Success!")
imgData, err := os.Open("tv.webp")
if err != nil {
fmt.Println(err)
return
}
log.Printf("%+v", imgData)
image, err := webp.Decode(imgData)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(image.Bounds())
}
Here is the URL IMG URL
download file is not webp type. it's png.
package main
import (
"fmt"
"image"
"io"
"log"
"net/http"
"os"
_ "image/png"
)
func main() {
response, e := http.Get("https://www.sony.com/is/image/gwtprod/0abe7672ff4c6cb4a0a4d4cc143fd05b?fmt=png-alpha")
if e != nil {
log.Fatal(e)
}
defer response.Body.Close()
file, err := os.Create("dump")
if err != nil {
log.Fatal(err)
}
defer file.Close()
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println("Success!")
imageFile, err := os.Open("dump")
if err != nil {
panic(err)
}
m, name, err := image.Decode(imageFile)
if err != nil {
panic(err)
}
fmt.Println("image type is ", name, m.Bounds())
}
I've written the following code to tar a file, code works but strangely if I untar the archive the file permissions are gone so I can't read it unless I then chmod the file:
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
hdr := &tar.Header{Name: f.Name(),
Size: fi.Size(),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
r, err := ioutil.ReadFile("sample.txt")
if err != nil {
log.Fatalln(err)
}
if _, err := tw.Write(r); err != nil {
log.Fatalln(err)
}
if err := tw.Close(); err != nil {
log.Fatalln(err)
}
}
Any idea what I'm doing wrong?
You're not preserving the original permissions of the file. You're manually creating a header, and specifying only the name and size. Instead, use tar.FileInfoHeader to build the header.
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
// create header from FileInfo
hdr, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatalln(err)
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
// instead of reading the whole file into memory, prefer io.Copy
r, err := io.Copy(tw, f)
if err != nil {
log.Fatalln(err)
}
log.Printf("Wrote %d bytes\n", r)
}
Also note that I used io.Copy to copy data from the file (an io.Reader) to the tar writer (an io.Writer). This will work much better for larger files.
Also - pay special attention to this note from the docs:
Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
In this simple example, you're just using sample.txt so you shouldn't run into trouble. If you wanted to preserve a directory structure in your tar, you may have to modify the Name field in the header.
I'm attempting to create a compressed tar archive using the Go standard library, specifically compress/gzip and archive/tar. I can successfully create a tar archive, but when I try to compress said archive, the resulting tarball can't be decompressed. On OSX, I get "Error 1 - Operation Not Permitted"
To run this code, you'll need a file named foo.txt in the same directory.
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"io/ioutil"
"log"
"os"
)
func main() {
var b bytes.Buffer
// Create a new zip archive.
w := tar.NewWriter(gzip.NewWriter(&b))
fi, err := os.Stat("foo.txt")
if err != nil {
log.Fatal(err)
}
header, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatal(err)
}
err = w.WriteHeader(header)
if err != nil {
log.Fatal(err)
}
contents, err := ioutil.ReadFile("foo.txt")
if err != nil {
log.Fatal(err)
}
_, err = w.Write(contents)
if err != nil {
log.Fatal(err)
}
err = w.Close()
// Make sure to check the error on Close.
err = ioutil.WriteFile("foo.tar.gz", b.Bytes(), 0666)
if err != nil {
log.Fatal(err)
}
}
You need to close the underlying gzip writer so that it you are guaranteed all bytes are flushed out to the file. Like so:
// gzip writer
gz := gzip.NewWriter(f)
// Create a new tar archive.
w := tar.NewWriter(gz)
// add things to the tar archive
// ...
// make sure the gzip writer flushes any pending bytes
if err = gz.Close(); err != nil {
log.Fatal(err)
}