The below code tries to download and decompress a tar.gz file which in turn packs a binary inside it. The decompression seems to be going wrong since after the decompression is done and the file is there, the file is not being recognized as a binary executable by the system (Mac OS X). I would really appreciate it if anyone can help me identify what is going wrong here.
Here's the code
resp, err := svc.GetObject(&s3.GetObjectInput{
Bucket: aws.String(S3Bucket),
Key: aws.String(s3Key),
})
if err != nil { return err }
defer resp.Body.Close()
ExtractTarGz(resp.Body)
func ExtractTarGz(gzipStream io.Reader) {
uncompressedStream, err := gzip.NewReader(gzipStream)
if err != nil {
log.Fatal("ExtractTarGz: NewReader failed")
}
tarReader := tar.NewReader(uncompressedStream)
for true {
header, err := tarReader.Next()
if err == io.EOF {
break
}
if err != nil {
log.Fatalf("ExtractTarGz: Next() failed: %s", err.Error())
}
switch header.Typeflag {
case tar.TypeDir:
if err := os.Mkdir(header.Name, 0755); err != nil {
log.Fatalf("ExtractTarGz: Mkdir() failed: %s", err.Error())
}
case tar.TypeReg:
outFile, err := os.Create(header.Name)
if err != nil {
log.Fatalf("ExtractTarGz: Create() failed: %s", err.Error())
}
if _, err := io.Copy(outFile, tarReader); err != nil {
log.Fatalf("ExtractTarGz: Copy() failed: %s", err.Error())
}
outFile.Close()
default:
log.Fatalf(
"ExtractTarGz: uknown type: %s in %s",
header.Typeflag,
header.Name)
}
}
}
(Normal unzipping the file from the tar file using Keka for Mac works fine and the resulting unzipped file is recognized as a binary. I have checked the file size, it's the same as that of the expected binary.
)
Related
I wrote a function that makes a .zip archive from a directory (and subdir-s, files in it)
func ZipDirectory(backupName string) error {
file, err := os.Create(backupName)
if err != nil {
return err
}
defer file.Close()
w := zip.NewWriter(file)
defer w.Close()
walker := func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return fmt.Errorf("error opening file %q: %w", path, err)
}
defer file.Close()
f, err := w.Create(path)
if err != nil {
return fmt.Errorf("error creating file %q: %w", path, err)
}
_, err = io.Copy(f, file)
if err != nil {
return fmt.Errorf("error copying file %q: %w", path, err)
}
return nil
}
err = filepath.Walk(config.FilePath, walker)
if err != nil {
return err
}
return nil
}
but if directory weights more then 3GB it creates empty archive which weights exactly as it should be.
it looks like this:
help, don't understand this mystic result....
Big zip archives shows empty after archivation because you don't have installed software for unzipiing it (like 7Zip)
After installation and openning with 7Zip everything works
I wonder if possible to copy running .exe file to another folder. I am trying to do this using usual copy approach in Go like that.
func copy(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return err
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return err
}
defer out.Close()
_, err = io.Copy(out, in)
if err != nil {
return err
}
return out.Close()
}
...
copyErr := copy(os.Args[0], "D:"+"\\"+"whs.exe")
if copyErr != nil {
log.Panicf("copy -> %v", copyErr)
}
The file copied with the same size but I can't open it correctly. I have only a fast cmd flash. After several milliseconds, cmd is closing and I can't see even any errors.
I was trying to write errors to log file but it's empty.
f, err := os.OpenFile("debug.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0777)
if err != nil {
log.Panicf("setLogOutput -> %v", err)
}
defer f.Close()
log.SetOutput(f)
If I open not copied .exe file everything works correctly.
I've reduced my program to only one main method. The result was the same.
func main() {
log.Println("Starting...")
copyErr := copy(os.Args[0], "F:"+"\\"+"whs.exe")
if copyErr != nil {
log.Panicf("copy -> %v", copyErr)
}
os.Stdin.Read([]byte{0})
}
I have found an error.
The process cannot access the file because it is being used by another process.
I was trying to copy the .exe file to its own path.
func copy(src, dst string) error {
if _, err := os.Stat(dst); os.IsNotExist(err) {
in, err := os.Open(src)
if err != nil {
return err
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return err
}
defer out.Close()
_, err = io.Copy(out, in)
if err != nil {
return err
}
}
return nil
}
I'm trying to move a file from my C-drive to my H-drive using os.Replace().
The code looks as follows:
func MoveFile(source string, destination string) {
err := os.Rename(source, destination)
if err != nil {
fmt.Println(err)
}
}
However, when I run the code I get the following error:
rename C:\old\path\to\file.txt H:\new\path\to\file.txt: The system cannot move the file to a different disk drive.
I found this issue on GitHub that specifies the problem but it appears that they will not change this function to allow it to move file on different disk drives.
I already searched for other possibilities to move files, but found nothing in the standard documentation or the internet.
So, what should I do now to be able to move files on different disk drives?
As the comment said, you'll need to create a new file on the other disk, copy the contents, and then remove the original. It's straightforward using os.Create, io.Copy, and os.Remove:
import (
"fmt"
"io"
"os"
)
func MoveFile(sourcePath, destPath string) error {
inputFile, err := os.Open(sourcePath)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
outputFile, err := os.Create(destPath)
if err != nil {
inputFile.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer outputFile.Close()
_, err = io.Copy(outputFile, inputFile)
inputFile.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
// The copy was successful, so now delete the original file
err = os.Remove(sourcePath)
if err != nil {
return fmt.Errorf("Failed removing original file: %s", err)
}
return nil
}
You need to make sure that you handle all cases on both Linux and Windows. For example, for any size file,
package main
import (
"flag"
"fmt"
"io"
"os"
)
func MoveFile(source, destination string) (err error) {
src, err := os.Open(source)
if err != nil {
return err
}
defer src.Close()
fi, err := src.Stat()
if err != nil {
return err
}
flag := os.O_WRONLY | os.O_CREATE | os.O_TRUNC
perm := fi.Mode() & os.ModePerm
dst, err := os.OpenFile(destination, flag, perm)
if err != nil {
return err
}
defer dst.Close()
_, err = io.Copy(dst, src)
if err != nil {
dst.Close()
os.Remove(destination)
return err
}
err = dst.Close()
if err != nil {
return err
}
err = src.Close()
if err != nil {
return err
}
err = os.Remove(source)
if err != nil {
return err
}
return nil
}
func main() {
var src, dst string
flag.StringVar(&src, "src", "", "source file")
flag.StringVar(&dst, "dst", "", "destination file")
flag.Parse()
if src == "" || dst == "" {
flag.Usage()
os.Exit(1)
}
err := MoveFile(src, dst)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("moved %q to %q\n", src, dst)
}
Output (Linux):
$ cp move.file src.file && go build movefile.go && ./movefile -src=src.file -dst=dst.file
moved "src.file" to "dst.file"
$
Output (Windows):
>copy /Y move.file src.file && go build movefile.go && movefile -src=src.file -dst=dst.file
moved "src.file" to "dst.file"
>
This solution Moves the file and preserves permissions:
func MoveFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
out, err := os.Create(dst)
if err != nil {
in.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer out.Close()
_, err = io.Copy(out, in)
in.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
err = out.Sync()
if err != nil {
return fmt.Errorf("Sync error: %s", err)
}
si, err := os.Stat(src)
if err != nil {
return fmt.Errorf("Stat error: %s", err)
}
err = os.Chmod(dst, si.Mode())
if err != nil {
return fmt.Errorf("Chmod error: %s", err)
}
err = os.Remove(src)
if err != nil {
return fmt.Errorf("Failed removing original file: %s", err)
}
return nil
}
If only want to Copy the file without remove the original:
func CopyFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
out, err := os.Create(dst)
if err != nil {
in.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer out.Close()
_, err = io.Copy(out, in)
in.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
err = out.Sync()
if err != nil {
return fmt.Errorf("Sync error: %s", err)
}
si, err := os.Stat(src)
if err != nil {
return fmt.Errorf("Stat error: %s", err)
}
err = os.Chmod(dst, si.Mode())
if err != nil {
return fmt.Errorf("Chmod error: %s", err)
}
return nil
}
Maybe you can use a magic approach, just using the syscall.MoveFile as follows.
func main() {
oldpath := "D:\\black.txt"
newpath := "E:\\black-new.txt"
from, _ := syscall.UTF16PtrFromString(oldpath)
to, _ := syscall.UTF16PtrFromString(newpath)
fmt.Println(*from, *to)
err := syscall.MoveFile(from, to)
if err != nil {
panic(err)
}
}
the program works.
if you want a cross-platform compatibility program, you can implement your own MoveFile.
func MoveFile(src string, dst string) error {
if runtime.GOOS == "windows" {
from, _ := syscall.UTF16PtrFromString(src)
to, _ := syscall.UTF16PtrFromString(dst)
return syscall.MoveFile(from, to)
} else {
return os.Rename(src, dst)
}
}
I am writing a simple script to get download unzip the tar.gz file and then remove it. Whenever I try to remove it I get an error:
The process cannot access the file because it is being used by another process.
I assume the error is in how I pass the file to the extractTarGz function, but I am not sure.
Here is the code:
package main
import (
"archive/tar"
"compress/gzip"
"io"
"log"
"os"
)
func main() {
f, err := os.Open("file.tar.gz")
if err != nil {
panic(err)
}
defer f.Close()
extractTarGz(f)
err = os.Remove("file.tar.gz")
}
func extractTarGz(gzipStream io.Reader) {
uncompressedStream, err := gzip.NewReader(gzipStream)
if err != nil {
log.Fatal("ExtractTarGz: NewReader failed")
}
tarReader := tar.NewReader(uncompressedStream)
for true {
header, err := tarReader.Next()
if err == io.EOF {
break
}
if err != nil {
log.Fatalf("ExtractTarGz: Next() failed: %s", err.Error())
}
switch header.Typeflag {
case tar.TypeDir:
if err := os.Mkdir(header.Name, 0755); err != nil {
log.Fatalf("ExtractTarGz: Mkdir() failed: %s", err.Error())
}
case tar.TypeReg:
outFile, err := os.Create(header.Name)
if err != nil {
log.Fatalf("ExtractTarGz: Create() failed: %s", err.Error())
}
defer outFile.Close()
if _, err := io.Copy(outFile, tarReader); err != nil {
log.Fatalf("ExtractTarGz: Copy() failed: %s", err.Error())
}
default:
log.Fatalf(
"ExtractTarGz: uknown type: %s in %s",
header.Typeflag,
header.Name)
}
}
}
You should first close the file, and then attempt to remove it. Since you close it using defer, that will / would be called after the os.Remove() call.
Try it like this:
name := "file.tar.gz"
defer func() {
if err = os.Remove(name); err != nil {
log.Printf("Failed to remove %s: %v", name, err)
}
}()
f, err := os.Open(name)
if err != nil {
panic(err)
}
defer f.Close()
extractTarGz(f)
Deferred functions are executed in LIFO (last-in-first-out) order, so first f.Close() will be called, and then the other which tries to remove the file. Quoting from Spec: Deferred statements:
...deferred functions are invoked immediately before the surrounding function returns, in the reverse order they were deferred.
f, err := os.Open("file.tar.gz")
if err != nil {
panic(err)
}
defer f.Close()
extractTarGz(f)
err = os.Remove("file.tar.gz")
At the very least, you need to close the file before you removeit.
err = f.Close()
if err != nil {
panic(err)
}
err = os.Remove("file.tar.gz")
defer f.Close() won't run until the end of the function.
I have the code
if config.Process.TraceLog != "" {
f, err := os.OpenFile(config.Process.TraceLog, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("error opening log file: %v", err)
os.Exit(1)
}
Trace.SetOutput(f)
}
if config.Process.InfoLog != "" {
f, err := os.OpenFile(config.Process.InfoLog, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("error opening log file: %v", err)
os.Exit(1)
}
Info.SetOutput(f)
}
if config.Process.WarningLog != "" {
f, err := os.OpenFile(config.Process.WarningLog, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("error opening log file: %v", err)
os.Exit(1)
}
Warning.SetOutput(f)
}
if config.Process.ErrorLog != "" {
f, err := os.OpenFile(config.Process.ErrorLog, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("error opening log file: %v", err)
os.Exit(1)
}
Error.SetOutput(f)
}
What is good way to shorten it? I don't want to repeat same very similar block 4 times.
How to make it in a loop? I tried to create a map with file name and logger object, but don't know how to post a reference correctly
This is how i finally did it
m := map[string]*log.Logger{
config.Process.TraceLog: Trace,
config.Process.InfoLog: Info,
config.Process.WarningLog: Warning,
config.Process.ErrorLog: Error,
}
for filepath, l := range m {
if filepath != "" {
f, err := os.OpenFile(filepath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("error opening log file: %v", err)
os.Exit(1)
}
l.SetOutput(f)
}
}
Something like this could work (you might need to adjust the types of the function in the interface or drop the interface completely):
setLogIfNotEmpty(config.Process.TraceLog, Trace)
setLogIfNotEmpty(config.Process.InfoLog, Info)
setLogIfNotEmpty(config.Process.WarningLog, Warning)
setLogIfNotEmpty(config.Process.ErrorLog, Error)
interface SetOutputer {
SetOutput(*os.File)
}
func setLogIfNotEmpty(file string, log SetOutputer) {
if file != "" {
f, err := os.OpenFile(file, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("error opening log file: %v", err)
os.Exit(1)
}
log.SetOutput(f)
}
}