I need to take tmp1.zip and append it's tmp1.signed file to the end of it; creating a new tmp1.zip.signed file using Go.
It's essentially same as cat | sc
I could call cmd line from Go, but that seems super inefficient (and cheesy).
So far
Google-ing the words "go combine files" et. al. yields minimal help.
But I have come across a couple of options that I have tried such as ..
f, err := os.OpenFile("tmp1.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
if _, err := f.Write([]byte("appended some data\n")); err != nil {
log.Fatal(err)
}
if err := f.Close(); err != nil {
log.Fatal(err)
}
But that is just getting strings added to the end of the file, not really merging the two files, or appending the signature to the original file.
Question
Assuming I am asking the right questions to get one file appended to another, Is there a better sample of how exactly to merge two files into one using Go?
Based on your question, you want to create a new file with the content of both files.
You can use io.Copy to achieve that.
Here is a simple command-line tool implementing it.
package main
import (
"io"
"log"
"os"
)
func main() {
if len(os.Args) != 4 {
log.Fatalln("Usage: %s <zip> <signed> <output>\n", os.Args[0])
}
zipName, signedName, output := os.Args[1], os.Args[2], os.Args[3]
zipIn, err := os.Open(zipName)
if err != nil {
log.Fatalln("failed to open zip for reading:", err)
}
defer zipIn.Close()
signedIn, err := os.Open(signedName)
if err != nil {
log.Fatalln("failed to open signed for reading:", err)
}
defer signedIn.Close()
out, err := os.OpenFile(output, os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatalln("failed to open outpout file:", err)
}
defer out.Close()
n, err := io.Copy(out, zipIn)
if err != nil {
log.Fatalln("failed to append zip file to output:", err)
}
log.Printf("wrote %d bytes of %s to %s\n", n, zipName, output)
n, err = io.Copy(out, signedIn)
if err != nil {
log.Fatalln("failed to append signed file to output:", err)
}
log.Printf("wrote %d bytes of %s to %s\n", n, signedName, output)
}
Basically, it open both files you want to merge, create a new one and copy the content of each file to the new file.
Related
I'm getting the following golintci message:
testdrive/utils.go:92:16: G110: Potential DoS vulnerability via decompression bomb (gosec)
if _, err := io.Copy(targetFile, fileReader); err != nil {
^
Read the corresponding CWE and I'm not clear on how this is expected to be corrected.
Please offer pointers.
func unzip(archive, target string) error {
reader, err := zip.OpenReader(archive)
if err != nil {
return err
}
for _, file := range reader.File {
path := filepath.Join(target, file.Name) // nolint: gosec
if file.FileInfo().IsDir() {
if err := os.MkdirAll(path, file.Mode()); err != nil {
return err
}
continue
}
fileReader, err := file.Open()
if err != nil {
return err
}
defer fileReader.Close() // nolint: errcheck
targetFile, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.Mode())
if err != nil {
return err
}
defer targetFile.Close() // nolint: errcheck
if _, err := io.Copy(targetFile, fileReader); err != nil {
return err
}
}
return nil
}
The warning you get comes from a rule provided in gosec.
The rule specifically detects usage of io.Copy on file decompression.
This is a potential issue because io.Copy:
copies from src to dst until either EOF is reached on src or an error occurs.
So, a malicious payload might cause your program to decompress an unexpectedly big amount of data and go out of memory, causing denial of service as mentioned in the warning message.
In particular, gosec will check (source) the AST of your program and warn you about usage of io.Copy or io.CopyBuffer together with any one of the following:
"compress/gzip".NewReader
"compress/zlib".NewReader or NewReaderDict
"compress/bzip2".NewReader
"compress/flate".NewReader or NewReaderDict
"compress/lzw".NewReader
"archive/tar".NewReader
"archive/zip".NewReader
"*archive/zip".File.Open
Using io.CopyN removes the warning because (quote) it "copies n bytes (or until an error) from src to dst", thus giving you (the program writer) control of how many bytes to copy. So you could pass an arbitrarily large n that you set based on the available resources of your application, or copy in chunks.
Based on various pointers provided, replaced
if _, err := io.Copy(targetFile, fileReader); err != nil {
return err
}
with
for {
_, err := io.CopyN(targetFile, fileReader, 1024)
if err != nil {
if err == io.EOF {
break
}
return err
}
}
PS while this helps memory footprint, this wouldn't help a DDOS attack copying very long and/or infinite stream ...
Assuming that you're working on compressed data, you need to use io.CopyN.
You can try a workaround with --nocompress flag. But this will cause the data to be included uncompressed.
See the following PR and related issue : https://github.com/go-bindata/go-bindata/pull/50
I have a empty file called a.txt, I want to output a value(int) to it in a loop, and overwrite last content in file a.txt. For example,
// open a file
f, err := os.Open("test.txt")
if err != nil {
log.Fatal(err)
}
defer f.Close()
// another file
af, err := os.OpenFile("a.txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
if err != nil {
log.Fatal(err)
}
defer af.Close()
b := []byte{}
scanner := bufio.NewScanner(f)
for scanner.Scan() {
b = append(b, scanner.Bytes()...)
// how to output len(b) into a.txt?
}
You can also try:
os.OpenFile with custom flags to truncate file, as shown below
package main
import (
"log"
"os"
)
func main() {
f, err := os.OpenFile("notes.txt", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0755)
if err != nil {
log.Fatal(err)
}
if err := f.Close(); err != nil {
log.Fatal(err)
}
}
Just use the truncate method and write again to file starting at the begining.
err = f.Truncate(0)
_, err = f.Seek(0, 0)
_, err = fmt.Fprintf(f, "%d", len(b))
Use os.Create() instead:
f, err := os.Create("test.txt")
From the func's doc:
Create creates or truncates the named file. If the file already exists, it is truncated. If the file does not exist, it is created ...
I am trying to read gzip files using compress/gzip. I am using http.DetectContentType as I do not know if I get a normal txt file or a gzipped one. My code is very straight forward and as below:
f, err := os.Open(fullpath)
if err != nil {
log.Panicf("Can not open file %s: %v", fullpath, err)
return ""
}
defer f.Close()
buff := make([]byte, 512)
_, err = f.Read(buff)
if err != nil && err != io.EOF{
log.Panicf("Cannot read buffer %v", err);
return ""
}
switch filetype := http.DetectContentType(buff); filetype {
case "application/x-gzip":
log.Println("File Type is", filetype)
reader, err := gzip.NewReader(f)
if err != nil && err != io.EOF{
log.Panicf("Cannot read gzip archive %v", err);
return ""
}
defer reader.Close()
target := "/xx/yy/abcd.txt"
writer, err := os.Create(target)
if err != nil {
log.Panicf("Cannot write unarchived file %v", err);
return ""
}
defer writer.Close()
_, err = io.Copy(writer, reader)
return target
The problem is that the gzip reader always errors out saying "Cannot read gzip archive gzip: invalid header" I have tried the zlib library too but in vain. I gzipped the source file in mac using the command line gzip tool.
Please show me where I am going wrong.
You're reading the first 512 bytes of the file, so the gzip.Reader won't ever see that. Since these are regular files, you can seek back to the start after a successful Read:
f.Seek(0, os.SEEK_SET)
I'm trying to run a fairly simple bash command from my Go code. My program writes out an IPTables config file and I need to issue a command to make IPTables refresh from this config. This is very straightforward at the commandline:
/sbin/iptables-restore < /etc/iptables.conf
However, I can't for the life of me figure out how to issue this command with exec.Command(). I tried a few things to accomplish this:
cmd := exec.Command("/sbin/iptables-restore", "<", "/etc/iptables.conf")
// And also
cmd := exec.Command("/sbin/iptables-restore", "< /etc/iptables.conf")
No surprise, neither of those worked. I also tried to feed the filename into the command by piping in the file name to stdin:
cmd := exec.Command("/sbin/iptables-restore")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
io.WriteString(stdin, "/etc/iptables.conf")
That doesn't work either, no surprise. I can use stdin to pipe in the contents of the file, but this seems silly when I can just tell iptables-restore what data to go read. So how might I get Go to run the command /sbin/iptables-restore < /etc/iptables.conf?
first read this /etc/iptables.conf file content then write it to cmd.StdinPipe() like this:
package main
import (
"io"
"io/ioutil"
"log"
"os/exec"
)
func main() {
bytes, err := ioutil.ReadFile("/etc/iptables.conf")
if err != nil {
log.Fatal(err)
}
cmd := exec.Command("/sbin/iptables-restore")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
_, err = io.WriteString(stdin, string(bytes))
if err != nil {
log.Fatal(err)
}
}
cmd := exec.Command("/usr/sbin/iptables-restore", "--binary", iptablesFilePath)
_, err := cmd.CombinedOutput()
if err != nil {
return err
}
return nil
this work fine on my Raspberry Pi3
The os/exec package does not invoke the system shell, nor does it implement the < redirection syntax typically handled by a shell.
Open the input file and use that file as stdin:
stdin, err := os.Open("/etc/iptables.conf")
if err != nil {
log.Fatal(err)
}
defer stdin.Close()
cmd := exec.Command("/sbin/iptables-restore")
cmd.Stdin = stdin // <-- use open file as stdin
result, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", result)
i am trying to write to to a file. i read the whole content of the file and now i want to change the content of the file based on some word that i have got from the file. but when i check, the content of the file, it is still the same and it has not change. this is what i used
if strings.Contains(string(read), sam) {
fmt.Println("this file contain that word")
temp := strings.ToUpper(sam)
fmt.Println(temp)
err := ioutil.WriteFile(fi.Name(), []byte(temp), 0644)
} else {
fmt.Println(" the word is not in the file")
}
Considering that your call to ioutil.WriteFile() is consistent with what is used in "Go by Example: Writing Files", this should work.
But that Go by example article check the err just after the write call.
You check the err outside the scope of your test:
if matched {
read, err := ioutil.ReadFile(path)
//fmt.Println(string(read))
fmt.Println(" This is the name of the file", fi.Name())
if strings.Contains(string(read), sam) {
fmt.Println("this file contain that word")
Value := strings.ToUpper(sam)
fmt.Println(Value)
err = ioutil.WriteFile(fi.Name(), []byte(Value), 0644)
} else {
fmt.Println(" the word is not in the file")
}
check(err) <===== too late
}
The err you are testing is the one you got when reading the file (ioutil.ReadFile), because of blocks and scope.
You need to check the error right after the Write call
err = ioutil.WriteFile(fi.Name(), []byte(Value), 0644)
check(err) <===== too late
Since WriteFile overwrite the all file, you could strings.Replace() to replace your word by its upper case equivalent:
r := string(read)
r = strings.Replace(r, sam, strings.ToUpper(sam), -1)
err := ioutil.WriteFile(fi.Name(), []byte(r), 0644)
For a replace which is case insensitive, use a regexp as in "How do I do a case insensitive regular expression in Go?".
The, use func (*Regexp) ReplaceAllString:
re := regexp.MustCompile("(?i)\\b"+sam+"\\b")
r = re.ReplaceAllString(r, strings.ToUpper(sam))
err := ioutil.WriteFile(fi.Name(), []byte(r), 0644)
Note the \b: word boundary to find the any word starting and ending with sam content (instead of finding substrings containing sam content).
If you want to replace substrings, simply drop the \b:
re := regexp.MustCompile("(?i)"+sam)
It's not clear what you want to do. My best guess is something like this:
package main
import (
"bytes"
"errors"
"fmt"
"io/ioutil"
"os"
)
func UpdateWord(filename string, data, word []byte) (int, error) {
n := 0
f, err := os.OpenFile(filename, os.O_WRONLY, 0644)
if err != nil {
return n, err
}
uWord := bytes.ToUpper(word)
if len(word) < len(uWord) {
err := errors.New("Upper case longer than lower case:" + string(word))
return n, err
}
if len(word) > len(uWord) {
uWord = append(uWord, bytes.Repeat([]byte{' '}, len(word))...)[:len(word)]
}
off := int64(0)
for {
i := bytes.Index(data[off:], word)
if i < 0 {
break
}
off += int64(i)
_, err = f.WriteAt(uWord, off)
if err != nil {
return n, err
}
n++
off += int64(len(word))
}
f.Close()
if err != nil {
return n, err
}
return n, nil
}
func main() {
// Test file
filename := `ltoucase.txt`
// Create test file
lcase := []byte(`update a bc def ghij update klmno pqrstu update vwxyz update`)
perm := os.FileMode(0644)
err := ioutil.WriteFile(filename, lcase, perm)
if err != nil {
fmt.Println(err)
return
}
// Read test file
data, err := ioutil.ReadFile(filename)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(data))
// Update word in test file
word := []byte("update")
n, err := UpdateWord(filename, data, word)
if err != nil {
fmt.Println(n, err)
return
}
fmt.Println(filename, string(word), n)
data, err = ioutil.ReadFile(filename)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(data))
}
Output:
update a bc def ghij update klmno pqrstu update vwxyz update
ltoucase.txt update 4
UPDATE a bc def ghij UPDATE klmno pqrstu UPDATE vwxyz UPDATE