Transfer contents of directory over net's TCP connection - go

I am currently learning Go and I am trying to send the contents of a directory to another machine over a plain tcp connection using Go's net package.
It works fine with individual files and small folders, but I run into issues if the folder contains many subfolders and larger files. I am using the filepath.Walk function to traverse over all files in the given directory. For each file or directory I send, I also send a header that provides the receiver with file name, file size, isDir properties so I know for how long I need to read for when reading the content. The issue I am having is that after a while when reading the header, I am reading actual file content of the previous file even though I already read that file from the connection
Here is the writer side. I simply traverse over the directory.
func transferDir(session *Session, dir string) error {
return filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header := Header{Name: info.Name(), Size: info.Size(), Path: path}
if info.IsDir() {
header.SetDirBit()
session.WriteHeader(header)
return nil // nothing more to write
}
// content is a file. write the file now byte by byte
file, err := os.Open(path)
inf, err := file.Stat()
header.Size = inf.Size() // get the true size of the file
session.WriteHeader(header)
defer file.Close()
if err != nil {
return err
}
buf := make([]byte, BUF_SIZE)
for {
n, err := file.Read(buf)
if err != nil {
if err == io.EOF {
session.Write(buf[:n])
session.Flush()
break
} else {
log.Println(err)
return err
}
}
session.Write(buf[:n])
session.Flush()
}
return nil
})
And here is the reader part
func (c *Clone) readFile(h Header) error {
file, err := os.Create(h.Path)
defer file.Close()
if err != nil {
return err
}
var receivedByts int64
fmt.Printf("Reading File: %s Size: %d\n", h.Name, h.Size)
for {
if (h.Size - receivedByts) < BUF_SIZE {
n, err := io.CopyN(file, c.sesh, (h.Size - receivedByts))
fmt.Println("Written: %d err: %s\n", n, err)
break
}
n, err := io.CopyN(file, c.sesh, BUF_SIZE)
fmt.Println("Written: %d err: %s\n", n, err)
receivedByts += BUF_SIZE
fmt.Println("Bytes Read: ", receivedByts)
}
return nil
}
Now the weird part is that when I am looking at the print statements I see something like:
Reading File: test.txt Size: 14024
Written 1024 nil
Bytes Read 1024
... This continues all the way to the break statement
And the total of the Bytes read equals the actual file size. Yet, the subsequent read for the header will return content from the test.txt file. Almost like there is still stuff in the buffer, but I think I read it already....

Related

Blocking read from input file continuous streaming

I have a file open that I am constantly writing to in Golang and reading in julia live. For some reason my code fails that it read EOF at the first blocking read. What am I doing wrong?
ERROR: LoadError: EOFError: read end of file
R.buffer = open("data/live.bin", "r")
if !isopen(R.buffer)
return nothing
end
# Error is here, why would it say EOF if the file is still being written to and open?
# Read the length in first 4 bytes
msglen = read(R.buffer, UInt32)
# Then read up to that length
bytes = read(R.buffer, msglen)
Note the file is still open by golang while julia is reading.
It gets traced to an error on line 399 here https://github.com/JuliaLang/julia/blob/5584620db87603fb8be313092712a9f052b8876f/base/iostream.jl#L399
I have a feeling I need to ccall some methods to get this working.
And here is the golang code
enc, err := os.OpenFile("data/live.bin", os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0660)
if err != nil {
log.Error("Error in File Create!", err.Error())
panic(err)
}
msgLen := make([]byte, 4)
for {
msg := []byte("mymessage")
binary.LittleEndian.PutUint32(msgLen, uint32(len(msg)))
_, err := enc.Write(msgLen)
if err != nil {
log.Fatal(err)
}
// Write message
_, err = enc.Write(msg)
if err != nil {
log.Fatal(err)
}
time.Sleep(5 * time.Second)
}

G110: Potential DoS vulnerability via decompression bomb (gosec)

I'm getting the following golintci message:
testdrive/utils.go:92:16: G110: Potential DoS vulnerability via decompression bomb (gosec)
if _, err := io.Copy(targetFile, fileReader); err != nil {
^
Read the corresponding CWE and I'm not clear on how this is expected to be corrected.
Please offer pointers.
func unzip(archive, target string) error {
reader, err := zip.OpenReader(archive)
if err != nil {
return err
}
for _, file := range reader.File {
path := filepath.Join(target, file.Name) // nolint: gosec
if file.FileInfo().IsDir() {
if err := os.MkdirAll(path, file.Mode()); err != nil {
return err
}
continue
}
fileReader, err := file.Open()
if err != nil {
return err
}
defer fileReader.Close() // nolint: errcheck
targetFile, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.Mode())
if err != nil {
return err
}
defer targetFile.Close() // nolint: errcheck
if _, err := io.Copy(targetFile, fileReader); err != nil {
return err
}
}
return nil
}
The warning you get comes from a rule provided in gosec.
The rule specifically detects usage of io.Copy on file decompression.
This is a potential issue because io.Copy:
copies from src to dst until either EOF is reached on src or an error occurs.
So, a malicious payload might cause your program to decompress an unexpectedly big amount of data and go out of memory, causing denial of service as mentioned in the warning message.
In particular, gosec will check (source) the AST of your program and warn you about usage of io.Copy or io.CopyBuffer together with any one of the following:
"compress/gzip".NewReader
"compress/zlib".NewReader or NewReaderDict
"compress/bzip2".NewReader
"compress/flate".NewReader or NewReaderDict
"compress/lzw".NewReader
"archive/tar".NewReader
"archive/zip".NewReader
"*archive/zip".File.Open
Using io.CopyN removes the warning because (quote) it "copies n bytes (or until an error) from src to dst", thus giving you (the program writer) control of how many bytes to copy. So you could pass an arbitrarily large n that you set based on the available resources of your application, or copy in chunks.
Based on various pointers provided, replaced
if _, err := io.Copy(targetFile, fileReader); err != nil {
return err
}
with
for {
_, err := io.CopyN(targetFile, fileReader, 1024)
if err != nil {
if err == io.EOF {
break
}
return err
}
}
PS while this helps memory footprint, this wouldn't help a DDOS attack copying very long and/or infinite stream ...
Assuming that you're working on compressed data, you need to use io.CopyN.
You can try a workaround with --nocompress flag. But this will cause the data to be included uncompressed.
See the following PR and related issue : https://github.com/go-bindata/go-bindata/pull/50

Recently copied file returns all 0s as byte array

I have a go file server that can receive requests of files up 10GB in size. To keep memory usage low I read the multipart form data into a tmp file. I know behind the scenes FormFile does the same but I still need to transfer it to a regular file for some post upload processing.
f, header, err := r.FormFile("file")
if err != nil {
return nil, fmt.Errorf("could not get file from request %w", err)
}
tmpFile, err := ioutil.TempFile("", "oriio-")
if err != nil {
return nil, err
}
if _, err := io.Copy(tmpFile, f); err != nil {
return nil, fmt.Errorf("could not copy request body to file %w", err)
}
After this I need to grab the first 261 bytes of the file to determine its MIME type.
head := make([]byte, 261)
if _, err := tmpFile.Read(head); err != nil {
return nil, err
}
The issue I'm running into is if I try to read directly from tmpFile the byte array returns 261 0 when I print fmt.Prinf("%x", head) aka invalid data. To verify the data is valid I was saving it to a regular file opening it in my system and the file (in this case an image file) was perfectly in tact. So it is not a corrupt file issue. To get around the problem I now close the tmp file and then reopen it again and that seems to fix everything.
tmpFile, err := ioutil.TempFile("", "oriio-")
if err != nil {
return nil, err
}
if _, err := io.Copy(tmpFile, f); err != nil {
return nil, fmt.Errorf("could not copy request body to file %w", err)
}
tmpFile.Close()
tmpFile, err = os.Open(tmpFile.Name())
if err != nil {
panic(err)
}
head := make([]byte, 261)
if _, err := tmpFile.Read(head); err != nil {
return nil, err
}
Now when I print out the head byte array the proper content is printed. Why is this? Is there some sort of Sync or Flush I have to do with the original tmp file to make it work?
Reading/writing a file changes the current location in the file. After copy, the tmpFile is positioned at the end, so reading from it will read 0 bytes. You have to seek first if you want to read from the beginning of the file:
io.Copy(tmpFile, f)
tmpFile.Seek(0,0)
tmpFile.Read(head)

Out of memory when transferring large file

I'm using Go Gob to transfer large files (~ 1 GB) or many small files (~ 30 MB). Server is running in a loop, and will receive files when clients send it.
My code is working if I send one large file, or few small files, but when sending a large file for the second time, it returns a 'fatal error: runtime: out of memory'. If I send a large file, stops the program, then starts again and send another large file, it works.
It looks after receiving file via Gob and writing into a file, it is not releasing memory.
Server code
type FileGob struct {
FileName string
FileLen int
FileContent []byte
}
func handleConnection(conn net.Conn) {
transf := &FileGob{}
dec := gob.NewDecoder(conn)
err := dec.Decode(transf) // file from conn to var transf
if err != nil {
fmt.Println("error to decode into buffer:", err)
}
conn.Close()
file, err := os.Create("sent_" + transf.FileName)
if err != nil {
fmt.Println("error to create file:", err)
}
file.Write(transf.FileContent) // writes into file
fileStat, err := file.Stat()
if err != nil {
fmt.Println("error to get File Stat:", err)
}
file.Close()
fmt.Printf("File %v was transferred\n", transf.FileName)
fmt.Printf("Transferred: %d, Expected: %d\n", fileStat.Size(), transf.FileLen)
}

Error on trying to read gzip files in golang

I am trying to read gzip files using compress/gzip. I am using http.DetectContentType as I do not know if I get a normal txt file or a gzipped one. My code is very straight forward and as below:
f, err := os.Open(fullpath)
if err != nil {
log.Panicf("Can not open file %s: %v", fullpath, err)
return ""
}
defer f.Close()
buff := make([]byte, 512)
_, err = f.Read(buff)
if err != nil && err != io.EOF{
log.Panicf("Cannot read buffer %v", err);
return ""
}
switch filetype := http.DetectContentType(buff); filetype {
case "application/x-gzip":
log.Println("File Type is", filetype)
reader, err := gzip.NewReader(f)
if err != nil && err != io.EOF{
log.Panicf("Cannot read gzip archive %v", err);
return ""
}
defer reader.Close()
target := "/xx/yy/abcd.txt"
writer, err := os.Create(target)
if err != nil {
log.Panicf("Cannot write unarchived file %v", err);
return ""
}
defer writer.Close()
_, err = io.Copy(writer, reader)
return target
The problem is that the gzip reader always errors out saying "Cannot read gzip archive gzip: invalid header" I have tried the zlib library too but in vain. I gzipped the source file in mac using the command line gzip tool.
Please show me where I am going wrong.
You're reading the first 512 bytes of the file, so the gzip.Reader won't ever see that. Since these are regular files, you can seek back to the start after a successful Read:
f.Seek(0, os.SEEK_SET)

Resources