I'm using Go Gob to transfer large files (~ 1 GB) or many small files (~ 30 MB). Server is running in a loop, and will receive files when clients send it.
My code is working if I send one large file, or few small files, but when sending a large file for the second time, it returns a 'fatal error: runtime: out of memory'. If I send a large file, stops the program, then starts again and send another large file, it works.
It looks after receiving file via Gob and writing into a file, it is not releasing memory.
Server code
type FileGob struct {
FileName string
FileLen int
FileContent []byte
}
func handleConnection(conn net.Conn) {
transf := &FileGob{}
dec := gob.NewDecoder(conn)
err := dec.Decode(transf) // file from conn to var transf
if err != nil {
fmt.Println("error to decode into buffer:", err)
}
conn.Close()
file, err := os.Create("sent_" + transf.FileName)
if err != nil {
fmt.Println("error to create file:", err)
}
file.Write(transf.FileContent) // writes into file
fileStat, err := file.Stat()
if err != nil {
fmt.Println("error to get File Stat:", err)
}
file.Close()
fmt.Printf("File %v was transferred\n", transf.FileName)
fmt.Printf("Transferred: %d, Expected: %d\n", fileStat.Size(), transf.FileLen)
}
Related
I have a file open that I am constantly writing to in Golang and reading in julia live. For some reason my code fails that it read EOF at the first blocking read. What am I doing wrong?
ERROR: LoadError: EOFError: read end of file
R.buffer = open("data/live.bin", "r")
if !isopen(R.buffer)
return nothing
end
# Error is here, why would it say EOF if the file is still being written to and open?
# Read the length in first 4 bytes
msglen = read(R.buffer, UInt32)
# Then read up to that length
bytes = read(R.buffer, msglen)
Note the file is still open by golang while julia is reading.
It gets traced to an error on line 399 here https://github.com/JuliaLang/julia/blob/5584620db87603fb8be313092712a9f052b8876f/base/iostream.jl#L399
I have a feeling I need to ccall some methods to get this working.
And here is the golang code
enc, err := os.OpenFile("data/live.bin", os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0660)
if err != nil {
log.Error("Error in File Create!", err.Error())
panic(err)
}
msgLen := make([]byte, 4)
for {
msg := []byte("mymessage")
binary.LittleEndian.PutUint32(msgLen, uint32(len(msg)))
_, err := enc.Write(msgLen)
if err != nil {
log.Fatal(err)
}
// Write message
_, err = enc.Write(msg)
if err != nil {
log.Fatal(err)
}
time.Sleep(5 * time.Second)
}
I have a go file server that can receive requests of files up 10GB in size. To keep memory usage low I read the multipart form data into a tmp file. I know behind the scenes FormFile does the same but I still need to transfer it to a regular file for some post upload processing.
f, header, err := r.FormFile("file")
if err != nil {
return nil, fmt.Errorf("could not get file from request %w", err)
}
tmpFile, err := ioutil.TempFile("", "oriio-")
if err != nil {
return nil, err
}
if _, err := io.Copy(tmpFile, f); err != nil {
return nil, fmt.Errorf("could not copy request body to file %w", err)
}
After this I need to grab the first 261 bytes of the file to determine its MIME type.
head := make([]byte, 261)
if _, err := tmpFile.Read(head); err != nil {
return nil, err
}
The issue I'm running into is if I try to read directly from tmpFile the byte array returns 261 0 when I print fmt.Prinf("%x", head) aka invalid data. To verify the data is valid I was saving it to a regular file opening it in my system and the file (in this case an image file) was perfectly in tact. So it is not a corrupt file issue. To get around the problem I now close the tmp file and then reopen it again and that seems to fix everything.
tmpFile, err := ioutil.TempFile("", "oriio-")
if err != nil {
return nil, err
}
if _, err := io.Copy(tmpFile, f); err != nil {
return nil, fmt.Errorf("could not copy request body to file %w", err)
}
tmpFile.Close()
tmpFile, err = os.Open(tmpFile.Name())
if err != nil {
panic(err)
}
head := make([]byte, 261)
if _, err := tmpFile.Read(head); err != nil {
return nil, err
}
Now when I print out the head byte array the proper content is printed. Why is this? Is there some sort of Sync or Flush I have to do with the original tmp file to make it work?
Reading/writing a file changes the current location in the file. After copy, the tmpFile is positioned at the end, so reading from it will read 0 bytes. You have to seek first if you want to read from the beginning of the file:
io.Copy(tmpFile, f)
tmpFile.Seek(0,0)
tmpFile.Read(head)
I am currently learning Go and I am trying to send the contents of a directory to another machine over a plain tcp connection using Go's net package.
It works fine with individual files and small folders, but I run into issues if the folder contains many subfolders and larger files. I am using the filepath.Walk function to traverse over all files in the given directory. For each file or directory I send, I also send a header that provides the receiver with file name, file size, isDir properties so I know for how long I need to read for when reading the content. The issue I am having is that after a while when reading the header, I am reading actual file content of the previous file even though I already read that file from the connection
Here is the writer side. I simply traverse over the directory.
func transferDir(session *Session, dir string) error {
return filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header := Header{Name: info.Name(), Size: info.Size(), Path: path}
if info.IsDir() {
header.SetDirBit()
session.WriteHeader(header)
return nil // nothing more to write
}
// content is a file. write the file now byte by byte
file, err := os.Open(path)
inf, err := file.Stat()
header.Size = inf.Size() // get the true size of the file
session.WriteHeader(header)
defer file.Close()
if err != nil {
return err
}
buf := make([]byte, BUF_SIZE)
for {
n, err := file.Read(buf)
if err != nil {
if err == io.EOF {
session.Write(buf[:n])
session.Flush()
break
} else {
log.Println(err)
return err
}
}
session.Write(buf[:n])
session.Flush()
}
return nil
})
And here is the reader part
func (c *Clone) readFile(h Header) error {
file, err := os.Create(h.Path)
defer file.Close()
if err != nil {
return err
}
var receivedByts int64
fmt.Printf("Reading File: %s Size: %d\n", h.Name, h.Size)
for {
if (h.Size - receivedByts) < BUF_SIZE {
n, err := io.CopyN(file, c.sesh, (h.Size - receivedByts))
fmt.Println("Written: %d err: %s\n", n, err)
break
}
n, err := io.CopyN(file, c.sesh, BUF_SIZE)
fmt.Println("Written: %d err: %s\n", n, err)
receivedByts += BUF_SIZE
fmt.Println("Bytes Read: ", receivedByts)
}
return nil
}
Now the weird part is that when I am looking at the print statements I see something like:
Reading File: test.txt Size: 14024
Written 1024 nil
Bytes Read 1024
... This continues all the way to the break statement
And the total of the Bytes read equals the actual file size. Yet, the subsequent read for the header will return content from the test.txt file. Almost like there is still stuff in the buffer, but I think I read it already....
I'm writing a simple tcp server in golang that writes to "output.txt". All the "server part" works fine, but it does not write to file.(but it does create "output.txt")
func handleRequest(conn net.Conn) {
// Make a buffer to hold incoming data.
buf := make([]byte, 1024)
// Read the incoming connection into the buffer.
size, err := conn.Read(buf)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
s := string(buf[:size])
fmt.Println(s)
perm := os.FileMode(0644)
err = ioutil.WriteFile("output.txt", buf, perm)
check(err)
// Close the connection when you're done with it.
conn.Close()
}
Why does it not write to file and how to fix this?
I am trying to read gzip files using compress/gzip. I am using http.DetectContentType as I do not know if I get a normal txt file or a gzipped one. My code is very straight forward and as below:
f, err := os.Open(fullpath)
if err != nil {
log.Panicf("Can not open file %s: %v", fullpath, err)
return ""
}
defer f.Close()
buff := make([]byte, 512)
_, err = f.Read(buff)
if err != nil && err != io.EOF{
log.Panicf("Cannot read buffer %v", err);
return ""
}
switch filetype := http.DetectContentType(buff); filetype {
case "application/x-gzip":
log.Println("File Type is", filetype)
reader, err := gzip.NewReader(f)
if err != nil && err != io.EOF{
log.Panicf("Cannot read gzip archive %v", err);
return ""
}
defer reader.Close()
target := "/xx/yy/abcd.txt"
writer, err := os.Create(target)
if err != nil {
log.Panicf("Cannot write unarchived file %v", err);
return ""
}
defer writer.Close()
_, err = io.Copy(writer, reader)
return target
The problem is that the gzip reader always errors out saying "Cannot read gzip archive gzip: invalid header" I have tried the zlib library too but in vain. I gzipped the source file in mac using the command line gzip tool.
Please show me where I am going wrong.
You're reading the first 512 bytes of the file, so the gzip.Reader won't ever see that. Since these are regular files, you can seek back to the start after a successful Read:
f.Seek(0, os.SEEK_SET)