This question already has an answer here:
Write and Read File with same *os.File in Go
(1 answer)
Closed 8 months ago.
This post was edited and submitted for review 8 months ago and failed to reopen the post:
Original close reason(s) were not resolved
In Go, I am trying to write data to a temp file that I then turn around and read but have not been successful. Below is a stripped down test program. I have verified that the data are being written to the file by inspecting the temporary file. So, at least I know that data are making it into the file. I just am then unable to read it out.
Thank you for your help in advance
package main
import (
"bufio"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
)
func main() {
tmpFile, err := ioutil.TempFile("", fmt.Sprintf("%s-", filepath.Base(os.Args[0])))
if err != nil {
log.Fatal("Could not create temporary file", err)
}
fmt.Println("Created temp file: ", tmpFile.Name())
// defer os.Remove(tmpFile.Name())
fmt.Println("Writing some data to the temp file")
if _, err = tmpFile.WriteString("test data"); err != nil {
log.Fatal("Unable to write to temporary file", err)
} else {
fmt.Println("data should have been written")
}
fmt.Println("Trying to read the temp file now")
s := bufio.NewScanner(tmpFile)
for s.Scan() {
fmt.Println(s.Text())
}
err = s.Err()
if err != nil {
log.Fatal("error reading temp file", err)
}
}
ioutil.TempFile creates a temp file and opens the file for reading and writing and returns the resulting *os.File (file descriptor). So when you're writing inside the file, the pointer is moved to that offset, i.e., it's currently at the end of the file.
But as your requirement is read from the file, you need to Seek back to the beginning or wherever desired offset using *os.File.Seek method. So, adding tmpFile.Seek(0, 0) will give you the desired behaviour.
Also, as a good practice, do not forget to close the file. Notice I've used defer tmpFile.Close() which closes the file before exiting.
Refer the following example:
package main
import (
"bufio"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
)
func main() {
tmpFile, err := ioutil.TempFile("", fmt.Sprintf("%s-", filepath.Base(os.Args[0])))
if err != nil {
log.Fatal("Could not create temporary file", err)
}
defer tmpFile.Close()
fmt.Println("Created temp file: ", tmpFile.Name())
fmt.Println("Writing some data to the temp file")
if _, err = tmpFile.WriteString("test data"); err != nil {
log.Fatal("Unable to write to temporary file", err)
} else {
fmt.Println("Data should have been written")
}
fmt.Println("Trying to read the temp file now")
// Seek the pointer to the beginning
tmpFile.Seek(0, 0)
s := bufio.NewScanner(tmpFile)
for s.Scan() {
fmt.Println(s.Text())
}
if err = s.Err(); err != nil {
log.Fatal("error reading temp file", err)
}
}
Update:
Comment from OP:
Is the deferred close needed given that deleting the actual file is also deferred? If so, I imagine order of deferral would matter.
So, that's a nice question. So the basic rule of thumb would be to close the file and then remove. So, it might even be possible to delete first and later close it, but that is OS-dependent.
If you refer C++'s doc:
If the file is currently open by the current or another process, the behavior of this function is implementation-defined (in particular, POSIX systems unlink the file name, although the file system space is not reclaimed even if this was the last hardlink to the file until the last running process closes the file, Windows does not allow the file to be deleted)
So, on Windows, that's a problem for sure if you try deleting it first without closing it.
So, as defer's are stacked, so the order of execution would be
defer os.Remove(tmpFile.Name()) // Called 2nd
defer tmpFile.Close() // Called 1st
package main
import (
"fmt"
"io/ioutil"
"log"
)
func main() {
content, err := ioutil.ReadFile("testdata/hello")
if err != nil {
log.Fatal(err)
}
fmt.Printf("File contents: %s", content)
according to the official golang docs.
Related
I have downloaded over 500 Gb of data to a single directory off of AWS.
Whenever I try to access that directory, the command line hangs and doesn't show me anything.
I'm trying to run some code that will interact with the files by printing out the path of each file but the command line hangs and then exits the program.
The program definitely starts execution because "Printing file path's" gets displayed to the console.
func main() {
fmt.Println("Printing file path's")
err := filepath.Walk(source,
func(fpath string, info os.FileInfo, err error) {
if !info.IsDir() && file path.Ext(fpath)==".txt" {
fmt.Println(fpath)
}
}
}
}
How should I handle the situation of being able to view all the files in the command line and why is this program not working?
UPDATE:
By using
files, err := dir.Readdir(10)
if err == io.EOF {
break
}
I was able to snap up the first 10 folders/files in the directory.
Using a loop I could keep doing this until I hit the end of the directory.
This doesn't rely on ordering the files/folders as does the walk function and so its more efficient.
The possible performance issue with filepath.Walk is clearly documented:
The files are walked in lexical order, which makes the output deterministic but means that for very large directories Walk can be inefficient.
Use os.File.Readdir to iterate files in filesystem order:
Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by Lstat, in directory order. Subsequent calls on the same file will yield further FileInfos.
package main
import (
"fmt"
"io"
"log"
"os"
"time"
)
func main() {
dir, err := os.Open("/tmp")
if err != nil {
log.Fatal(err)
}
for {
files, err := dir.Readdir(10)
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
for _, fi := range files {
classifier := ""
if fi.IsDir() {
classifier = "/"
}
fmt.Printf("%v %12d %s%s\n",
fi.ModTime().UTC().Truncate(time.Second),
fi.Size(),
fi.Name(), classifier,
)
}
}
}
I'm new to golang and using ioutil.ReadFile(os.Args[1]) to fetch a file path from the cli and then processing each line using:
package main
import (
"bufio"
"fmt"
"log"
"os"
)
func main() {
file, err := os.Open(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
}
Now I can get the output at stdout. But also get the following ERROR in the end:
msg:"BRUMBRUM";reference:cve,CVE-2007-2810;reference:blah;
msg:"WAKANDA";reference:cve,CVE-2007-2810;reference:blah; file name too long
exit status 1
My file path input is data/srulz.tcl
. FYI, the error message is not a part of the text file.
I need to know where is this going wrong here and how can this be improved?
UPDATE:
Provided issue reproducible code.
Go-ing with flag pkg for now until this mystery is solved
In the first line,
filePath, err := ioutil.ReadFile(os.Args[1])
Above step will read the whole file contents and return slice of byte and error. filePath variable will not store the file path instead its storing the content of file in bytes. I am wondering why are you not getting compile time error as filepath variable is slice of bytes whereas os.Open(filepath) the argument to os.Open will be string.
There are certain fmt.Print statements that I want to save into a .txt file.
I don't want to store all print statments. Can I do this ?
package main
import (
"fmt"
"io"
"log"
"os"
)
func main() {
file, err := os.Create("myfile")
if err != nil {
log.Fatal(err)
}
mw := io.MultiWriter(os.Stdout, file)
fmt.Fprintln(mw, "This line will be written to stdout and also to a file")
}
Use the fmt.Fprint() method for calls you want to save to a file. There are also fmt.Fprintf() and fmt.Fprintln().
These functions take a destination io.Writer as the first argument, to which you can pass your file (*os.File).
For example:
f, err := os.Open("data.txt")
if err != nil {
log.Fatal(err)
}
defer f.Close()
fmt.Println("This goes to standard output.")
fmt.Fprintln(f, "And this goes to the file")
fmt.Fprintf(f, "Also to file, with some formatting. Time: %v, line: %d\n",
time.Now(), 2)
If you want all fmt.PrintXX() calls to go to the file which you have no control over (e.g. you can't change them to fmt.FprintXX() because they are part of another library), you may change os.Stdout temporarily, so all further fmt.PrintXX() calls will write to the output you set, e.g.:
// Temporarily set your file as the standard output (and save the old)
old, os.Stdout = os.Stdout, f
// Now all fmt.PrintXX() calls output to f
somelib.DoSomething()
// Restore original standard output
os.Stdout = old
I want to do this:
Read a line from a text file.
Process the line.
Delete the line.
My first thought was to read the entire file into memory with ioutil.Readfile(),
but I'm not sure how to update the text file after the line has been processed,
and what happens if extra lines is added to the text file after it has been read into memory?
I normally write shell scripts and would do something like this:
while read -r line; do
echo "${line}"
sed -i 1d "${myList}"
done < "${myList}"
What is the best way to do this in Golang?
Use the bufio package.
Here's the basic syntax for opening a text file and looping through each line.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// Open the file.
f, _ := os.Open("C:\\programs\\file.txt")
// Create a new Scanner for the file.
scanner := bufio.NewScanner(f)
// Loop over all lines in the file and print them.
for scanner.Scan() {
line := scanner.Text()
fmt.Println(line)
}
}
you have some options:
1- read file, process it, then write it back (you need to lock that file).
2- use binary file and invent (make use of) special data structure (like linked list) to optimize text processing (with line locking).
3- use ready made databases.
4- use Virtual filesystem inside your file, and treat each line like one file, see: https://github.com/lotrfan/vfs and https://github.com/blang/vfs
using file manager (like database server) solves the file locking dilemma.
and if the purpose of using file is one way communication which sender program just adds new line and receiver program just removes it, it is better to use os pipes (named pipe (FIFO)) or other interop methods.
see for Linux: Unix FIFO in go?
for Windows: https://github.com/natefinch/npipe
sample file writer:
package main
import (
"bufio"
"fmt"
"os"
"time"
)
func main() {
f, err := os.OpenFile("/tmp/file.txt", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for i := 0; ; i++ {
w := bufio.NewWriter(f)
_, err := fmt.Fprintln(w, i)
if err != nil {
panic(err)
}
w.Flush() // Flush writes any buffered data to the underlying io.Writer.
f.Sync() // commit the current contents of the file to stable storage.
fmt.Println("write", i)
time.Sleep(500 * time.Millisecond)
}
}
sample file reader:
package main
import (
"fmt"
"os"
"time"
)
func main() {
f, err := os.OpenFile("/tmp/file.txt", os.O_RDWR, 0666)
if err != nil {
panic(err)
}
defer f.Close()
i := 0
for {
n, err := fmt.Fscanln(f, &i)
if n == 1 {
fmt.Println(i)
}
if err != nil {
fmt.Println(err)
return
}
time.Sleep(500 * time.Millisecond)
}
}
I've modified the official documentation example for the zlib package to use an opened file rather than a set of hardcoded bytes (code below).
The code reads in the contents of a source text file and compresses it with the zlib package. I then try to read back the compressed file and print its decompressed contents into stdout.
The code doesn't error, but it also doesn't do what I expect it to do; which is to display the decompressed file contents into stdout.
Also: is there another way of displaying this information, rather than using io.Copy?
package main
import (
"compress/zlib"
"io"
"log"
"os"
)
func main() {
var err error
// This defends against an error preventing `defer` from being called
// As log.Fatal otherwise calls `os.Exit`
defer func() {
if err != nil {
log.Fatalln("\nDeferred log: \n", err)
}
}()
src, err := os.Open("source.txt")
if err != nil {
return
}
defer src.Close()
dest, err := os.Create("new.txt")
if err != nil {
return
}
defer dest.Close()
zdest := zlib.NewWriter(dest)
defer zdest.Close()
if _, err := io.Copy(zdest, src); err != nil {
return
}
n, err := os.Open("new.txt")
if err != nil {
return
}
r, err := zlib.NewReader(n)
if err != nil {
return
}
defer r.Close()
io.Copy(os.Stdout, r)
err = os.Remove("new.txt")
if err != nil {
return
}
}
Your defer func doesn't do anything, because you're shadowing the err variable on every new assignment. If you want a defer to run, return from a separate function, and call log.Fatal after the return statement.
As for why you're not seeing any output, it's because you're deferring all the Close calls. The zlib.Writer isn't flushed until after the function exits, and neither is the destination file. Call Close() explicitly where you need it.
zdest := zlib.NewWriter(dest)
if _, err := io.Copy(zdest, src); err != nil {
log.Fatal(err)
}
zdest.Close()
dest.Close()
I think you messed up the code logic with all this defer stuff and your "trick" err checking.
Files are definitively written when flushed or closed. You just copy into new.txt without closing it before opening it to read it.
Defering the closing of the file is neat inside a function which has multiple exits: It makes sure the file is closed once the function is left. But your main requires the new.txt to be closed after the copy, before re-opening it. So don't defer the close here.
BTW: Your defense against log.Fatal terminating the code without calling your defers is, well, at least strange. The files are all put into some proper state by the OS, there is absolutely no need to complicate the stuff like this.
Check the error from the second Copy:
2015/12/22 19:00:33
Deferred log:
unexpected EOF
exit status 1
The thing is, you need to close zdest immediately after you've done writing. Close it after the first Copy and it works.
I would have suggested to use io.MultiWriter.
In this way you read only once from src. Not much gain for small files but is faster for bigger files.
w := io.MultiWriter(dest, os.Stdout)