This question already has answers here:
How can I read a whole file into a string variable
(7 answers)
Closed 4 years ago.
I'm trying to read "file.txt" and put the contents into a variable using Golang. Here is what I've tried...
package main
import (
"fmt"
"os"
"log"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
fmt.Print(file)
}
The file gets read successfully and the return from os.Open returns a type of *os.File
It depends on what you are trying to do.
file, err := os.Open("file.txt")
fmt.print(file)
The reason it outputs &{0xc082016240}, is because you are printing the pointer value of a file-descriptor (*os.File), not file-content. To obtain file-content, you may READ from a file-descriptor.
To read all file content(in bytes) to memory, ioutil.ReadAll
package main
import (
"fmt"
"io/ioutil"
"os"
"log"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
b, err := ioutil.ReadAll(file)
fmt.Print(b)
}
But sometimes, if the file size is big, it might be more memory-efficient to just read in chunks: buffer-size, hence you could use the implementation of io.Reader.Read from *os.File
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
buf := make([]byte, 32*1024) // define your buffer size here.
for {
n, err := file.Read(buf)
if n > 0 {
fmt.Print(buf[:n]) // your read buffer.
}
if err == io.EOF {
break
}
if err != nil {
log.Printf("read %d bytes: %v", n, err)
break
}
}
}
Otherwise, you could also use the standard util package: bufio, try Scanner. A Scanner reads your file in tokens: separator.
By default, scanner advances the token by newline (of course you can customise how scanner should tokenise your file, learn from here the bufio test).
package main
import (
"fmt"
"os"
"log"
"bufio"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
scanner := bufio.NewScanner(file)
for scanner.Scan() { // internally, it advances token based on sperator
fmt.Println(scanner.Text()) // token in unicode-char
fmt.Println(scanner.Bytes()) // token in bytes
}
}
Lastly, I would also like to reference you to this awesome site: go-lang file cheatsheet. It encompassed pretty much everything related to working with files in go-lang, hope you'll find it useful.
Related
I am trying to write a small code in Go that will collect and save stats from IPFS.
So my Go code will execute IPFS command and save its output in .txt file and keep updating that .txt file.
I am having trouble doing that.
This is my code:
package main
import (
"fmt"
"io"
"log"
"os"
"os/exec"
"time"
)
func ipfsCommand() (ipfsOutput string) {
// output and error
out, err := exec.Command("ipfs","stats","bitswap","--human").Output()
// if there are errors, print/log them
if err != nil {
log.Printf("error!")
log.Fatal(err)
} else {
log.Printf("no error, printing output")
fmt.Printf("%s", out)
}
return
}
func writeToFile(message string) error {
f, err := os.Create("outputTest2_2.txt")
if err != nil {
return err
}
defer f.Close()
l, err := io.WriteString(f, message)
if err != nil {
fmt.Println(err)
f.Close()
return err
}
fmt.Println(l, "bytes written successfully")
return f.Sync()
}
func main() {
// get current time
currentTime := time.Now()
fmt.Println("YYYY.MM.DD : ", currentTime.Format("2006.01.02 15:04:05"))
writeToFile(currentTime)
// get output from ipfs command
msg := ipfsCommand()
// write the output to file
writeToFile(msg)
fmt.Println("file written!!!")
/* // write to file many times
for i:=0;i<3;i++{
// get output from ipfs command
msg := ipfsCommand()
// write the output to file
writeToFile(msg)
}*/
}
When the above code is run, this is the error:
# command-line-arguments
.\test2.go:49:13: cannot use currentTime (type time.Time) as type string in argument to writeToFile
Again, I want to get output from IPFS and save it to .txt file along with current time. I want to do this in loop because I want to save output from IPFS over a long period of time.
I tried to fix your script as is, but it just has too many issues. Here is a
rewrite, maybe you can use it as a new starting point:
package main
import (
"os"
"os/exec"
"time"
)
func main() {
f, err := os.Create("outputTest2_2.txt")
if err != nil {
panic(err)
}
defer f.Close()
currentTime, err := time.Now().MarshalText()
if err != nil {
panic(err)
}
f.Write(append(currentTime, '\n'))
msg, err := exec.Command("go", "env").Output()
if err != nil {
panic(err)
}
f.Write(msg)
}
I'm studying networks, and I'm doing a tcp server with Go. One of the challenges I'm studying is to send binaries or strings by a socket client to a server, save the server response to a txt, and compare it to the original data that was sent.
The problem is that the binaries do not arrive completely on the server.
Server
package main
import (
"fmt"
"io"
"log"
"net"
)
func main() {
l, err := net.Listen("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if nil != err {
log.Println(err)
continue
}
defer conn.Close()
go ConnHandler(conn)
}
}
func ConnHandler(conn net.Conn) {
recvBuf := make([]byte, 4096)
for {
n, err := conn.Read(recvBuf)
if nil != err {
if io.EOF == err {
log.Println(err)
return
}
log.Println(err)
return
}
if 0 < n {
data := recvBuf[:n]
fmt.Println(string(data))
}
}
}
Client
package main
import (
"fmt"
"log"
"net"
)
func main() {
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
var s string
fmt.Scanln(&s)
conn.Write([]byte(s))
conn.Close()
}
I'm generating the binaries using the command on linux:
head -c100000 /dev/urandom > binary_message.txt
I run the server:
./server > result.txt
And I send this data by the client using:
./client < binary_data.txt
In the end the file binary_data.txt have 98KB but the result .txt only has 0KB.
The problem is with scanning the binary from input. You didn't see it because the errors were ignored and not printed or otherwise handled. fmt.Scanln returns an error (so does the Write function). You should always check for possible errors happening.
I rewrote the client to load the file from disk itself as I don't think using stdin is a good fit for binary data.
package main
import (
"flag"
"io"
"log"
"net"
"os"
)
var fileName = flag.String("file", "", "file to send")
func main() {
flag.Parse()
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer conn.Close()
f, err := os.Open(*fileName)
if err != nil {
log.Println(err)
return
}
defer f.Close()
b := make([]byte, 1024)
for {
n, err := f.Read(b)
if err != nil {
if err == io.EOF {
log.Println("Done sending")
return
}
log.Println(err)
return
}
if _, err := conn.Write(b[:n]); err != nil {
log.Println(err)
return
}
}
}
You can use it with:
go run . -file=binary_message.txt
or if you have built the binary:
./client -file=binary_message.txt
I suggest you do the same for the server. Open a file for writing and write the binary data into that file. Use a flag to pass in the filename to write to. That will be cleaner than piping stdout to a file.
I am try to use go-bindata and packr, but those packages do not show how to pack an SQLite database file in to a binary file.
I don't need to update the database in any way, I just want to read the data from it on startup.
How can I embed an SQLite database file in a Go binary file?
The SQLite driver can't read a database file from memory (e.g. from a byte slice). But you can write the data to a temporary file, and open that:
//go:generate go run gen.go
package main
import (
"database/sql"
"fmt"
"io/ioutil"
"log"
"os"
_ "github.com/mattn/go-sqlite3"
)
func main() {
// Create temporary file for database.
tmpDB, err := ioutil.TempFile("", "db*.sqlite3")
if err != nil {
log.Fatal(err)
}
// Remove this file after on exit.
defer func() {
err := os.Remove(tmpDB.Name())
if err != nil {
log.Print(err)
}
}()
// Write database to file.
_, err = tmpDB.Write(sqlDB)
if err != nil {
log.Print(err)
}
err = tmpDB.Close()
if err != nil {
log.Print(err)
}
// Open DB.
db, err := sql.Open("sqlite3", tmpDB.Name()+"?mode=ro")
if err != nil {
log.Fatal(err)
}
// Make sure it's loaded correct.
rows, err := db.Query("select * from test")
if err != nil {
log.Fatal(err)
}
for rows.Next() {
var c string
err := rows.Scan(&c)
if err != nil {
log.Fatal(err)
}
fmt.Println(c)
}
}
And you can write the database to db.go with something like:
// +build generate
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"strings"
)
func main() {
// Read source database file.
d, err := ioutil.ReadFile("source.sqlite3")
if err != nil {
log.Fatal(err)
}
fp, err := os.Create("db.go")
if err != nil {
log.Fatal(err)
}
_, err = fmt.Fprintf(fp, "// Code generated by gen.go; DO NOT EDIT.\n\n"+
"package main\n\n"+
"var sqlDB = %s\n", asbyte(d))
if err != nil {
log.Fatal(err)
}
}
// Write any data as byte array.
func asbyte(s []byte) string {
var b strings.Builder
for i, c := range s {
if i%19 == 0 {
b.WriteString("\n\t\t")
}
b.WriteString(fmt.Sprintf("%#x, ", c))
}
return "[]byte{" + b.String() + "}"
}
You can also use go-bindata or packr for that if you prefer, but I don't really see an advantage.
An alternative way is to use a memory database, which may be faster depending on what you want to do.
Embed the SQL schema and rows you want in your Go binary as strings.
Open a new memory database when your program starts (sql.Open("sqlite3",:memory:`) and create the schema and insert the rows.
There is no disk access with this method, so querying it will probably be a bit faster at the expensive of slower startup times (benchmark to be sure!)
I am trying to get the content of a publicly available file using ioutil.ReadFile() but it doesn't find the file: panic: open http://www.pdf995.com/samples/pdf.pdf: No such file or directory
Here's my code:
// Reading and writing files are basic tasks needed for
// many Go programs. First we'll look at some examples of
// reading files.
package main
import (
"fmt"
"io/ioutil"
)
// Reading files requires checking most calls for errors.
// This helper will streamline our error checks below.
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
fileInUrl, err := ioutil.ReadFile("http://www.pdf995.com/samples/pdf.pdf")
if err != nil {
panic(err)
}
fmt.Printf("HERE --- fileInUrl: %+v", fileInUrl)
}
Here's a go playground example
ioutil.ReadFile() does not support http.
If you look at the source code(https://golang.org/src/io/ioutil/ioutil.go?s=1503:1549#L42), open the file using os.Open.
I think I can do this coding.
package main
import (
"io"
"net/http"
"os"
)
func main() {
fileUrl := "http://www.pdf995.com/samples/pdf.pdf"
if err := DownloadFile("example.pdf", fileUrl); err != nil {
panic(err)
}
}
func DownloadFile(filepath string, url string) error {
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Write the body to file
_, err = io.Copy(out, resp.Body)
return err
}
but, go playgound not protocol(go error dial tcp: Protocol not available).
so, You have to do it PC.
The ability to read (and write) a text file into and out of a string array is I believe a fairly common requirement. It is also quite useful when starting with a language removing the need initially to access a database. Does one exist in Golang?
e.g.
func ReadLines(sFileName string, iMinLines int) ([]string, bool) {
and
func WriteLines(saBuff[]string, sFilename string) (bool) {
I would prefer to use an existing one rather than duplicate.
As of Go1.1 release, there is a bufio.Scanner API that can easily read lines from a file. Consider the following example from above, rewritten with Scanner:
package main
import (
"bufio"
"fmt"
"log"
"os"
)
// readLines reads a whole file into memory
// and returns a slice of its lines.
func readLines(path string) ([]string, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
var lines []string
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines = append(lines, scanner.Text())
}
return lines, scanner.Err()
}
// writeLines writes the lines to the given file.
func writeLines(lines []string, path string) error {
file, err := os.Create(path)
if err != nil {
return err
}
defer file.Close()
w := bufio.NewWriter(file)
for _, line := range lines {
fmt.Fprintln(w, line)
}
return w.Flush()
}
func main() {
lines, err := readLines("foo.in.txt")
if err != nil {
log.Fatalf("readLines: %s", err)
}
for i, line := range lines {
fmt.Println(i, line)
}
if err := writeLines(lines, "foo.out.txt"); err != nil {
log.Fatalf("writeLines: %s", err)
}
}
Note: ioutil is deprecated as of Go 1.16.
If the file isn't too large, this can be done with the ioutil.ReadFile and strings.Split functions like so:
content, err := ioutil.ReadFile(filename)
if err != nil {
//Do something
}
lines := strings.Split(string(content), "\n")
You can read the documentation on ioutil and strings packages.
Cannot update first answer.
Anyway, after Go1 release, there are some breaking changes, so I updated as shown below:
package main
import (
"os"
"bufio"
"bytes"
"io"
"fmt"
"strings"
)
// Read a whole file into the memory and store it as array of lines
func readLines(path string) (lines []string, err error) {
var (
file *os.File
part []byte
prefix bool
)
if file, err = os.Open(path); err != nil {
return
}
defer file.Close()
reader := bufio.NewReader(file)
buffer := bytes.NewBuffer(make([]byte, 0))
for {
if part, prefix, err = reader.ReadLine(); err != nil {
break
}
buffer.Write(part)
if !prefix {
lines = append(lines, buffer.String())
buffer.Reset()
}
}
if err == io.EOF {
err = nil
}
return
}
func writeLines(lines []string, path string) (err error) {
var (
file *os.File
)
if file, err = os.Create(path); err != nil {
return
}
defer file.Close()
//writer := bufio.NewWriter(file)
for _,item := range lines {
//fmt.Println(item)
_, err := file.WriteString(strings.TrimSpace(item) + "\n");
//file.Write([]byte(item));
if err != nil {
//fmt.Println("debug")
fmt.Println(err)
break
}
}
/*content := strings.Join(lines, "\n")
_, err = writer.WriteString(content)*/
return
}
func main() {
lines, err := readLines("foo.txt")
if err != nil {
fmt.Println("Error: %s\n", err)
return
}
for _, line := range lines {
fmt.Println(line)
}
//array := []string{"7.0", "8.5", "9.1"}
err = writeLines(lines, "foo2.txt")
fmt.Println(err)
}
You can use os.File (which implements the io.Reader interface) with the bufio package for that. However, those packages are build with fixed memory usage in mind (no matter how large the file is) and are quite fast.
Unfortunately this makes reading the whole file into the memory a bit more complicated. You can use a bytes.Buffer to join the parts of the line if they exceed the line limit. Anyway, I recommend you to try to use the line reader directly in your project (especially if do not know how large the text file is!). But if the file is small, the following example might be sufficient for you:
package main
import (
"os"
"bufio"
"bytes"
"fmt"
)
// Read a whole file into the memory and store it as array of lines
func readLines(path string) (lines []string, err os.Error) {
var (
file *os.File
part []byte
prefix bool
)
if file, err = os.Open(path); err != nil {
return
}
reader := bufio.NewReader(file)
buffer := bytes.NewBuffer(make([]byte, 1024))
for {
if part, prefix, err = reader.ReadLine(); err != nil {
break
}
buffer.Write(part)
if !prefix {
lines = append(lines, buffer.String())
buffer.Reset()
}
}
if err == os.EOF {
err = nil
}
return
}
func main() {
lines, err := readLines("foo.txt")
if err != nil {
fmt.Println("Error: %s\n", err)
return
}
for _, line := range lines {
fmt.Println(line)
}
}
Another alternative might be to use io.ioutil.ReadAll to read in the complete file at once and do the slicing by line afterwards. I don't give you an explicit example of how to write the lines back to the file, but that's basically an os.Create() followed by a loop similar to that one in the example (see main()).
func readToDisplayUsingFile1(f *os.File){
defer f.Close()
reader := bufio.NewReader(f)
contents, _ := ioutil.ReadAll(reader)
lines := strings.Split(string(contents), '\n')
}
or
func readToDisplayUsingFile1(f *os.File){
defer f.Close()
slice := make([]string,0)
reader := bufio.NewReader(f)
for{
str, err := reader.ReadString('\n')
if err == io.EOF{
break
}
slice = append(slice, str)
}