I have the following bit of code:
imageFile, _ := os.Open("image.png") // returns *os.File, error
decodedImage, _, _ := image.Decode(imageFile) // returns image.Image, string, error
// imageFile has been modified!
When I try to work with imageFile after calling image.Decode it no longer behaves the same, leading me to believe image.Decode modified imageFile in some way.
Why is image.Decode modifying the original value while at the same time returning a new value for decodedImage - isn't this misleading?
How do I retain the original value? Is there a way to make a "real" copy of the file, pointing to a new part of allocated memory?
I just started out with Go, so apologies if I am missing something obvious here.
The file on disk is not modified by the code in the question.
The current position of imageFile is somewhere past the beginning of the file after the image is decoded. To read the fie again, Seek back the beginning of the file:
imageFile, err := os.Open("image.png")
if err != nil { log.Fatal(err) }
decodedImage, _, err := image.Decode(imageFile)
if err != nil { log.Fatal(err) }
// Rewind back to the start of the file.
_, err := imageFile.Seek(0, io.SeekStart)
if err != nil { log.Fatal(err) }
// Do something with imageFile here.
Replace the log.Fatal error handling with whatever is appropriate for your application.
Related
i have a file. it has some ip
1.1.1.0/24
1.1.2.0/24
2.2.1.0/24
2.2.2.0/24
i read this file to slice, and used *(*string)(unsafe.Pointer(&b)) to parse []byte to string, but is doesn't work
func TestInitIpRangeFromFile(t *testing.T) {
filepath := "/tmp/test"
file, err := os.Open(filepath)
if err != nil {
t.Errorf("failed to open ip range file:%s, err:%s", filepath, err)
}
reader := bufio.NewReader(file)
ranges := make([]string, 0)
for {
ip, _, err := reader.ReadLine()
if err != nil {
if err == io.EOF {
break
}
logger.Fatalf("failed to read ip range file, err:%s", err)
}
t.Logf("ip:%s", *(*string)(unsafe.Pointer(&ip)))
ranges = append(ranges, *(*string)(unsafe.Pointer(&ip)))
}
t.Logf("%v", ranges)
}
result:
task_test.go:71: ip:1.1.1.0/24
task_test.go:71: ip:1.1.2.0/24
task_test.go:71: ip:2.2.1.0/24
task_test.go:71: ip:2.2.2.0/24
task_test.go:75: [2.2.2.0/24 1.1.2.0/24 2.2.1.0/24 2.2.2.0/24]
why 1.1.1.0/24 changed to 2.2.2.0/24 ?
change
*(*string)(unsafe.Pointer(&ip))
to string(ip) it works
So, while reinterpreting a slice-header as a string-header the way you did is absolutely bonkers and has no guarantee whatsoever of working correctly, it's only indirectly the cause of your problem.
The real problem is that you're retaining a pointer to the return value of bufio/Reader.ReadLine(), but the docs for that method say "The returned buffer is only valid until the next call to ReadLine." Which means that the reader is free to reuse that memory later on, and that's what's happening.
When you do the cast in the proper way, string(ip), Go copies the contents of the buffer into the newly-created string, which remains valid in the future. But when you type-pun the slice into a string, you keep the exact same pointer, which stops working as soon as the reader refills its buffer.
If you decided to do the pointer trickery as a performance hack to avoid copying and allocation... too bad. The reader interface is going to force you to copy the data out anyway, and since it does, you should just use string().
I have a slightly funky issue in golang. Essentially I have a slice of strings which represent file paths. I then run a cat against those filepaths to combine the files before sorting, deduping, etc.
here is the section of code (where 'applicableReductions' is the string slice):
applicableReductions := []string{}
for _, fqFromListName := range fqFromListNames {
filePath := GetFilePath()
//BROKE CODE GOES HERE
}
applicableReductions = append(applicableReductions, filePath)
fileOut, err := os.Create(toListWriteTmpFilePath)
if err != nil {
return err
}
cat := exec.Command("cat", applicableReductions...)
catStdOut, err := cat.StdoutPipe()
if err != nil {
return err
}
go func(cat *exec.Cmd) error {
if err := cat.Start(); err != nil {
return fmt.Errorf("File reduction error (cat) : %s", err)
}
return nil
}(cat)
// Init Writer & write file
writer := bufio.NewWriter(fileOut)
defer writer.Flush()
_, err = io.Copy(writer, catStdOut)
if err != nil {
return err
}
if err = cat.Wait(); err != nil {
return err
}
fDiff.StandardiseData(fileOut, toListUpdateFolderPath, list.Name)
The above works fine. The problem comes when I try to append a new ele to the array. I have a seperate function which creates a new file from db content which is then added to the applicableReductions slice.
func RetrieveDomainsFromDB(collection *Collection, listName, outputPath string) error {
domains, err := domainReviews.GetDomainsForList(listName)
if err != nil {
return err
}
if len(domains) < 1 {
return ErrNoDomainReviewsForList
}
fh, err := os.OpenFile(outputPath, os.O_RDWR, 0774)
if err != nil {
fh, err = os.Create(outputPath)
if err != nil {
return err
}
}
defer fh.Close()
_, err = fh.WriteString(strings.Join(domains, "\n"))
if err != nil {
return err
}
return nil
}
If I call the above function and append the filePath to the applicableReduction slice, it is in there but doesnt get called by cat.
To clarify, when I put the following where it says BROKE CODE GOES HERE:
if dbSource {
err = r.RetrieveDomainsFromDB(collection, ToListName, filePath)
if err != nil {
return err
continue
}
}
The filepath can be seen when doing fmt.Println(applicableReductions) but the content of the files contents are not seen in the cat output file.
I thought perhaps a delay in the file being written so i tried adding a time.wait, tis didnt help. However the solution I found was to sort the slice, e.g this code above the call to exec cat solves the problem but I dont know why:
sort.Strings(applicableReductions)
I have confirmed all files present on both successful and unsucessful runs the only difference is without the sort, the content of the final appended file is missing
An explanation from a go-pro out there would be very much appreciated, let me know if you need more info, debug - happy to oblige to understand
UPDATE
It has been suggested that this is the same issue as here: Golang append an item to a slice, I think I understand the issue there and I'm not saying this isnt the same but I cannot see the same thing happenning - the slice in question is not touched from outside the main function (e.g. no editing of the slice in RetrieveDomainsFromDB function), I create the slice before a loop, append to it within a loop and then use it after the loop - Ive added an example at the top to show how the slice is built - please could someone clarify where this slice is being copied if this is the case
UPDATE AND CLOSE
Please close question - the issue was unrelated to the use of a string slice. Turns out that I was reading from the final output file before bufio-writer had been flushed (at end of function before defer flush kicked in on function return)
I think the sorting was just re-arranging the problem so I didnt notice it persisted or possibly giving some time for the buffer to flush. Either way sorted now with a manual call to flush.
Thanks for all help provided
I've modified the official documentation example for the zlib package to use an opened file rather than a set of hardcoded bytes (code below).
The code reads in the contents of a source text file and compresses it with the zlib package. I then try to read back the compressed file and print its decompressed contents into stdout.
The code doesn't error, but it also doesn't do what I expect it to do; which is to display the decompressed file contents into stdout.
Also: is there another way of displaying this information, rather than using io.Copy?
package main
import (
"compress/zlib"
"io"
"log"
"os"
)
func main() {
var err error
// This defends against an error preventing `defer` from being called
// As log.Fatal otherwise calls `os.Exit`
defer func() {
if err != nil {
log.Fatalln("\nDeferred log: \n", err)
}
}()
src, err := os.Open("source.txt")
if err != nil {
return
}
defer src.Close()
dest, err := os.Create("new.txt")
if err != nil {
return
}
defer dest.Close()
zdest := zlib.NewWriter(dest)
defer zdest.Close()
if _, err := io.Copy(zdest, src); err != nil {
return
}
n, err := os.Open("new.txt")
if err != nil {
return
}
r, err := zlib.NewReader(n)
if err != nil {
return
}
defer r.Close()
io.Copy(os.Stdout, r)
err = os.Remove("new.txt")
if err != nil {
return
}
}
Your defer func doesn't do anything, because you're shadowing the err variable on every new assignment. If you want a defer to run, return from a separate function, and call log.Fatal after the return statement.
As for why you're not seeing any output, it's because you're deferring all the Close calls. The zlib.Writer isn't flushed until after the function exits, and neither is the destination file. Call Close() explicitly where you need it.
zdest := zlib.NewWriter(dest)
if _, err := io.Copy(zdest, src); err != nil {
log.Fatal(err)
}
zdest.Close()
dest.Close()
I think you messed up the code logic with all this defer stuff and your "trick" err checking.
Files are definitively written when flushed or closed. You just copy into new.txt without closing it before opening it to read it.
Defering the closing of the file is neat inside a function which has multiple exits: It makes sure the file is closed once the function is left. But your main requires the new.txt to be closed after the copy, before re-opening it. So don't defer the close here.
BTW: Your defense against log.Fatal terminating the code without calling your defers is, well, at least strange. The files are all put into some proper state by the OS, there is absolutely no need to complicate the stuff like this.
Check the error from the second Copy:
2015/12/22 19:00:33
Deferred log:
unexpected EOF
exit status 1
The thing is, you need to close zdest immediately after you've done writing. Close it after the first Copy and it works.
I would have suggested to use io.MultiWriter.
In this way you read only once from src. Not much gain for small files but is faster for bigger files.
w := io.MultiWriter(dest, os.Stdout)
I have a text file where each line represents a JSON object. I am processing this file in Go with a simple for loop like this:
scanner := bufio.NewScanner(file)
for scanner.Scan() {
jsonBytes = scanner.Bytes()
var jsonObject interface{}
err := json.Unmarshal(jsonBytes, &jsonObject)
// do stuff with "jsonObject"...
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
When this code reaches a line with a particularly large JSON string (~67kb), I get the error message, "bufio.Scanner: token too long".
Is there an easy way to increase the max line size readable by NewScanner? Or is there another approach you can take altogether, when needing to read lines that are too large for NewScanner but are known to not be of unsafe size generally?
You can also do:
scanner := bufio.NewScanner(file)
buf := make([]byte, 0, 64*1024)
scanner.Buffer(buf, 1024*1024)
for scanner.Scan() {
// do your stuff
}
The second argument to scanner.Buffer() sets the maximum token size. In the above example you will be able to scan the file as long as none of the lines is larger than 1MB.
From the package docs:
Programs that need more control over error handling or large tokens,
or must run sequential scans on a reader, should use bufio.Reader
instead.
It looks like the preferred solution is bufio.Reader.ReadLine.
You surely don't want to be reading line-by-line in the first place. Why don't you just do this:
d := json.NewDecoder(file)
for {
var ob whateverType
err := d.Decode(&ob)
if err == io.EOF {
break
}
if err != nil {
log.Fatalf("Error decoding: %v", err)
}
// do stuff with "jsonObject"...
}
I am trying to take input from the keyboard and then store it in a text file but I am a bit confused on how to actually do it.
My current code is as follow at the moment:
// reads the file txt.txt
bs, err := ioutil.ReadFile("text.txt")
if err != nil {
panic(err)
}
// Prints out content
textInFile := string(bs)
fmt.Println(textInFile)
// Standard input from keyboard
var userInput string
fmt.Scanln(&userInput)
//Now I want to write input back to file text.txt
//func WriteFile(filename string, data []byte, perm os.FileMode) error
inputData := make([]byte, len(userInput))
err := ioutil.WriteFile("text.txt", inputData, )
There are so many functions in the "os" and "io" packages. I am very confused about which one I actually should use for this purpose.
I am also confused about what the third argument in the WriteFile function should be. In the documentation is says of type " perm os.FileMode" but since I am new to programming and Go I am a bit clueless.
Does anybody have any tips on how to proced?
Thanks in advance,
Marie
// reads the file txt.txt
bs, err := ioutil.ReadFile("text.txt")
if err != nil { //may want logic to create the file if it doesn't exist
panic(err)
}
var userInput []string
var err error = nil
var n int
//read in multiple lines from user input
//until user enters the EOF char
for ln := ""; err == nil; n, err = fmt.Scanln(ln) {
if n > 0 { //we actually read something into the string
userInput = append(userInput, ln)
} //if we didn't read anything, err is probably set
}
//open the file to append to it
//0666 corresponds to unix perms rw-rw-rw-,
//which means anyone can read or write it
out, err := os.OpenFile("text.txt", os.O_APPEND, 0666)
defer out.Close() //we'll close this file as we leave scope, no matter what
if err != nil { //assuming the file didn't somehow break
//write each of the user input lines followed by a newline
for _, outLn := range userInput {
io.WriteString(out, outLn+"\n")
}
}
I've made sure this compiles and runs on play.golang.org, but I'm not at my dev machine, so I can't verify that it's interacting with Stdin and the file entirely correctly. This should get you started though.
For example,
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
fname := "text.txt"
// print text file
textin, err := ioutil.ReadFile(fname)
if err == nil {
fmt.Println(string(textin))
}
// append text to file
f, err := os.OpenFile(fname, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0666)
if err != nil {
panic(err)
}
var textout string
fmt.Scanln(&textout)
_, err = f.Write([]byte(textout))
if err != nil {
panic(err)
}
f.Close()
// print text file
textin, err = ioutil.ReadFile(fname)
if err != nil {
panic(err)
}
fmt.Println(string(textin))
}
If you simply want to append the user's input to a text file, you could just read the
input as you've already done and use ioutil.WriteFile, as you've tried to do.
So you already got the right idea.
To make your way go, the simplified solution would be this:
// Read old text
current, err := ioutil.ReadFile("text.txt")
// Standard input from keyboard
var userInput string
fmt.Scanln(&userInput)
// Append the new input to the old using builtin `append`
newContent := append(current, []byte(userInput)...)
// Now write the input back to file text.txt
err = ioutil.WriteFile("text.txt", newContent, 0666)
The last parameter of WriteFile is a flag which specifies the various options for
files. The higher bits are options like file type (os.ModeDir, for example) and the lower
bits represent the permissions in form of UNIX permissions (0666, in octal format, stands for user rw, group rw, others rw). See the documentation for more details.
Now that your code works, we can improve it. For example by keeping the file open
instead of opening it twice:
// Open the file for read and write (O_RDRW), append to it if it has
// content, create it if it does not exit, use 0666 for permissions
// on creation.
file, err := os.OpenFile("text.txt", os.O_RDWR|os.O_APPEND|os.O_CREATE, 0666)
// Close the file when the surrounding function exists
defer file.Close()
// Read old content
current, err := ioutil.ReadAll(file)
// Do something with that old content, for example, print it
fmt.Println(string(current))
// Standard input from keyboard
var userInput string
fmt.Scanln(&userInput)
// Now write the input back to file text.txt
_, err = file.WriteString(userInput)
The magic here is, that you use the flag os.O_APPEND while opening the file,
which makes file.WriteString() append. Note that you need to close the file after
opening it, which we do after the function exists using the defer keyword.