Using golang gin, I read file data using:
file, fileHeader, err:=ctx.Request.FormFile("blabla...")
Do I need to write this?
defer file.Close()
I jump to the source code, it says:
// Open opens and returns the FileHeader's associated File.
func (fh *FileHeader) Open() (File, error) {
if b := fh.content; b != nil {
r := io.NewSectionReader(bytes.NewReader(b), 0, int64(len(b)))
fmt.Printf("TODDLINE:152\n")
fmt.Printf("TODDLINE:154:fmpfile:%#v\n", fh.tmpfile)
fmt.Printf("TODDLINE:154:Filename:%#v\n", fh.Filename)
return sectionReadCloser{r}, nil
}
fmt.Printf("TODDLINE:155\n")
return os.Open(fh.tmpfile)
}
If it uses os.Open, I guess I must close the file, but if it retuns the sectionReadCloser{r}, the Close function shows like this:
func (rc sectionReadCloser) Close() error {
return nil
}
The close function of seciontReadCloser doesn't do anything.
And I find that it does return the sectionReadCloser{r}.
I guess I should close the file, but I still want to know when it will return the os.Open. I will keep going to read the source code and try to understand it. It would be nice if someone gives me some advice.
If the file returned implements io.Closer (i.e., if it has a Close method), assume you are responsible for closing it unless the documentation explicitly states otherwise.
Related
Let's assume for shake of simplicity I have a simple function that just tries to open a file based on a string passed as parameter (do not rely on the simplicity of the file opening process, given that the actual example will be more convoluted)
func retrieveConfig(fs billy.Filesystem, app string) (Spec, error) {
var v Spec
filename := "releases/" + app + ".yaml"
file, err := fs.Open(filename)
if err != nil {
errOpenFile := ErrOpeningConfigFile{filename}
return Spec{}, new errOpenFile{filename}
}
Here is the definition of the custom error I am using
type ErrOpeningConfigFile struct {
s string
}
func (errConfigFileNotFound *ErrOpeningConfigFile) Error() string {
return fmt.Sprintf("Error opening configuration file %s")
}
I thought of following this approach because my idea is that it would give more coverage and make the core more testable.
I can test the error paths more easily now given that I can type assert on the custom error in my unit tests.
However this definitely increases the codebase.
I assume the other way around is to just
return nil, errors.New("Configuration file not found")
that would limit the code, but make testing clumsier (or more narrowed scoped).
What is the appropriate (or Go idiomatic way) of going about this?
I'm coming at Golang from a Python background and I am trying to wrap my head around various new concepts.
One thing I have come across is this function in net.go:
func (c *conn) ok() bool { return c != nil && c.fd != nil }
This function is called by multiple net.go methods, e.g. conn.Read:
// Read implements the Conn Read method.
func (c *conn) Read(b []byte) (int, error) {
if !c.ok() {
return 0, syscall.EINVAL
}
I am trying to understand how the ok() method can be called on conn, despite the fact that ok() does not appear to be an interface of conn.
Certainly I do not seem to be able to call ok() from my client code:
func main() {
conn, err := net.Dial("tcp", "www.reddit.com:80")
if err != nil {
os.Exit(-1)
}
fmt.Println(&conn.ok())
}
Output:
./server.go:14:22: conn.ok undefined (type net.Conn has no field or method ok)
Any pointers appreciated...
From Go document :
An identifier may be exported to permit access to it from another
package. An identifier is exported if the first character of the
identifier's name is a Unicode upper case letter
So , ok function is not exported and you can't access it outside of net package.
Go does not use public/private keywords for visibility of an identifier. If the initial character is an upper case letter, the identifier is exported(public); otherwise it is not:
upper case initial letter: Name is visible to clients of package
otherwise: name (or _Name) is not visible to clients of package
There is no field or method like ok in net.Conn that what the error says and that is correct.
when you try to read and write into the conn , you would get err and number of bytes read or write it into the connection.
I am using logrus for logging and have a few custom format loggers. Each is initialized to write to a different file like:
fp, _ := os.OpenFile(path, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0755)
// error handling left out for brevity
log.Out = fp
Later in the application, I need to change the file the logger is writing to (for a log rotation logic). What I want to achieve is to properly close the current file before changing the logger's output file. But the closest thing to the file handle logrus provides me is a Writer() method that returns a io.PipeWriter pointer. So would calling Close() on the PipeWriter also close the underlying file?
If not, what are my options to do this, other than keeping the file pointer stored somewhere.
For the record, twelve-factor tells us that applications should not concern themselves with log rotation. If and how logs are handled best depends on how the application is deployed. Systemd has its own logging system, for instance. Writing to files when deployed in (Docker) containers is annoying. Rotating files are annoying during development.
Now, pipes don't have an "underlying file". There's a Reader end and a Writer end, and that's it. From the docs for PipeWriter:
Close closes the writer; subsequent reads from the read half of the pipe will return no bytes and EOF.
So what happens when you close the writer depends on how Logrus handles EOF on the Reader end. Since Logger.Out is an io.Writer, Logrus cannot possibly call Close on your file.
Your best bet would be to wrap *os.File, perhaps like so:
package main
import "os"
type RotatingFile struct {
*os.File
rotate chan struct{}
}
func NewRotatingFile(f *os.File) RotatingFile {
return RotatingFile{
File: f,
rotate: make(chan struct{}, 1),
}
}
func (r RotatingFile) Rotate() {
r.rotate <- struct{}{}
}
func (r RotatingFile) doRotate() error {
// file rotation logic here
return nil
}
func (r RotatingFile) Write(b []byte) (int, error) {
select {
case <-r.rotate:
if err := r.doRotate(); err != nil {
return 0, err
}
default:
}
return r.File.Write(b)
}
Implementing log file rotation in a robust way is surprisingly tricky. For instance, closing the old file before creating the new one is not a good idea. What if the log directory permissions changed? What if you run out of inodes? If you can't create a new log file you may want to keep writing to the current file. Are you okay with ripping lines apart, or do you only want to rotate after a newline? Do you want to rotate empty files? How do you reliably remove old logs if someone deletes the N-1th file? Will you notice the Nth file or stop looking at the N-2nd?
The best advice I can give you is to leave log rotation to the pros. I like svlogd (part of runit) as a standalone log rotation tool.
The closing of io.PipeWriter will not affect actual Writer behind it. The chain of close execution:
PipeWriter.Close() -> PipeWriter.CloseWithError(err error) ->
pipe.CloseWrite(err error)
and it doesn't influence underlying io.Writer.
To close actual writer you need just to close Logger.Out that is an exported field.
When I do some file operations with golang, I firstly open a file and add the close() into defer list, then I try to rename that file. If I close the file manually, the defer will close it again. If I wait for the defer to close it, the rename will cause error because it is not closed yey. Code as below
func main() {
pfile1, _ := os.Open("myfile.log")
defer pfile1.Close() //It will be closed again.
...
...
pfile1.Close() //I have to close it before rename it.
os.Rename("myfile.log", "myfile1.log")
}
I found some ugly solution, such as create another function to separate the open file, does any better solution that below?
func main() {
var pfile1 *os.File
ugly_solution(pfile1)
os.Rename("myfile.log", "myfile1.log")
}
func ugly_solution(file *os.File) {
file, _ = os.Open("myfile.log")
defer file.Close()
}
There are a few things that are not clear to me about your code.
First of all, why do you open the file before renaming it? This is not required by the os.Rename function. The function takes two strings representing the old and new file name, there is no need to pass a file pointer.
func main() {
...
...
os.Rename("myfile.log", "myfile1.log")
}
Assuming you need to make changes to the file content (which doesn't seem to be the case given the ugly_solution method) and you have to open the file, then why deferring file.Close()? You don't have to defer the method if you need it to be called explicitly somewhere in the same method. Simply call it.
func main() {
pfile1, _ := os.Open("myfile.log")
...
...
pfile1.Close()
os.Rename("myfile.log", "myfile1.log")
}
You can put both closing and renaming the file in the defer:
func main() {
pfile1, _ := os.Open("myfile.log")
defer func(){
pfile1.Close()
os.Rename("myfile.log", "myfile1.log")
}()
...
...
}
In situation like in your sample
Maybe you want to follow this scenario:
Create easily an identifiable temporary file.
Write the data.
Close the file.
If successful rename the file.
In that case where you want to follow OS system action of underlying files maybe you want to simply not deferring the close on IO.file since you want to get the Error returned by the close function itself.
Also, in that case you maybe want to operate a file.sync() too.
See https://www.joeshaw.org/dont-defer-close-on-writable-files/
Does anyone have a good or any explanation of Golang's NopCloser function? I looked around but failed to find anything besides Golang's main doc's explanation of:
NopCloser returns a ReadCloser with a no-op Close method wrapping the
provided Reader r.
Any pointers or explanation would be appreciated. Thanks.
Whenever you need to return an io.ReadCloser, while making sure a Close() is available, you can use a NopCloser to build such a ReaderCloser.
You can see one example in this fork of gorest, in util.go
//Marshals the data in interface i into a byte slice, using the Marhaller/Unmarshaller specified in mime.
//The Marhaller/Unmarshaller must have been registered before using gorest.RegisterMarshaller
func InterfaceToBytes(i interface{}, mime string) (io.ReadCloser, error) {
v := reflect.ValueOf(i)
if v.Kind() == reflect.Ptr {
v = v.Elem()
}
switch v.Kind() {
case reflect.Bool:
x := v.Bool()
if x {
return ioutil.NopCloser(bytes.NewBuffer([]byte("true"))), nil
}
It's used for functions that require io.ReadCloser but your current object (for example a bytes.Buffer) doesn't provide a Close function.
It is for when you need to supply an item that has a Close function, but when
Close doesn't really make sense for that item. As such, the function pretty much
does nothing:
func (nopCloser) Close() error { return nil }
https://github.com/golang/go/blob/go1.16.3/src/io/io.go#L620