When I open file in mode like this:
file, _ := os.OpenFile("/path/to/my/file", os.O_RDWR|os.O_APPEND, os.FileMode(0666))
file.Seek(start, os.SEEK_SET)
io.CopyN(file, resp.Body, length)
io.CopyN does not respect the position where I sought. It seems it just append to the tail of the file. Instead if I open the file like this:
file, _ := os.OpenFile("/path/to/my/file", os.O_RDWR, os.FileMode(0666))
file.Seek(start, os.SEEK_SET)
io.CopyN(file, resp.Body, length)
It works as I expected. io.CopyN writes to the file from the "start" point I sought. Not sure if this is a feature or a bug?
It's definitely a feature (http://man7.org/linux/man-pages/man2/open.2.html) and it's controlled by underlying OS, not golang runtime.
O_APPEND
The file is opened in append mode. Before each write(2), the
file offset is positioned at the end of the file, as if with
lseek(2). O_APPEND may lead to corrupted files on NFS
filesystems if more than one process appends data to a file at
once. This is because NFS does not support appending to a
file, so the client kernel has to simulate it, which can't be
done without a race condition.
Related
I have a situation where I'd like to return a file handle from a routine. When the last byte of that file handle is read, the file being read is deleted. This will be a simple way to create a temporary file that is deleted once it's been read.
Tempfile doesn't really work for this purpose. To use Tempfile I would have to create the Tempfile, read and write in the entire contents of the source file to the Tempfile, rewind the Tempfile, then return the Tempfile object. Very wasteful. I just want to open a file and return a file handle, but one which deletes the file when the last byte is read.
I could probably write this class myself, but I'd rather use an existing solution if it exists.
Does it have to return a file handle type or can it be something that works on a file handle? If its the latter then you could use a closure and a lambda. Something like below:
def sp_handle(fname)
handle = File.open(fname)
->(){
line = handle.gets
line ? line : (File.unlink(fname); handle.close)
}
end
Using Go - lang, according to the documentation, the os.Rename should be able to rename either a file or directory on any operating system.
On Linux it works as it should, pass either a file or directory into it and the file or directory are moved.
On windows i recieve an 'Access is denied' Error when trying to pass a folder.
It works 100% for files.
example:
source = c:\sourcefolder
destination = c:\destinationfolder
source contains:
C:\sourcefolder\file1.xml
C:\sourcefolder\file2.xml
C:\sourcefolder\foldername1
C:\sourcefolder\foldername1\file3.xml
C:\sourcefolder\foldername2
C:\sourcefolder\foldername2\file4.xml
both file1.xml and file2.xml will successfully copy to c:\destination.
But the folders and files within the folders crash out with access denied
The script is pretty simple:
source := "C:\\sourcefolder"
destination := "C:\\destinationfolder"
pathSeperator := "\\"
files, err := ioutil.ReadDir(source)
if err != nil {
fmt.Println("Move command execution error: ", err)
}
for _, f := range files {
fmt.Println(f.Name())
fmt.Println(f.Mode())
err := os.Rename(source+pathSeperator+f.Name(), destination+pathSeperator+f.Name())
if err != nil {
fmt.Println("Move command execution error: ", err)
panic(err)
}
}
Having searched stackoverflow and golang's resources, i found the issue listed in 2016 that reported this fault and according to the issue it was fixed, but i am unable to get this to work. Nowhere else that i can find lists this issue go golang.
checking the f.Mode for access, i get drwxrwxrwx and have complete access to all the files and directories.
Any help with this would be great, racking my mind. Thank you.
Quoted from comment. solved my issue.
Found the cause of the fault to be, if a windows explorer window is
open and has ANY visibility of the folders being moved (i.e. in the
tree on the left or right-pane) then access is denied as it can not
move the folders. If i minimize all the tree's so that the
source\destination folders are not visible and select a different sub
folder in windows explorer then the os.Rename works as it should,
moving all content from A to B really quick (as per linux)
I had the same issue with copying files within the same folder. The following solution works just fine (without closing or minimizing windows):
// read original file
origFile, _:= os.ReadFile(filePath)
// create new file with a different name
newFile, _ := os.Create(filePath + ".new")
// print data from original file to new file.
fmt.Fprintf(newFile, "%s", string(origFile))
We can specify both of flag and perm at os.OpenFile.
They have really similar options, O_APPEND and ModeAppend. What's the difference between them?
f, _ := os.OpenFile("access.log", os.O_APPEND|os.O_CREATE, os.ModeAppend|0644)
The flag specify the flags used on the system call to open the file while perm sets the File mode on the file. The file mode includes the permissions and type of file eg. symlink, directory, etc...
os.O_APPEND tells the underlying OS that all the write calls you do on that file handler should always append to the file so you don't need to set the offset to write on the correct part of the file.
ModeAppend sets the file mode to be append. This means that the this file can only be modified by appending to it, not by rewriting the file contents. The specifics of this depends on the OS and file system you are using. I believe Plan 9, implements it by ignoring the offset on any write call to the file and always appending to it, while in linux it means that the file can only be open for writing in append mode. I think that on most linux distros you need to be root to set the file mode to append.
In 99.99% of cases you just want to use perm to set the file permissions rwx. In your case if you want to open a file and append to it you should use:
// os.O_WRONLY tells the computer you are only going to writo to the file, not read
// os.O_CREATE tells the computer to create the file if it doesn't exist
// os.O_APPEND tells the computer to append to the end of the file instead of overwritting or truncating it
f, err := os.OpenFile("access.log", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
You might have only ignore the return error on os.OpenFile to put the example online, but you should get used to always checking for errors. You have no idea how many users ran into trouble when starting with go because they ignore the errors. Sometimes is something stupid and easy to fix like a typo, but if you ignore the error you don't know what the issue is.
You can read more about the append file mode here.
Let's say I do an os.Stat() on a directory:
func main() {
fi, _ := os.Stat("/tmp")
println(fi.Size())
}
// 548
// Program exited.
https://play.golang.org/p/NIzGMHRYfi
What exactly is the [FileInfo].Size() value meant to represent? It's not the file size, so I'm guessing something like number of files? Inodes? I couldn't find a clear answer anywhere, so maybe someone can enlighten me?
FileInfo mentions
// length in bytes for regular files; system-dependent for others
So it really depends on the execution environment.
See for instance "Where does ext4 store directory sizes?"
In that example, a directory stat size returns 4096 bytes.
That's the actual size of the directory itself, not what it contains.
The stat command provides no facility for querying anything other then the size of a filesystem object (directory or file).
stat simply doesn't have a way to return multiple sizes—so it can only return the size of the directory itself, not of its contents.
And also "directory size including contents" becomes less clear when you have hardlinked files.
I am using twitter ruby gem to fetch twitter search result. The example code from Github extracts the information from search result.I am wondering how to save the search result, which is JSON i think, to a separate JSON file.
Here is part of the example code:
results = #search.perform("$aaa", 1000)
aFile = File.new("data.txt", "w")
results.map do |status|
myStr="#{status.from_user}: #{status.text} #{status.created_at}"
aFile.write(myStr)
aFile.write("\n")
end
Is there any way to save all the search result to a separate JSON file instead of writing strings to a file?
Thanks in advance.
If you want to save to a file all you need to do is open the file, write it it, then close it:
File.open("myFileName.txt", "a") do |mFile|
mFile.syswrite("Your content here")
mFile.close
end
When you use open you will create the file if it doesn't exist.
One thing to be aware of is that there are different ways to open file, of which will determine where the program writes to. The "a" indicates that it will append everything you write to the file, to the end of the current content.
Here is some of the options:
r Read-only mode. The file pointer is placed at the beginning of the file. This is the default mode.
r+ Read-write mode. The file pointer will be at the beginning of the file.
w Write-only mode. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
w+ Read-write mode. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
a Write-only mode. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
a+ Read and write mode. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing.
So in your case, you would want to pull out the data you want to save, then write it to a file as I have shown. You can also specify file paths by doing:
File.open("/the/path/to/yourfile/myFileName.txt", "a") do |mFile|
mFile.syswrite("Your content here")
mFile.close
end
Another thing to be aware of is that open does not create directories, so you will either need to create directories yourself, or you can do it with your program. Here is a link that is helpful for file input/output:
http://www.tutorialspoint.com/ruby/ruby_input_output.htm