Is there a package that exports the syncfs function in Go? I'd like to synchronize a particular file system.
I found the syscall package, but it only exports FSync, Fdatasync and Sync.
syncfs is just a system call, which are easy to trigger in Go.
However, since the syscall package does not have the syncfs syscall constant, you can use golang.org/x/sys/unix instead, which has it defined (not really necessity, since a syscall constant is just a number, but using that package doesn't hurt).
import "golang.org/x/sys/unix"
func syncfs(fd int) error {
_, _, err := unix.Syscall(unix.SYS_SYNCFS, uintptr(fd), 0, 0)
if err != 0 {
return err
}
return nil
}
For completeness, here is the solution just using the syscall package:
import "syscall"
func syncfs(fd int) error {
_, _, err := syscall.Syscall(306, uintptr(fd), 0, 0)
if err != 0 {
return err
}
return nil
}
Related
I want to generate 64-bit random integer using the secure crypto/rand package. I found this online:
package main
import (
"crypto/rand"
"encoding/base64"
)
// GenerateRandomBytes returns securely generated random bytes.
// It will return an error if the system's secure random
// number generator fails to function correctly, in which
// case the caller should not continue.
func GenerateRandomBytes(n int) ([]byte, error) {
b := make([]byte, n)
_, err := rand.Read(b)
// Note that err == nil only if we read len(b) bytes.
if err != nil {
return nil, err
}
return b, nil
}
But it seems to generate random bytes instead. I want a random 64-bit int. Namely, I want something like var i uint64 = rand(). Any ideas how to achieve this?
You can generate a random number with crypto.Rand, and then convert those bytes to an int64 using the binary package:
func randint64() (int64, error) {
var b [8]byte
if _, err := rand.Read(b[:]); err != nil {
return 0, err
}
return int64(binary.LittleEndian.Uint64(b[:])), nil
}
https://play.golang.org/p/2Q8tvttqbJ (result is cached)
If you look at the source code for LittleEndian.Uint64, you can see it's simply performing a few bit operations on the data; something that you could implemented for yourself.
You also use rand.Int in crypto/rand package
func randint64() (int64, error) {
val, err := rand.Int(rand.Reader, big.NewInt(int64(math.MaxInt64)))
if err != nil {
return 0, err
}
return val.Int64(), nil
}
https://play.golang.org/p/fqoQxpmjOSu
I'm struggling to handle nested zip files in Go (where a zip file contains another zip file). I'm trying to recurse a zip file and list all of the files it contains.
archive/zip gives you two methods for handling a zip file:
zip.NewReader
zip.OpenReader
OpenReader opens a file on disk. NewReader accepts an io.ReaderAt and a file size. As you iterate through the zipped files with either of these, you get out a zip.File for each file inside the zip. To get the file contents of file f, you call f.Open which gives you a zip.ReadCloser. To open a nested zip file, I'd need to use NewReader, but zip.File and zip.ReadCloser do not satisfy the io.ReaderAt interface.
zip.File has a private field zipr which is an io.ReaderAt and zip.ReadCloser has a private field f which is an os.File which should satisfy the requirements for NewReader.
My question: is there any way to open a nested zip file without first writing the contents to a file on disk, or reading the whole thing into memory.
It looks like everything that is needed is available in zip.File, but isn't exported. I'm hoping I missed something.
How about an io.ReaderAt from an io.Reader that reinitializes if you decided to go backwards: (this code is largely untested, but hopefully you get the idea)
package main
import (
"io"
"io/ioutil"
"os"
"strings"
)
type inefficientReaderAt struct {
rdr io.ReadCloser
cur int64
initer func() (io.ReadCloser, error)
}
func newInefficentReaderAt(initer func() (io.ReadCloser, error)) *inefficientReaderAt {
return &inefficientReaderAt{
initer: initer,
}
}
func (r *inefficientReaderAt) Read(p []byte) (n int, err error) {
n, err = r.rdr.Read(p)
r.cur += int64(n)
return n, err
}
func (r *inefficientReaderAt) ReadAt(p []byte, off int64) (n int, err error) {
// reset on rewind
if off < r.cur || r.rdr == nil {
r.cur = 0
r.rdr, err = r.initer()
if err != nil {
return 0, err
}
}
if off > r.cur {
sz, err := io.CopyN(ioutil.Discard, r.rdr, off-r.cur)
n = int(sz)
if err != nil {
return n, err
}
}
return r.Read(p)
}
func main() {
r := newInefficentReaderAt(func() (io.ReadCloser, error) {
return ioutil.NopCloser(strings.NewReader("ABCDEFG")), nil
})
io.Copy(os.Stdout, io.NewSectionReader(r, 0, 3))
io.Copy(os.Stdout, io.NewSectionReader(r, 1, 3))
}
If you mostly move forwards this probably works ok. Especially if you use a buffered reader.
I should note that this violates the io.ReaderAt guarantees: https://godoc.org/io#ReaderFrom , namely it doesn't allow parallel calls to ReadAt, and doesn't block on full reads, so this may not even work properly
I ran into the exact same need and came up with the following approach, not sure if its any help to you:
// NewZipFromReader ...
func NewZipFromReader(file io.ReadCloser, size int64) (*zip.Reader, error) {
in := file.(io.Reader)
if _, ok := in.(io.ReaderAt); ok != true {
buffer, err := ioutil.ReadAll(in)
if err != nil {
return nil, err
}
in = bytes.NewReader(buffer)
size = int64(len(buffer))
}
reader, err := zip.NewReader(in.(io.ReaderAt), size)
if err != nil {
return nil, err
}
return reader, nil
}
So if file doesn't implement io.ReaderAt it reads the whole contents into a buffer.
It's probably not safe to handle ZIP bombs, and will defenitely fail with OOM for files larger than RAM.
With Windows, the official way of guessing if the current 32-bit process is running on a 32 or 64-bit architecture (so on WOW64 or not) is to call the IsWow64Process function from kernel32.dll, and see if it is present (as I understand the doc).
In Go we can call functions exported in dll files with the syscall package, so here is my attempt:
package main
import (
"fmt"
"os"
"syscall"
)
func main() {
dll, err := syscall.LoadDLL("kernel32.dll")
if err != nil {
fmt.Println(err)
}
defer dll.Release()
proc, err := dll.FindProc("IsWow64Process")
if err != nil {
fmt.Println("Proc not found") // not a WOW64 so a 32 bit system?
fmt.Println(err)
}
fmt.Printf("%v\n", proc)
var handle uintptr = uintptr(os.Getpid())
var result uintptr
v, x, y := proc.Call(handle, result)
fmt.Printf("%v %v %v\n", v, x, y)
fmt.Printf("%v\n", result)
}
Unfortunately, testing with or without a WOW64 system displays the same in stdout:
&{0x10ada110 IsWow64Process 2088961457}
0 7 The handle is invalid.
0
What do I do wrong? How to achieve a test to determine if our 32-bit Go program runs on an emulated 32-bit on a 64-bit CPU (WOW64) or on a real 32-bit Windows?
I believe the issue is the handle parameter on your proc.Call. The expected parameter for IsWow64Process is a HANDLE which is not the same as a pid. Which is why it is indicating that the handle is invalid.
The following SO question How to get process handle from process id indicates that you need to call OpenProcess passsing in the pid and it returns the handle.
EDIT: GetCurrentProcess is defined in syscall. So I think you can replace the Getpid call with the following:
handle, err := syscall.GetCurrentProcess()
OK, so here is a working code:
package main
import (
"syscall"
"fmt"
"unsafe"
)
func main() {
dll, err := syscall.LoadDLL("kernel32.dll")
if err != nil {
fmt.Println("Can't load kernel32")
fmt.Println(err)
}
defer dll.Release()
proc, err := dll.FindProc("IsWow64Process")
if err != nil {
fmt.Println("Proc not found")
fmt.Println(err)
}
fmt.Printf("%v\n",proc)
handle, err := syscall.GetCurrentProcess()
if err != nil {
fmt.Println("Handle not found")
fmt.Println(err)
}
fmt.Printf("%v\n",handle)
var result bool
v, x, y := proc.Call(uintptr(handle), uintptr(unsafe.Pointer(&result)))
fmt.Printf("%v %v %v\n",v,x,y)
fmt.Printf("%v\n",result)
}
The result var will be true for a WOW64 system and false for a 32 bit system.
You can also use golang.org/x/sys/windows
package main
import (
"fmt"
"golang.org/x/sys/windows"
)
func main() {
handle := windows.CurrentProcess()
var isWow64 bool
err := windows.IsWow64Process(handle, &isWow64)
if err != nil {
panic(err)
}
fmt.Println(isWow64)
}
How might I get the count of items returned by io/ioutil.ReadDir()?
I have this code, which works, but I have to think isn't the RightWay(tm) in Go.
package main
import "io/ioutil"
import "fmt"
func main() {
files,_ := ioutil.ReadDir("/Users/dgolliher/Dropbox/INBOX")
var count int
for _, f := range files {
fmt.Println(f.Name())
count++
}
fmt.Println(count)
}
Lines 8-12 seem like way too much to go through to just count the results of ReadDir, but I can't find the correct syntax to get the count without iterating over the range. Help?
Found the answer in http://blog.golang.org/go-slices-usage-and-internals
package main
import "io/ioutil"
import "fmt"
func main() {
files,_ := ioutil.ReadDir("/Users/dgolliher/Dropbox/INBOX")
fmt.Println(len(files))
}
ReadDir returns a list of directory entries sorted by filename, so it is not just files. Here is a little function for those wanting to get a count of files only (and not dirs):
func fileCount(path string) (int, error){
i := 0
files, err := ioutil.ReadDir(path)
if err != nil {
return 0, err
}
for _, file := range files {
if !file.IsDir() {
i++
}
}
return i, nil
}
Starting with Go 1.16 (Feb 2021), a better option is os.ReadDir:
package main
import "os"
func main() {
d, e := os.ReadDir(".")
if e != nil {
panic(e)
}
println(len(d))
}
os.ReadDir returns fs.DirEntry instead of fs.FileInfo, which means that
Size and ModTime methods are omitted, making the process more efficient if
you just need an entry count.
https://golang.org/pkg/os#ReadDir
If you wanna get all files (not recursive) you can use len(files). If you need to just get the files without folders and hidden files just loop over them and increase a counter. And please don’t ignore errors
By looking at the code of ioutil.ReadDir
func ReadDir(dirname string) ([]fs.FileInfo, error) {
f, err := os.Open(dirname)
if err != nil {
return nil, err
}
list, err := f.Readdir(-1)
f.Close()
if err != nil {
return nil, err
}
sort.Slice(list, func(i, j int) bool { return list[i].Name() < list[j].Name() })
return list, nil
}
you would realize that it calls os.File.Readdir() then sorts the files.
In case of counting it, you don't need to sort, so you are better off calling os.File.Readdir() directly.
You can simply copy and paste this function then remove the sort.
But I did find out that f.Readdirnames(-1) is much faster than f.Readdir(-1).
Running time is almost half for /usr/bin/ with 2808 items (16ms vs 35ms).
So to summerize it in an example:
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.Open(os.Args[1])
if err != nil {
panic(err)
}
list, err := f.Readdirnames(-1)
f.Close()
if err != nil {
panic(err)
}
fmt.Println(len(list))
}
I need to read a file of integers into an array. I have it working with this:
package main
import (
"fmt"
"io"
"os"
)
func readFile(filePath string) (numbers []int) {
fd, err := os.Open(filePath)
if err != nil {
panic(fmt.Sprintf("open %s: %v", filePath, err))
}
var line int
for {
_, err := fmt.Fscanf(fd, "%d\n", &line)
if err != nil {
fmt.Println(err)
if err == io.EOF {
return
}
panic(fmt.Sprintf("Scan Failed %s: %v", filePath, err))
}
numbers = append(numbers, line)
}
return
}
func main() {
numbers := readFile("numbers.txt")
fmt.Println(len(numbers))
}
The file numbers.txt is just:
1
2
3
...
ReadFile() seems too long (maybe because of the error handing).
Is there a shorter / more Go idiomatic way to load a file?
Using a bufio.Scanner makes things nice. I've also used an io.Reader rather than taking a filename. Often that's a good technique, since it allows the code to be used on any file-like object and not just a file on disk. Here it's "reading" from a string.
package main
import (
"bufio"
"fmt"
"io"
"strconv"
"strings"
)
// ReadInts reads whitespace-separated ints from r. If there's an error, it
// returns the ints successfully read so far as well as the error value.
func ReadInts(r io.Reader) ([]int, error) {
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
var result []int
for scanner.Scan() {
x, err := strconv.Atoi(scanner.Text())
if err != nil {
return result, err
}
result = append(result, x)
}
return result, scanner.Err()
}
func main() {
tf := "1\n2\n3\n4\n5\n6"
ints, err := ReadInts(strings.NewReader(tf))
fmt.Println(ints, err)
}
I would do it like this:
package main
import (
"fmt"
"io/ioutil"
"strconv"
"strings"
)
// It would be better for such a function to return error, instead of handling
// it on their own.
func readFile(fname string) (nums []int, err error) {
b, err := ioutil.ReadFile(fname)
if err != nil { return nil, err }
lines := strings.Split(string(b), "\n")
// Assign cap to avoid resize on every append.
nums = make([]int, 0, len(lines))
for _, l := range lines {
// Empty line occurs at the end of the file when we use Split.
if len(l) == 0 { continue }
// Atoi better suits the job when we know exactly what we're dealing
// with. Scanf is the more general option.
n, err := strconv.Atoi(l)
if err != nil { return nil, err }
nums = append(nums, n)
}
return nums, nil
}
func main() {
nums, err := readFile("numbers.txt")
if err != nil { panic(err) }
fmt.Println(len(nums))
}
Your solution with fmt.Fscanf is fine. There are certainly a number of other ways to do though, depending on your situation. Mostafa's technique is one I use a lot (although I might allocate the result all at once with make. oops! scratch that. He did.) but for ultimate control you should learn bufio.ReadLine. See go readline -> string for some example code.