Fcntl in go doesn't work - go

I'm currently writing a http server, that has to run over all files inside a folder. One Request can write a file, where another Request can read the files at the same time, so i need to lock the file, that is open for writing. In the Routine, that will read the file, i just want to skip the file that is locked.
To make this, i want to use locklibrarys like that from rubyist or nightlyone.
The problem is, i didnt get them to work, so i started to call the syscall.FcntlFlock() function myself and it didn't work like i had expected.
This program doesn't work in the Go Playground, cause the playground seems to run not on a unix-based system (syscall.FcntlFlock is undefined)
The code that didn't work:
func main() {
time.Sleep(time.Second)
file, err := os.Open("lockfiletest.lock")
if err != nil {
log.Printf("error opening file2: %s", err)
return
}
defer file.Close()
flockT := syscall.Flock_t{
Type: syscall.F_WRLCK,
Whence: io.SeekStart,
Start: 0,
Len: 0,
}
err = syscall.FcntlFlock(file.Fd(), syscall.F_SETLK, &flockT)
if err != nil {
log.Printf("error locking file2: %s", err)
return
}
log.Println("lock2 accessed")
time.Sleep(time.Second * 5)
log.Println("func2 finished")
time.Sleep(time.Second * 15)
}
the console output:
2017/10/28 00:18:12 error locking file2: bad file descriptor
Process finished with exit code 0
What am i doing wrong? Are the syscalls broken?
I also tried it in C and there it works fine.
I test this program in Go1.8.3 and Go1.9.1 on ubuntu16.04
PS: This program also have to run on windows, so only implementing a FcntlLock isn't enough.
I also thoght about using sync.RWMutex, so it is locked via a Mutex instead of a fileLock. That is not exactly what i want, cause i only want to skip the file that is locked and not wait until the mutex can be locked.

If I create the file with touch lockfiletest.lock, that is with no file contents, your program fails with your error: error locking file2: bad file descriptor.
$ rm -f lockfiletest.lock
$ touch lockfiletest.lock
$ go run lockfiletest.go
2017/10/27 21:17:27 error locking file2: bad file descriptor
I changed the file open to closely match TestFcntlFlock.
$ uname -s
Linux
~/go/src/syscall$ $ go test -v -run=TestFcntlFlock syscall_unix_test.go
=== RUN TestFcntlFlock
--- PASS: TestFcntlFlock (0.01s)
PASS
ok syscall 0.008s
~/go/src/syscall$
For example,
package main
import (
"io"
"log"
"os"
"syscall"
"time"
)
func main() {
time.Sleep(time.Second)
name := "lockfiletest.lock"
file, err := os.OpenFile(name, syscall.O_CREAT|syscall.O_RDWR|syscall.O_CLOEXEC, 0666)
if err != nil {
log.Printf("error opening file: %s", err)
return
}
defer file.Close()
flockT := syscall.Flock_t{
Type: syscall.F_WRLCK,
Whence: io.SeekStart,
Start: 0,
Len: 0,
}
err = syscall.FcntlFlock(file.Fd(), syscall.F_SETLK, &flockT)
if err != nil {
log.Printf("error locking file: %s", err)
return
}
log.Println("lock2 accessed")
time.Sleep(time.Second * 5)
log.Println("func2 finished")
time.Sleep(time.Second * 15)
}
Output:
$ rm -f lockfiletest.lock
$ touch lockfiletest.lock
$ go run lockfiletest.go
2017/10/27 21:21:56 lock2 accessed
2017/10/27 21:22:01 func2 finished
$ rm -f lockfiletest.lock
$ go run lockfiletest.go
2017/10/27 21:22:25 lock2 accessed
2017/10/27 21:22:30 func2 finished
$ go run lockfiletest.go
2017/10/27 21:25:40 lock2 accessed
2017/10/27 21:25:45 func2 finished
$

You have to open your file WRONLY or RDWR, you're opening it read only to put a lock on it which doesn't work - you're getting a read only file descriptor.

Related

Can't access filesystem in Go Lambda

I used Lambda functions before, and if I remember correctly I'm supposed to have ~500Mb of (ephemeral) space in /tmp.
Nevertheless, my Go lambda function doesn't seem to interact with the fs properly:
exec.Command("ls -la /").Output() returns empty
exec.Command("rm -rf /tmp/xxx").Run() returns fork/exec : no such file or directory
exec.Command("mkdir -p /tmp/xxx").Run() returns fork/exec : no such file or directory
It's really weird.
It's using the go1.x environment (thus, I guess amazonlinux:2)
UPDATE
I CAN access the fs using Go os functions:
os.RemoveAll("/tmp/xxx")
if _, err := os.Stat("/tmp/xxx"); os.IsNotExist(err) {
if err := os.Mkdir("/tmp/xxx", os.ModePerm); err != nil {
return err
}
}
BUT I really need exec to run afterwards (a binary command), and write a file in that tmp folder. The error in that case is the same (no such file or directory). Even though I've just created the folder with the above commands.
You are close. The way you use exec.Command() is not yet 100% correct. Try the following:
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
o, err := exec.Command("ls", "-la", "/tmp").Output()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Printf("%s\n", o)
}
The first argument to Command() is the program you want to run and all the following arguments are the programs arguments.
See https://play.golang.org/p/WaVOU0IESmZ

Go's monkey.PatchInstanceMethod returns a "permission denied" error?

I've tried to come up with a simple, minimal example which reproduces this bug, but wasn't able to (it only occurs in one private repo), but I'll start by showing my attempt. Suppose we have a Go module with the following structure:
.
├── command
│   ├── command.go
│   └── command_test.go
├── go.mod
└── go.sum
where go.mod reads
module github.com/kurtpeek/monkeypatching
go 1.12
require (
bou.ke/monkey v1.0.2
github.com/google/go-cmp v0.3.1 // indirect
github.com/pkg/errors v0.8.1 // indirect
github.com/stretchr/testify v1.4.0
gotest.tools v2.2.0+incompatible
)
and command.go reads
package command
import "os/exec"
// RunCommand runs a command
func RunCommand() ([]byte, error) {
return exec.Command("profiles", "list", "-all").Output()
}
and command_test.go
package command
import (
"os/exec"
"reflect"
"testing"
"bou.ke/monkey"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestRunCommand(t *testing.T) {
var cmd *exec.Cmd
patchGuard := monkey.PatchInstanceMethod(reflect.TypeOf(cmd), "Output", func(_ *exec.Cmd) ([]byte, error) {
return []byte("foobar"), nil
})
defer patchGuard.Unpatch()
output, err := RunCommand()
require.NoError(t, err)
assert.Equal(t, []byte("foobar"), output)
}
This test passes.
Now, in my 'real' repo I have a similar unit test
func TestFindIdentity(t *testing.T) {
certPEM, err := ioutil.ReadFile("testdata/6dc9bf91-37c6-4882-bfaf-301f118f7fac.pem")
require.NoError(t, err)
var cmd *exec.Cmd
patchGuard := monkey.PatchInstanceMethod(reflect.TypeOf(cmd), "Output", func(_ *exec.Cmd) ([]byte, error) {
output, err := ioutil.ReadFile("testdata/find_identity_match.txt")
require.NoError(t, err)
return output, nil
})
defer patchGuard.Unpatch()
found, err := FindIdentity(certPEM)
assert.True(t, found)
}
where FindIdentity() reads
// FindIdentity checks whether there is an identity (certificate + private key) for the given certificate in the system keychain
func FindIdentity(certPEM []byte) (bool, error) {
ctx, cancel := context.WithTimeout(context.TODO(), time.Second*5)
defer cancel()
fingerprint, err := GetFingerprint(certPEM)
if err != nil {
return false, fmt.Errorf("get cert fingerprint: %v", err)
}
output, err := exec.CommandContext(ctx, cmdSecurity, "find-identity", systemKeychain).Output()
if err != nil {
return false, fmt.Errorf("find identity: %v", err)
}
return strings.Contains(string(output), fingerprint), nil
}
// GetFingerprint generates a SHA-1 fingerprint of a certificate, which is how it can be identified from the `security` command
func GetFingerprint(certPEM []byte) (string, error) {
block, _ := pem.Decode(certPEM)
if block == nil {
return "", errors.New("failed to decode cert PEM")
}
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return "", fmt.Errorf("parse certificate: %v", err)
}
fingerprint := fmt.Sprintf("%x", sha1.Sum(cert.Raw))
fingerprint = strings.Replace(fingerprint, " ", "", -1)
return strings.ToUpper(fingerprint), nil
}
So similarly, it uses a Command which is patched in the unit test. However, if I try to run the unit test I get this error:
Running tool: /usr/local/opt/go#1.12/bin/go test -timeout 30s github.com/fleetsmith/agent/agent/auth/defaultauth -run ^(TestFindIdentity)$
--- FAIL: TestFindIdentity (0.00s)
panic: permission denied [recovered]
panic: permission denied
goroutine 25 [running]:
testing.tRunner.func1(0xc000494100)
/usr/local/Cellar/go#1.12/1.12.12/libexec/src/testing/testing.go:830 +0x392
panic(0x48fede0, 0xc000554730)
/usr/local/Cellar/go#1.12/1.12.12/libexec/src/runtime/panic.go:522 +0x1b5
bou.ke/monkey.mprotectCrossPage(0x41c20e0, 0xc, 0x7)
/Users/kurt/go/pkg/mod/bou.ke/monkey#v1.0.2/replace_unix.go:15 +0xe6
bou.ke/monkey.copyToLocation(0x41c20e0, 0xc0000ebd2c, 0xc, 0xc)
/Users/kurt/go/pkg/mod/bou.ke/monkey#v1.0.2/replace_unix.go:26 +0x6d
bou.ke/monkey.replaceFunction(0x41c20e0, 0xc0001a2510, 0x13, 0x41c20e0, 0x48c3b00)
/Users/kurt/go/pkg/mod/bou.ke/monkey#v1.0.2/replace.go:29 +0xe6
bou.ke/monkey.patchValue(0x48c3b60, 0xc0000bc078, 0x13, 0x48c3b60, 0xc0001a2510, 0x13)
/Users/kurt/go/pkg/mod/bou.ke/monkey#v1.0.2/monkey.go:87 +0x22f
bou.ke/monkey.PatchInstanceMethod(0x4b359a0, 0x4996280, 0x49d0699, 0x6, 0x48c3b60, 0xc0001a2510, 0x0)
/Users/kurt/go/pkg/mod/bou.ke/monkey#v1.0.2/monkey.go:62 +0x160
github.com/fleetsmith/agent/agent/auth/defaultauth.TestFindIdentity(0xc000494100)
/Users/kurt/go/src/github.com/fleetsmith/agent/agent/auth/defaultauth/keychain_test.go:46 +0x146
testing.tRunner(0xc000494100, 0x4a30260)
/usr/local/Cellar/go#1.12/1.12.12/libexec/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
/usr/local/Cellar/go#1.12/1.12.12/libexec/src/testing/testing.go:916 +0x35a
FAIL github.com/fleetsmith/agent/agent/auth/defaultauth 0.390s
Error: Tests failed.
Concretely, I get a permission denied panic upon calling monkey.PatchInstanceMethod at this line:
patchGuard := monkey.PatchInstanceMethod(reflect.TypeOf(cmd), "Output", func(_ *exec.Cmd) ([]byte, error) {
})
Any idea what could cause this? It must be some difference between my 'real' repo and my scratch repo.
mprotect syscall fails on MacOS Catalina, more explanation on it: Using mprotect to make text segment writable on macOS
If you are using a go monkey library in go test. Then you can not run go test directly. Actually go test do the followings:
compile the test functions into a temporary binary file
execute the binary
delete the temporary binary file
To work around this issue, we need to first generate the test binary, the modify the compiled binary with dd. e.g.
$ go test -c -o test-bin mytest/abc
With -c and -o option, go test will generate a binary named test-bin. Then modify the binary by dd command to set __TEXT(max_prot) to 0x7 after linking:
$ printf '\x07' | dd of=test-bin bs=1 seek=160 count=1 conv=notrunc
Finally, you can run the test binary:
./test-bin
You could try this: https://github.com/eisenxp/macos-golink-wrapper
It is a solution to "syscall.Mprotect panic: permission denied" in Golang on macOS Catalina 10.15.x when using gomonkey or gohook.
Download the tool.
cd `go env GOPATH`
git clone https://github.com/eisenxp/macos-golink-wrapper.git
Rename file link to original_link
mv `go env GOTOOLDIR`/link `go env GOTOOLDIR`/original_link
Copy the tool to GOTOOLDIR
cp `go env GOPATH`/macos-golink-wrapper/link `go env GOTOOLDIR`/link
Add execution permission to link
chmod +x `go env GOTOOLDIR`/link
This solved my problem, hope it helps you.

Golang: How to auto-restart process when binary updated?

I built the golang app in local, then scp to server. I need to stop the process and restart manually. Is there any way to auto-restart the process when binary updated?
While this is generally better to be implemented off-process using something like daemontools or similar, there are some cases when you want/need it to be done inside your program.
Doing it inside your program can be tricky depending on the program characteristics such as connections or files it may have open, etc.
Having said that, here you have an implementation which would work in most cases:
package main
import (
"log"
"os"
"syscall"
"time"
"github.com/fsnotify/fsnotify"
"github.com/kardianos/osext"
)
func setupWatcher() (chan struct{}, error) {
file, err := osext.Executable()
if err != nil {
return nil, err
}
log.Printf("watching %q\n", file)
w, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
done := make(chan struct{})
go func() {
for {
select {
case e := <-w.Events:
log.Printf("watcher received: %+v", e)
err := syscall.Exec(file, os.Args, os.Environ())
if err != nil {
log.Fatal(err)
}
case err := <-w.Errors:
log.Printf("watcher error: %+v", err)
case <-done:
log.Print("watcher shutting down")
return
}
}
}()
err = w.Add(file)
if err != nil {
return nil, err
}
return done, nil
}
func main() {
log.Print("program starting")
watcher, err := setupWatcher()
if err != nil {
// do something sensible
log.Fatal(err)
}
// continue with app startup
time.Sleep(100 * time.Minute) // just for testing
// eventually you may need to end the watcher
close(watcher) // this way you can
}
Then you do
% go build main.go
% ./main
2016/12/29 14:15:06 program starting
2016/12/29 14:15:06 watching "/home/plalloni/tmp/v/main"
And here the output it produced after you run (in other terminal) some successive "go build main.go" (which "updates" the running binary).
2016/12/29 14:15:32 watcher received: "/home/plalloni/tmp/v/main": CHMOD
2016/12/29 14:15:32 program starting
2016/12/29 14:15:32 watching "/home/plalloni/tmp/v/main"
2016/12/29 14:15:38 watcher received: "/home/plalloni/tmp/v/main": CHMOD
2016/12/29 14:15:38 program starting
2016/12/29 14:15:38 watching "/home/plalloni/tmp/v/main"
Hope it helps.
You can use https://github.com/slayer/autorestart
package main
import "github.com/slayer/autorestart"
func main() {
autorestart.StartWatcher()
http.ListenAndServe(":8080", nil) // for example
}
Does it need to be sophisticated? You could have entr running and trigger an updater script when the binary changes.
http://entrproject.org/
e.g.
echo 'binary_path' | entr script.sh &
I have a resolution about this case.
See also.
https://github.com/narita-takeru/cmdrevive
example
cmdrevive ./htmls/ ".html$" (application) (arguments)
So, this case applicable.
cmdrevive "/(app path)" "(app filename)" (app full path) (arguments)
If (app filename) changed on (app path) directory, then restart (app full path) with (arguments).
How about this one?

Start detached command with redirect to file

I'm trying to start a command in a detached process so that it can continue after go program exits. I need to redirect the output of the command to a file.
What I need is something like this:
func main() {
command := exec.Command("/tmp/test.sh", ">", "/tmp/out")
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Obviously such redirect doesn't work. As I immediately exit from the program after starting the long running command, I cannot open a file and bind it to the Stdout.
Is there any way to achieve such a redirect?
You may start a shell which executes your command / app, and you may redirect its output to a file. The shell will continue to run and execute your script / app even if your Go app exits.
Example:
cmd := exec.Command("sh", "-c", "/tmp/test.sh > /tmp/out")
if err := cmd.Start(); err != nil {
panic(err)
}
fmt.Println("Process ID:", cmd.Process.Pid)
Test it with this simple Go app (replace /tmp/test.sh with the name of the executable binary you compile this into):
package main
import ("fmt"; "time")
func main() {
for i := 0; i < 10; i++ {
fmt.Printf("%d.: %v\n", i, time.Now())
time.Sleep(time.Second)
}
}
This app simply prints a line to the standard output once every second. You can see how the output file is being written e.g. with tail -f /tmp/out.
Note that you may use other shells to execute your scripts to your liking (and to what the test.sh script dictates).
For example to use bash:
cmd := exec.Command("/bin/bash", "-c", "/tmp/test.sh > /tmp/out")
// rest is unchanged
Note that the command to be executed by the shell is passed as a single string argument, and it is not broken down into multiple as you would do it if you were to execute it directly in the command prompt.
Maybe you can try to use this: https://stackoverflow.com/a/28918814/2728768
Opening a file (and os.File implements io.Writer), and then passing it as the command.Stdout could do the trick:
func main() {
command := exec.Command("./tmp/test.sh")
f, err := os.OpenFile("/tmp/out", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Printf("error opening file: %v", err)
}
defer f.Close()
// On this line you're going to redirect the output to a file
command.Stdout = f
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Not sure this could be a viable solution for your case. I've tried it locally and it seems working... remember that your user should be able to create/update the file.

Exec external program/script and detect if it requests user input

given the following example:
// test.go
package main
import (
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("login")
in, _ := cmd.StdinPipe()
in.Write([]byte("user"))
out, err := cmd.CombinedOutput()
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%s", out)
}
How can I detect that the process is not going to finish, because it is waiting for user input?
I'm trying to be able to run any script, but abort it if for some reason it tries to read from stdin.
Thanks!
Detecting that the process is not going to finish is a difficult problem. In fact, it is one of the classic "unsolvable" problems in Computer Science: the Halting Problem.
In general, when you are calling exec.Command and will not be passing it any input, it will cause the program to read from your OS's null device (see documentation in the exec.Cmd fields). In your code (and mine below), you explicitly create a pipe (though you should check the error return of StdinPipe in case it is not created correctly), so you should subsequently call in.Close(). In either case, the subprocess will get an EOF and should clean up after itself and exit.
To help with processes that don't handle input correctly or otherwise get themselves stuck, the general solution is to use a timeout. In Go, you can use goroutines for this:
// Set your timeout
const CommandTimeout = 5 * time.Second
func main() {
cmd := exec.Command("login")
// Set up the input
in, err := cmd.StdinPipe()
if err != nil {
log.Fatalf("failed to create pipe for STDIN: %s", err)
}
// Write the input and close
go func() {
defer in.Close()
fmt.Fprintln(in, "user")
}()
// Capture the output
var b bytes.Buffer
cmd.Stdout, cmd.Stderr = &b, &b
// Start the process
if err := cmd.Start(); err != nil {
log.Fatalf("failed to start command: %s", err)
}
// Kill the process if it doesn't exit in time
defer time.AfterFunc(CommandTimeout, func() {
log.Printf("command timed out")
cmd.Process.Kill()
}).Stop()
// Wait for the process to finish
if err := cmd.Wait(); err != nil {
log.Fatalf("command failed: %s", err)
}
// Print out the output
fmt.Printf("Output:\n%s", b.String())
}
In the code above, there are actually three main goroutines of interest: the main goroutine spawns the subprocess and waits for it to exit; a timer goroutine is sent off in the background to kill the process if it's not Stopped in time; and a goroutine that writes the output to the program when it's ready to read it.
Although this would not allow you to "detect" the program trying to read from stdin, I would just close stdin. This way, the child process will just receive an EOF when it tried to read. Most programs know how to handle a closed stdin.
// All error handling excluded
cmd := exec.Command("login")
in, _ := cmd.StdinPipe()
cmd.Start()
in.Close()
cmd.Wait()
Unfortunately, this means you can't use combined output, the following code should allow you to do the same thing. It requires you to import the bytes package.
var buf = new(bytes.Buffer)
cmd.Stdout = buf
cmd.Stderr = buf
After cmd.Wait(), you can then do:
out := buf.Bytes()
I think the solution is to run the child process with closed stdin - by adjusting the Cmd.Stdin appropriately and then Runinng it afterwards instead of using CombinedOutput().
Finally, I'm going to implement a combination of Kyle Lemons answer and forcing the new process have it's own session without a terminal attached to it, so that the executed comand will be aware that there is no terminal to read from.
// test.go
package main
import (
"log"
"os/exec"
"syscall"
)
func main() {
cmd := exec.Command("./test.sh")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal("error:", err)
}
log.Printf("%s", out)
}

Resources