Golang: How to auto-restart process when binary updated? - go

I built the golang app in local, then scp to server. I need to stop the process and restart manually. Is there any way to auto-restart the process when binary updated?

While this is generally better to be implemented off-process using something like daemontools or similar, there are some cases when you want/need it to be done inside your program.
Doing it inside your program can be tricky depending on the program characteristics such as connections or files it may have open, etc.
Having said that, here you have an implementation which would work in most cases:
package main
import (
"log"
"os"
"syscall"
"time"
"github.com/fsnotify/fsnotify"
"github.com/kardianos/osext"
)
func setupWatcher() (chan struct{}, error) {
file, err := osext.Executable()
if err != nil {
return nil, err
}
log.Printf("watching %q\n", file)
w, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
done := make(chan struct{})
go func() {
for {
select {
case e := <-w.Events:
log.Printf("watcher received: %+v", e)
err := syscall.Exec(file, os.Args, os.Environ())
if err != nil {
log.Fatal(err)
}
case err := <-w.Errors:
log.Printf("watcher error: %+v", err)
case <-done:
log.Print("watcher shutting down")
return
}
}
}()
err = w.Add(file)
if err != nil {
return nil, err
}
return done, nil
}
func main() {
log.Print("program starting")
watcher, err := setupWatcher()
if err != nil {
// do something sensible
log.Fatal(err)
}
// continue with app startup
time.Sleep(100 * time.Minute) // just for testing
// eventually you may need to end the watcher
close(watcher) // this way you can
}
Then you do
% go build main.go
% ./main
2016/12/29 14:15:06 program starting
2016/12/29 14:15:06 watching "/home/plalloni/tmp/v/main"
And here the output it produced after you run (in other terminal) some successive "go build main.go" (which "updates" the running binary).
2016/12/29 14:15:32 watcher received: "/home/plalloni/tmp/v/main": CHMOD
2016/12/29 14:15:32 program starting
2016/12/29 14:15:32 watching "/home/plalloni/tmp/v/main"
2016/12/29 14:15:38 watcher received: "/home/plalloni/tmp/v/main": CHMOD
2016/12/29 14:15:38 program starting
2016/12/29 14:15:38 watching "/home/plalloni/tmp/v/main"
Hope it helps.

You can use https://github.com/slayer/autorestart
package main
import "github.com/slayer/autorestart"
func main() {
autorestart.StartWatcher()
http.ListenAndServe(":8080", nil) // for example
}

Does it need to be sophisticated? You could have entr running and trigger an updater script when the binary changes.
http://entrproject.org/
e.g.
echo 'binary_path' | entr script.sh &

I have a resolution about this case.
See also.
https://github.com/narita-takeru/cmdrevive
example
cmdrevive ./htmls/ ".html$" (application) (arguments)
So, this case applicable.
cmdrevive "/(app path)" "(app filename)" (app full path) (arguments)
If (app filename) changed on (app path) directory, then restart (app full path) with (arguments).
How about this one?

Related

`go tool pprof` reports error `unrecognized profile format`

I am trying to generate PNG of goroutine profile with this demo, but it reports error parsing profile: unrecognized profile format. failed to fetch any source profiles
package main
import (
"fmt"
"log"
"os"
"os/exec"
"runtime/pprof"
"time"
)
func main() {
// start demo go routine
go func() {
for {
time.Sleep(time.Second)
}
}()
// get executable binary path
exePath, err := os.Executable()
if err != nil {
log.Println(err.Error())
return
}
log.Println("exePath", exePath)
// generate goroutine profile
profilePath := exePath + ".profile"
f, err := os.Create(profilePath)
if err != nil {
log.Println(err.Error())
return
}
defer f.Close()
if err := pprof.Lookup("goroutine").WriteTo(f, 2); err != nil {
log.Println(err.Error())
return
}
// generate PNG
pngPath := exePath + ".png"
result, err := exec.Command("go", "tool", "pprof", "-png", "-output", pngPath, exePath, profilePath).CombinedOutput()
if err != nil {
log.Println("make error:", err.Error())
}
log.Println("make result:", string(result))
}
You are using a debug value of 2 in pprof.Lookup("goroutine").WriteTo(f, 2). This will produce an output which the pprof tool can't parse, it is meant as directly human readable.
From the pprof.(*Profile).WriteTo docs:
The debug parameter enables additional output. Passing debug=0 writes the gzip-compressed protocol buffer described in https://github.com/google/pprof/tree/master/proto#overview. Passing debug=1 writes the legacy text format with comments translating addresses to function names and line numbers, so that a programmer can read the profile without tools.
The predefined profiles may assign meaning to other debug values; for example, when printing the "goroutine" profile, debug=2 means to print the goroutine stacks in the same form that a Go program uses when dying due to an unrecovered panic.
Changing this line to pprof.Lookup("goroutine").WriteTo(f, 0) resolves the issue.
Also, not relevant to the issue, but you might want to consider closing the file f.Close() before using it in the pprof command. WriteTo should flush the file to disk, but it can't hurt.
if err := pprof.Lookup("goroutine").WriteTo(f, 2); err != nil {
log.Println(err.Error())
return
}
if err := f.Close(); err != nil {
log.Println(err.Error())
return
}
// generate PNG
...

A go code program failed to run in the background

Go code running in the background
I am a beginner of the go language.
I wrote a small program that made a keyboard sound.
After go build main.go, you can hear the sound of the button in the current shell.
But when running ./main in the background or when re-opening a new shell will not hear the button sound.
This is what I need help with.
package main
import (
"fmt"
"github.com/eiannone/keyboard"
"github.com/faiface/beep"
"github.com/faiface/beep/speaker"
"github.com/faiface/beep/wav"
"os"
"time"
"log"
"path/filepath"
)
func main(){
env:= os.Getenv("HOME")
fmt.Println(env)
err := keyboard.Open()
if err != nil {
panic(err)
}
defer keyboard.Close()
for {
char, key, err := keyboard.GetKey()
if err != nil {
log.Fatal(err)
}else if (key == keyboard.KeyEsc){
break
}
n := int(char)
path := "./wav/*.wav"
fpath,_ := filepath.Glob(path)
name := fpath[n]
f, err := os.Open(name)
if err != nil {
log.Fatal(err)
}
streamer, format, err := wav.Decode(f)
if err != nil {
log.Fatal(err)
}
defer streamer.Close()
speaker.Init(format.SampleRate, format.SampleRate.N(time.Second/10))
done := make(chan bool)
speaker.Play(beep.Seq(streamer, beep.Callback(func() {
done <- true
})))
<-done
}
}
I want it to run in the background. I can hear the corresponding sound after pressing the keyboard.
If the program is running in the background it will not get any input from the keyboard.
Also since n depends on user input it would be easy for it to exceed the number of sound files and you would get an index out of bounds error at run-time. You should do something like this:
if fpath, _ := filepath.Glob("./wav/*.wav"); len(fpath) > 0 {
n := int(char)%len(fpath)
name := fpath[n]
...

Can`t figure out how to use buffers with binary web socket

everyone!
I'm trying to get my go code work with openstack serial console. It`s exposed via web socket. And i have problems with it.
I found gorrilla websocket lib (which is great) and took this example as a reference
With a few tweaks, now i have a code like this:
package main
import (
"log"
"net/url"
"os"
"os/signal"
"time"
"net/http"
"github.com/gorilla/websocket"
)
func main() {
DialSettings := &websocket.Dialer {
Proxy: http.ProxyFromEnvironment,
HandshakeTimeout: 45 * time.Second,
Subprotocols: []string{"binary",},
ReadBufferSize: 4096,
WriteBufferSize: 4096,
}
log.SetFlags(0)
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
u, _ := url.Parse("ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5") //websocket url as it outputs from 'nova get-serial-console test' cmd
log.Printf("connecting to %s", u.String())
c, _, err := DialSettings.Dial(u.String(), nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("%s", message)
}
}()
c.WriteMessage(websocket.TextMessage, []byte("\n")) //just to force output to console
for {
select {
case <-done:
return
case <-interrupt:
log.Println("interrupt")
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
And i get output like this:
connecting to ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5
CentOS Linux 7
(C
ore)
K
erne
l
3.10.0-862.el7.x86_64
o
n an
x
86_64
centos
-test login:
Total mess...
I think it's because i recieve just a chunks of bytes with no way to delimit them. I need some buffer to store them and when do something like bufio.ReadLine. But i'm not most experienced go programmer, and i run out of ideas how to do this. At the end i just need strings to work with.
The log package writes each log message on a separate line. If the log message does not end with a newline, then the log package will add one.
These extra newlines are garbling the output. To fix the output, replace the call to log.Printf("%s", message) with a function that does not add newlines to the output. Here are some options:
Write the message to stderr (same destination as default log package config):
os.Stderr.Write(message)
Write the message to stdout (a more conventional location to write program output):
os.Stdout.Write(message)

how can I get stdin to exec cmd in golang

I have this code
subProcess := exec.Cmd{
Path: execAble,
Args: []string{
fmt.Sprintf("-config=%s", *configPath),
fmt.Sprintf("-serverType=%s", *serverType),
fmt.Sprintf("-reload=%t", *reload),
fmt.Sprintf("-listenFD=%d", fd),
},
Dir: here,
}
subProcess.Stdout = os.Stdout
subProcess.Stderr = os.Stderr
logger.Info("starting subProcess:%s ", subProcess.Args)
if err := subProcess.Run(); err != nil {
logger.Fatal(err)
}
and then I do os.Exit(1) to stop the main process
I can get output from the subprocess
but I also want to put stdin to
I try
subProcess.Stdin = os.Stdin
but it does not work
I made a simple program (for testing). It reads a number and writes the given number out.
package main
import (
"fmt"
)
func main() {
fmt.Println("Hello, What's your favorite number?")
var i int
fmt.Scanf("%d\n", &i)
fmt.Println("Ah I like ", i, " too.")
}
And here is the modified code
package main
import (
"fmt"
"io"
"os"
"os/exec"
)
func main() {
subProcess := exec.Command("go", "run", "./helper/main.go") //Just for testing, replace with your subProcess
stdin, err := subProcess.StdinPipe()
if err != nil {
fmt.Println(err) //replace with logger, or anything you want
}
defer stdin.Close() // the doc says subProcess.Wait will close it, but I'm not sure, so I kept this line
subProcess.Stdout = os.Stdout
subProcess.Stderr = os.Stderr
fmt.Println("START") //for debug
if err = subProcess.Start(); err != nil { //Use start, not run
fmt.Println("An error occured: ", err) //replace with logger, or anything you want
}
io.WriteString(stdin, "4\n")
subProcess.Wait()
fmt.Println("END") //for debug
}
You interested about these lines
stdin, err := subProcess.StdinPipe()
if err != nil {
fmt.Println(err)
}
defer stdin.Close()
//...
io.WriteString(stdin, "4\n")
//...
subProcess.Wait()
Explanation of the above lines
We gain the subprocess' stdin, now we can write to it
We use our power and we write a number
We wait for our subprocess to complete
Output
START
Hello, What's your favorite number?
Ah I like 4 too.
END
For better understanding
There's now an updated example available in the Go docs: https://golang.org/pkg/os/exec/#Cmd.StdinPipe
If the subprocess doesn't continue before the stdin is closed, the io.WriteString() call needs to be wrapped inside an anonymous function:
func main() {
cmd := exec.Command("cat")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
io.WriteString(stdin, "values written to stdin are passed to cmd's standard input")
}()
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}
Though this question is a little old, but here is my answer:
This question is of course very platform specific as how standard IO is handled depends on the OS implementation and not on Go language. However, as general rule of thumb (due to some OSes being prevailing), "what you ask is not possible".
On most of modern operating systems you can pipe standard streams (as in #mraron's answer), you can detach them (this is how daemons work), but you cannot reassign or delegate them to another process.
I think this limitation is more because of security concern. There are still from time to time bugs being discovered that allow remote code execution, imagine if OS was allowing to reassign/delegate STDIN/OUT, then with such vulnerabilities the consequences would be disastrous.
While you cannot directly do this as #AlexKey wrote earlier still you can make some workarounds. If os prevents you to pipe your own standard streams who cares all you need 2 channels and 2 goroutines
var stdinChan chan []byte
var stdoutChan chan []byte
//when something happens in stdout of your code just pipe it to stdout chan
stdoutChan<-somehowGotDataFromStdOut
then you need two funcs as i mentioned before
func Pipein(){
for {
stdinFromProg.Write(<-stdInChan)
}
}
The same idea for the stdout

Parallel zip compression in Go

I am trying build a zip archive from a large number of small-medium sized files. I want to be able to do this concurrently, since compression is CPU intensive, and I'm running on a multi core server. Also I don't want to have the whole archive in memory, since its might turn out to be large.
My question is that do I have to compress every file and then combine manually combine everything together with zip header, checksum etc?
Any help would be greatly appreciated.
I don't think you can combine the zip headers.
What you could do is, run the zip.Writer sequentially, in a separate goroutine, and then spawn a new goroutine for each file that you want to read, and pipe those to the goroutine that is zipping them.
This should reduce the IO overhead that you get by reading the files sequentially, although it probably won't leverage multiple cores for the archiving itself.
Here's a working example. Note that, to keep things simple,
it does not handle errors nicely, just panics if something goes wrong,
and it does not use the defer statement too much, to demonstrate the order in which things should happen.
Since defer is LIFO, it can sometimes be confusing when you stack a lot of them together.
package main
import (
"archive/zip"
"io"
"os"
"sync"
)
func ZipWriter(files chan *os.File) *sync.WaitGroup {
f, err := os.Create("out.zip")
if err != nil {
panic(err)
}
var wg sync.WaitGroup
wg.Add(1)
zw := zip.NewWriter(f)
go func() {
// Note the order (LIFO):
defer wg.Done() // 2. signal that we're done
defer f.Close() // 1. close the file
var err error
var fw io.Writer
for f := range files {
// Loop until channel is closed.
if fw, err = zw.Create(f.Name()); err != nil {
panic(err)
}
io.Copy(fw, f)
if err = f.Close(); err != nil {
panic(err)
}
}
// The zip writer must be closed *before* f.Close() is called!
if err = zw.Close(); err != nil {
panic(err)
}
}()
return &wg
}
func main() {
files := make(chan *os.File)
wait := ZipWriter(files)
// Send all files to the zip writer.
var wg sync.WaitGroup
wg.Add(len(os.Args)-1)
for i, name := range os.Args {
if i == 0 {
continue
}
// Read each file in parallel:
go func(name string) {
defer wg.Done()
f, err := os.Open(name)
if err != nil {
panic(err)
}
files <- f
}(name)
}
wg.Wait()
// Once we're done sending the files, we can close the channel.
close(files)
// This will cause ZipWriter to break out of the loop, close the file,
// and unblock the next mutex:
wait.Wait()
}
Usage: go run example.go /path/to/*.log.
This is the order in which things should be happening:
Open output file for writing.
Create a zip.Writer with that file.
Kick off a goroutine listening for files on a channel.
Go through each file, this can be done in one goroutine per file.
Send each file to the goroutine created in step 3.
After processing each file in said goroutine, close the file to free up resources.
Once each file has been sent to said goroutine, close the channel.
Wait until the zipping has been done (which is done sequentially).
Once zipping is done (channel exhausted), the zip writer should be closed.
Only when the zip writer is closed, should the output file be closed.
Finally everything is closed, so close the sync.WaitGroup to tell the calling function that we're good to go. (A channel could also be used here, but sync.WaitGroup seems more elegant.)
When you get the signal from the zip writer that everything is properly closed, you can exit from main and terminate nicely.
This might not answer your question, but I've been using similar code to generate zip archives on-the-fly for a web service some time ago. It performed quite well, even though the actual zipping was done in a single goroutine. Overcoming the IO bottleneck can already be an improvement.
From the look of it, you won't be able to parallelise the compression using the standard library archive/zip package because:
Compression is performed by the io.Writer returned by zip.Writer.Create or CreateHeader.
Calling Create/CreateHeader implicitly closes the writer returned by the previous call.
So passing the writers returned by Create to multiple goroutines and writing to them in parallel will not work.
If you wanted to write your own parallel zip writer, you'd probably want to structure it something like this:
Have multiple goroutines compress files using the compress/flate module, and keep track of the CRC32 value and length of the uncompressed data. The output should be directed to temporary files. Note the compressed size of the data.
Once everything has been compressed, start writing the Zip file starting with the header.
Write out the file header followed by the contents of the corresponding temporary file for each compressed file.
Write out the central directory record and end record at the end of the file. All the required information should be available at this point.
For added parallelism, step 1 could be performed in parallel with the remaining steps by using a channel to indicate when compression of each file completes.
Due to the file format, you won't be able to perform parallel compression without either storing compressed data in memory or in temporary files.
With Go1.17, parallel compression and merging of zip files are possible using the archive/zip package.
An example is below. In the example, I create zip workers to create individual zip files and an entry provider worker which provides entries to be added to a zip file via a channel to zip workers. Actual files can be provided to the zip workers but I skipped that part.
package main
import (
"archive/zip"
"context"
"fmt"
"io"
"log"
"os"
"strings"
"golang.org/x/sync/errgroup"
)
const numOfZipWorkers = 10
type entry struct {
name string
rc io.ReadCloser
}
func main() {
log.SetFlags(log.LstdFlags | log.Lshortfile)
entCh := make(chan entry, numOfZipWorkers)
zpathCh := make(chan string, numOfZipWorkers)
group, ctx := errgroup.WithContext(context.Background())
for i := 0; i < numOfZipWorkers; i++ {
group.Go(func() error {
return zipWorker(ctx, entCh, zpathCh)
})
}
group.Go(func() error {
defer close(entCh) // Signal workers to stop.
return entryProvider(ctx, entCh)
})
err := group.Wait()
if err != nil {
log.Fatal(err)
}
f, err := os.OpenFile("output.zip", os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
zw := zip.NewWriter(f)
close(zpathCh)
for path := range zpathCh {
zrd, err := zip.OpenReader(path)
if err != nil {
log.Fatal(err)
}
for _, zf := range zrd.File {
err := zw.Copy(zf)
if err != nil {
log.Fatal(err)
}
}
_ = zrd.Close()
_ = os.Remove(path)
}
err = zw.Close()
if err != nil {
log.Fatal(err)
}
err = f.Close()
if err != nil {
log.Fatal(err)
}
}
func entryProvider(ctx context.Context, entCh chan<- entry) error {
for i := 0; i < 2*numOfZipWorkers; i++ {
select {
case <-ctx.Done():
return ctx.Err()
case entCh <- entry{
name: fmt.Sprintf("file_%d", i+1),
rc: io.NopCloser(strings.NewReader(fmt.Sprintf("content %d", i+1))),
}:
}
}
return nil
}
func zipWorker(ctx context.Context, entCh <-chan entry, zpathch chan<- string) error {
f, err := os.CreateTemp(".", "tmp-part-*")
if err != nil {
return err
}
zw := zip.NewWriter(f)
Loop:
for {
var (
ent entry
ok bool
)
select {
case <-ctx.Done():
err = ctx.Err()
break Loop
case ent, ok = <-entCh:
if !ok {
break Loop
}
}
hdr := &zip.FileHeader{
Name: ent.name,
Method: zip.Deflate, // zip.Store can also be used.
}
hdr.SetMode(0644)
w, e := zw.CreateHeader(hdr)
if e != nil {
_ = ent.rc.Close()
err = e
break
}
_, e = io.Copy(w, ent.rc)
_ = ent.rc.Close()
if e != nil {
err = e
break
}
}
if e := zw.Close(); e != nil && err == nil {
err = e
}
if e := f.Close(); e != nil && err == nil {
err = e
}
if err == nil {
select {
case <-ctx.Done():
err = ctx.Err()
case zpathch <- f.Name():
}
}
return err
}

Resources