I have a Go program hosting a simple HTTP service on localhost:8080 so I can connect my public nginx host to it via the proxy_pass directive, as a reverse proxy to serve part of my site's requests. This is all working great, no problems there.
I want to convert the Go program to host the HTTP service on a Unix domain socket instead of a local TCP socket for improved security and to reduce the unnecessary protocol overhead of TCP.
PROBLEM:
The problem is that Unix domain sockets cannot be reused once they are bind() to, even after program termination. The second time (and every time after) I run the Go program it exits with a fatal error "address already in use".
Common practice is to unlink() Unix domain sockets (i.e. remove the file) when the server shuts down. However, this is tricky in Go. My first attempt was to use the defer statement in my main func (see below), but it is not getting run if I interrupt the process with a signal like CTRL-C. I suppose this is to be expected. Disappointing, but not unexpected.
QUESTION: Is there a best practice on how to unlink() the socket when the server process shuts down (either gracefully or ungracefully)?
Here's part of my func main() that starts the server listening for reference:
// Create the HTTP server listening on the requested socket:
l, err := net.Listen("unix", "/tmp/mysocket")
if err != nil {
log.Fatal(err)
} else {
// Unix sockets must be unlink()ed before being reused again.
// Unfortunately, this defer is not run when a signal is received, e.g. CTRL-C.
defer func() {
os.Remove("/tmp/mysocket")
}()
log.Fatal(http.Serve(l, http.HandlerFunc(indexHtml)))
}
Here is the complete solution I used. The code I posted in my question was a simplified version for clear demonstration purposes.
// Create the socket to listen on:
l, err := net.Listen(socketType, socketAddr)
if err != nil {
log.Fatal(err)
return
}
// Unix sockets must be unlink()ed before being reused again.
// Handle common process-killing signals so we can gracefully shut down:
sigc := make(chan os.Signal, 1)
signal.Notify(sigc, os.Interrupt, os.Kill, syscall.SIGTERM)
go func(c chan os.Signal) {
// Wait for a SIGINT or SIGKILL:
sig := <-c
log.Printf("Caught signal %s: shutting down.", sig)
// Stop listening (and unlink the socket if unix type):
l.Close()
// And we're done:
os.Exit(0)
}(sigc)
// Start the HTTP server:
log.Fatal(http.Serve(l, http.HandlerFunc(indexHtml)))
I sure hope this is good and effective Go code that would make the Go authors proud. It certainly looks so to me. If it is not, that would be embarrassing on my part. :)
For anyone curious, this is part of https://github.com/JamesDunne/go-index-html which is a simple HTTP directory listing generator with some extra features that web servers don't give you out of the box.
You can end your main func with the signal handler and spawn separate go routines for your other tasks instead. That way, you can leverage the defer mechanism and handle all (signal-based or not) shut downs cleanly:
func main() {
// Create the HTTP server listening on the requested socket:
l, err := net.Listen("unix", "/tmp/mysocket")
if err != nil {
log.Fatal(err)
return
}
// Just work with defer here; this works as long as the signal handling
// happens in the main Go routine.
defer l.Close()
// Make sure the server does not block the main
go func() {
log.Fatal(http.Serve(l, http.HandlerFunc(indexHtml)))
}()
// Use a buffered channel so we don't miss any signals
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt, os.Kill, syscall.SIGTERM)
// Block until a signal is received.
s := <-c
fmt.Println("Got signal:", s)
// ...and exit, running all the defer statements
}
In modern Go, you may use the syscall.Unlink() - docs here:
import (
"net"
"syscall"
...
)
...
socketpath := "/tmp/somesocket"
// carry on with your socket creation:
addr, err := net.ResolveUnixAddr("unixgram", socketpath)
if err != nil {
return err;
}
// always remove the named socket from the fs if its there
err = syscall.Unlink(socketpath)
if err != nil {
// not really important if it fails
log.Error("Unlink()",err)
}
// carry on with socket bind()
conn, err := net.ListenUnixgram("unixgram", addr);
if err != nil {
return err;
}
Related
I'm working on a Go project that require calling an initiation function (initFunction) in a separated goroutine (to ensure this function does not interfere with the rest of the project). initFunction must not take more than 30 seconds, so I thought I would use context.WithTimeout. Lastly, initFunction must be able to notify errors to the caller, so I thought of making an error channel and calling initFunction from an anonymous function, to recieve and report the error.
func RunInitGoRoutine(initFunction func(config string)error) error {
initErr := make(chan error)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Seconds)
go func() {
<-ctx.Done() // Line 7
err := initFunction(config)
initErr <-err
}()
select {
case res := <-initErr:
return res.err
case <-ctx.Done():
err := errors.New("Deadline")
return err
}
}
I'm quite new to Go, so I'm asking for feedbacks about the above code.
I have some doubt about Line 7. I used this to ensure the anonymous function is "included" under ctx and is therefore killed and freed and everything once timeout expires, but I'm not sure I have done the right thing.
Second thing is, I know I should be calling cancel( ) somewhere, but I can't put my finger around where.
Lastly, really any feedback is welcome, being it about efficency, style, correctness or anything.
In Go the pratice is to communicate via channels. So best thing is probably to share a channel on your context so others can consume from the channel.
As you are stating you are new to Go, I wrote a whole bunch of articles on Go (Beginner level) https://marcofranssen.nl/categories/golang.
Read from old to new to get familiar with the language.
Regarding the channel specifics you should have a look at this article.
https://marcofranssen.nl/concurrency-in-go
A pratical example of a webserver listening for ctrl+c and then gracefully shutting down the server using channels is described in this blog post.
https://marcofranssen.nl/improved-graceful-shutdown-webserver
In essence we run the server in a background routine
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
srv.l.Fatal("Could not listen on", zap.String("addr", srv.Addr), zap.Error(err))
}
}()
and then we have some code that is blocking the main routine by listening on a channel for the shutdown signal.
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt)
sig := <-quit
srv.l.Info("Server is shutting down", zap.String("reason", sig.String()))
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
srv.SetKeepAlivesEnabled(false)
if err := srv.Shutdown(ctx); err != nil {
srv.l.Fatal("Could not gracefully shutdown the server", zap.Error(err))
}
srv.l.Info("Server stopped")
This is very similar to your usecase. So running your init in a background routine and then consume the channel waiting for the result of this init routine.
package main
import (
"fmt"
"time"
)
type InitResult struct {
Message string
}
func main() {
initResult := make(chan InitResult, 0)
go func(c chan<- InitResult) {
time.Sleep(5 * time.Second)
// here we are publishing the result on the channel
c <- InitResult{Message: "Initialization succeeded"}
}(initResult)
fmt.Println("Started initializing")
// here we have a blocking operation consuming the channel
res := <-initResult
fmt.Printf("Init result: %s", res.Message)
}
https://play.golang.org/p/_YGIrdNVZx6
You could also add an error field on the struct so you could do you usual way of error checking.
While practicing a bit on concurrency with go, I faced a problem.
I am making a dns filtering server, it receives a dns packet, and act as a recursive server with caching and its own records.
Starting the server works all great, but when I try to close the listener to shut the server down, it just hangs until there is another packet arriving.
So what I have been trying to do is run another anonymous function that receives a message from a channel, close the listener, clear cache and exit from the server goroutine.
Related code:
func StartDNSServer(recursiveServer, ip string, port int, ch chan bool) {
stopChn := make(chan bool)
//Listen on UDP Port
addr := net.UDPAddr{
Port: port,
IP: net.ParseIP(ip),
}
fmt.Println("[*] Starting dns server...")
u, err := net.ListenUDP("udp", &addr)
//u.ReadFrom is a blocking command, so if we try to stop the dns server it wont detect the data in ch until there is a new dns packet.
// anonymous function to close the udp listener to realease the blocking.
go func() {
select {
case <-ch:
stopChn <- false
u.Close()
}
}()
defer u.Close()
if err != nil {
log.Panic("Could not listen on given port, maybe another application is using it?")
}
fmt.Println("[*] Started!")
// Wait to get request on that port
for {
select {
case <-stopChn:
fmt.Println("[*] Deleting records from cache.")
Cache.DeleteExpired()
fmt.Println("[*] Stopping dns server")
runtime.Goexit()
default:
tmp := make([]byte, 1024)
_, addr, _ := u.ReadFrom(tmp)
clientAddr := addr
packet := gopacket.NewPacket(tmp, layers.LayerTypeDNS, gopacket.Default)
dnsPacket := packet.Layer(layers.LayerTypeDNS)
tcp, _ := dnsPacket.(*layers.DNS)
go handleDNS(u, clientAddr, tcp, recursiveServer)
continue
}
}
}
From what I managed to get, the select statement under the infinite for loop is entering the default section, and then waiting for a packet, after that its handling the packet, it is "noticing" that the stopChn has received a signal, and shutting down.
So if this is really the case, I then tried to change the stopChn := make(chan bool) into a buffered channel stopChn := make(chan bool, 1).
Now it really does trigger the desired action immediately, but panics with the error:
panic: runtime error: index out of range [0] with length 0
goroutine 16 [running]:
/C/Users/moti_/OneDrive/Desktop/GoDNSFilter/internals/dnsserver.handleDNS(0xc000006660,
0x0, 0x0, 0xc0001ea180, 0xa98f34, 0x7)
C:/Users/moti_/OneDrive/Desktop/GoDNSFilter/internals/dnsserver/utils.go:32
+0x75b created by /C/Users/moti_/OneDrive/Desktop/GoDNSFilter/internals/dnsserver.StartDNSServer
C:/Users/moti_/OneDrive/Desktop/GoDNSFilter/internals/dnsserver/utils.go:151
+0x2be
I looked for it online, and some answers said that I might be sending too many signals into the stopChn but it would be blocking, not panicking..
Not sure if I am approaching this correctly, any help would be appreciated!
So, After #mh-cbon helped me a lot, heres the solution if any one ever gonna face the same problem.
As said before, I cannot close one goroutine from another, so I had to use 2 channels, one for the not blocked goroutine, which will close the listener and then clear out cache and all.
Still got to compare error with error vars, but for solution its enough for now.
func StartDNSServer(recursiveServer, ip string, port int, ch chan bool) {
//Listen on UDP Port
addr := net.UDPAddr{
Port: port,
IP: net.ParseIP(ip),
}
fmt.Println("[*] Starting dns server...")
u, err := net.ListenUDP("udp", &addr)
//u.ReadFrom is a blocking command, so if we try to stop the dns server it wont detect the data in ch until there is a new dns packet.
// anonymous function to close the udp listener to realease the blocking.
go func() {
select {
case <-ch:
u.Close()
}
}()
defer u.Close()
if err != nil {
log.Panic("Could not listen on given port, maybe another application is using it?")
}
fmt.Println("[*] Started!")
// Wait to get request on that port
for {
tmp := make([]byte, 1024)
_, addr, err := u.ReadFrom(tmp)
if err != nil {
if strings.Contains(err.Error(), "use of closed network connection") {
fmt.Println("[*] Deleting records from cache.")
Cache.DeleteExpired()
fmt.Println("[*] Stopping dns server")
return
}
clientAddr := addr
packet := gopacket.NewPacket(tmp, layers.LayerTypeDNS, gopacket.Default)
dnsPacket := packet.Layer(layers.LayerTypeDNS)
tcp, _ := dnsPacket.(*layers.DNS)
go handleDNS(u, clientAddr, tcp, recursiveServer)
continue
}
}
}
I have a gRPC server, and I have implemented graceful shutdown of my gRPC server something like this
fun main() {
//Some code
term := make(chan os.Signal)
go func() {
if err := grpcServer.Serve(lis); err != nil {
term <- syscall.SIGINT
}
}()
signal.Notify(term, syscall.SIGTERM, syscall.SIGINT)
<-term
server.GracefulStop()
closeDbConnections()
}
This works fine.
If instead I write the grpcServer.Serve() logic in main goroutine and instead put the shutdown handler logic into another goroutine, statements after server.GracefulStop() usually do not execute. Some DbConnections are closed, if closeDbConnections() is executed at all.
server.GracefulStop() is a blocking call. Definitely grpcServer.Serve() finishes before server.GracefulStop() completes. So, how long does main goroutine take to stop after this call returns?
The problematic code
func main() {
term := make(chan os.Signal)
go func() {
signal.Notify(term, syscall.SIGTERM, syscall.SIGINT)
<-term
server.GracefulStop()
closeDbConnections()
}()
if err := grpcServer.Serve(lis); err != nil {
term <- syscall.SIGINT
}
}
This case does not work as expected. After server.GracefulStop() is done, closeDbConnections() may or may not run (usually does not run to completion). I was testing the later case by sending SIGINT by hitting Ctrl-C from my terminal.
Can someone please explain this behavior?
I'm not sure about your question (please clarify it), but I would suggest you to refactor your main in this way:
func main() {
// ...
errChan := make(chan error)
stopChan := make(chan os.Signal)
// bind OS events to the signal channel
signal.Notify(stopChan, syscall.SIGTERM, syscall.SIGINT)
// run blocking call in a separate goroutine, report errors via channel
go func() {
if err := grpcServer.Serve(lis); err != nil {
errChan <- err
}
}()
// terminate your environment gracefully before leaving main function
defer func() {
server.GracefulStop()
closeDbConnections()
}()
// block until either OS signal, or server fatal error
select {
case err := <-errChan:
log.Printf("Fatal error: %v\n", err)
case <-stopChan:
}
I don't think it's a good idea to mix system events and server errors, like you do in your example: in case if Serve fails, you just ignore the error and emit system event, which actually didn't happen. Try another approach when there are two transports (channels) for two different kind of event that cause process termination.
http.Serve either returns an error as soon as it is called or blocks if successfully executing.
How can I make it so that if it blocks it does so in its own goroutine? I currently have the following code:
func serveOrErr(l net.Listener, handler http.Handler) error {
starting := make(chan struct{})
serveErr := make(chan error)
go func() {
starting <- struct{}{}
if err := http.Serve(l, handler); err != nil {
serveErr <- err
}
}()
<-starting
select {
case err := <-serveErr:
return err
default:
return nil
}
}
This seemed like a good start and works on my test machine but I believe that there are no guarantees that serveErr <- err would be called before case err := <-serveErr therefore leading to inconsistent results due to a data race if http.Serve were to produce an error.
http.Serve either returns an error as soon as it is called or blocks if successfully executing
This assumption is not correct. And I believe it rarely occurs. http.Serve calls net.Listener.Accept in the loop – an error can occur any time (socket closed, too many open file descriptors etc.). It's http.ListenAndServe, usually being used for running http servers, which often fails early while binding listening socket (no permissions, address already in use).
In my opinion what you're trying to do is wrong, unless really your net.Listener.Accept is failing on the first call for some reason. Is it? If you want to be 100% sure your server is working, you could try to connect to it (and maybe actually transmit something), but once you successfully bound the socket I don't see it really necessary.
You could use a timeout on your select statement, e.g.
timeout := time.After(5 * time.Millisecond) // TODO: ajust the value
select {
case err := <-serveErr:
return err
case _ := <- timeout:
return nil
}
This way your select will block until serveErr has a value or the specified timeout has elapsed. Note that the execution of your function will therefore block the calling goroutine for up to the duration of the specified timeout.
Rob Pike's excellent talk on go concurrency patterns might be helpful.
It doesn't seem possible to have two way communication via channels with a goroutine which is performing file operations, unless you block the channel communication on the file operations. How can I work around the limits this imposes?
Another way to phrase this question...
If I have a loop similar to the following running in a goroutine, how can I tell it to close the connection and exit without blocking on the next Read?
func readLines(response *http.Response, outgoing chan string) error {
defer response.Body.Close()
reader := bufio.NewReader(response.Body)
for {
line, err := reader.ReadString('\n')
if err != nil {
return err
}
outgoing <- line
}
}
It's not possible for it to read from a channel that tells it when to close down because it's blocking on the network reads (in my case, that can take hours).
It doesn't appear to be safe to simply call Close() from outside the goroutine, since the Read/Close methods don't appear to be fully thread safe.
I could simply put a lock around references to response.Body that used inside/outside the routine, but would cause the external code to block until a pending read completes, and I specifically want to be able to interrupt an in-progress read.
To address this scenario, several io.ReadCloser implementations in the standard library support concurrent calls to Read and Close where Close interrupts an active Read.
The response body reader created by net/http Transport is one of those implementations. It is safe to concurrently call Read and Close on the response body.
You can also interrupt an active Read on the response body by calling the Transport CancelRequest method.
Here's how implement cancel using close on the body:
func readLines(response *http.Response, outgoing chan string, done chan struct{}) error {
cancel := make(chan struct{})
go func() {
select {
case <-done:
response.Body.Close()
case <-cancel:
return
}()
defer response.Body.Close()
defer close(cancel) // ensure that goroutine exits
reader := bufio.NewReader(response.Body)
for {
line, err := reader.ReadString('\n')
if err != nil {
return err
}
outgoing <- line
}
}
Calling close(done) from another goroutine will cancel reads on the body.