I use https://github.com/NeowayLabs/wabbit/
When amqp.Channel is closing after wrong try channel.Consume, we have a not listened chan and function is not responding.
My code:
package main
import (
"fmt"
"github.com/NeowayLabs/wabbit"
"github.com/NeowayLabs/wabbit/amqptest"
"github.com/NeowayLabs/wabbit/amqptest/server"
)
func someFunc(amqpURL string) error {
conn, err := amqptest.Dial(amqpURL)
defer conn.Close()
channel, err := conn.Channel()
defer channel.Close()
consumer, err := channel.Consume(
"queue",
"consumer",
wabbit.Option{},
)
if err != nil {
return err // err = "Unknown queue 'queue'", but we never response it
}
fmt.Println(<-consumer)
return nil
}
func main() {
amqpURL := "127.0.0.1:32773"
fakeServer := server.NewServer(amqpURL)
err := fakeServer.Start()
defer fakeServer.Stop()
err = someFunc(amqpURL)
if err != nil {
panic(err)
}
fmt.Println("Happy end")
}
someFunc never responding with error, but I want to handle consumer errors.
someFunc never responds with an error because it gets hung up in the defer code.
When someFunc gets to the return err line, then it tries to run the defer statements that you set up in the beginning of the function. The first one that it tries is defer channel.Close().
The problem seems to be with this block in the wabbit library: https://github.com/NeowayLabs/wabbit/blob/d8bc549279ecd80204a8a83a868a14fdd81d1a1b/amqptest/server/channel.go#L315-L317
I think, although I am not sure, that writing to the consumer.done channel is a blocking operation because the channel is not buffered and does not have a receiver. See this: https://gobyexample.com/non-blocking-channel-operations for more information.
I commented that block of code out when running this locally and found that the rest of the code ran as you expected that it would, finally resulting in a panic: Unknown queue 'queue'
Related
I just want to follow a demo to try use kafka in go. I can successfully produce message by sarama, but when i want to consume the message, can not get it.
package main
import (
"fmt"
"github.com/Shopify/sarama"
)
// kafka consumer
func main() {
consumer, err := sarama.NewConsumer([]string{"127.0.0.1:9092"}, nil)
if err != nil {
fmt.Printf("fail to start consumer, err:%v\n", err)
return
}
partitionList, err := consumer.Partitions("test")
if err != nil {
fmt.Printf("fail to get list of partition:err%v\n", err)
return
}
fmt.Println(partitionList)
for partition := range partitionList {
pc, err := consumer.ConsumePartition("test", int32(partition), sarama.OffsetNewest)
if err != nil {
fmt.Printf("failed to start consumer for partition %d,err:%v\n", partition, err)
return
}
defer pc.AsyncClose()
go func(sarama.PartitionConsumer) {
for msg := range pc.Messages() {
fmt.Printf("Partition:%d Offset:%d Key:%v Value:%v", msg.Partition, msg.Offset, msg.Key, msg.Value)
}
}(pc)
}
}
The return of the code is
[0]
-1
But actually i can get the message through kafka-console-consumer.
I believe you are not wating for the messages to come...
here is the list fo issues you have in your code:
defer pc.AsyncClose() will trigger on function exit, not scope exit.
goroutine is launching into nowhere... nothing blocks or wait for the results to come.
go func(sarama.PartitionConsumer) {
for msg := range pc.Messages() {
fmt.Printf("Partition:%d Offset:%d Key:%v Value:%v", msg.Partition, msg.Offset, msg.Key, msg.Value)
}
}(pc)
not passing argument to goroutine. go func(sarama.PartitionConsumer) { this is only type. go func(pc sarama.PartitionConsumer) {.
Remove goroutine, and just check the consumer channel if you want to make hello world example.
I'm implementing a TCP server application that accepts incoming TCP connections in an infinite loop.
I'm trying to use Context throughout the application to allow shutting down, which is generally working great.
The one thing I'm struggling with is cancelling a net.Listener that is waiting on Accept(). I'm using a ListenConfig which, I believe, has the advantage of taking a Context when then creating a Listener. However, cancelling this Context does not have the intended effect of aborting the Accept call.
Here's a small app that demonstrates the same problem:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
lc := net.ListenConfig{}
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(2*time.Second)
fmt.Println("cancelling context...")
cancel()
}()
ln, err := lc.Listen(ctx, "tcp", ":9801")
if err != nil {
fmt.Println("error creating listener:", err)
} else {
fmt.Println("listen returned without error")
defer ln.Close()
}
conn, err := ln.Accept()
if err != nil {
fmt.Println("accept returned error:", err)
} else {
fmt.Println("accept returned without error")
defer conn.Close()
}
}
I expect that, if no clients connect, when the Context is cancelled 2 seconds after startup, the Accept() should abort. However, it just sits there until you Ctrl-C out.
Is my expectation wrong? If so, what is the point of the Context passed to ListenConfig.Listen()?
Is there another way to achieve the same goal?
I believe you should be closing the listener when your timeout runs out. Then, when Accept returns an error, check that it's intentional (e.g. the timeout elapsed).
This blog post shows how to do a safe shutdown of a TCP server without a context. The interesting part of the code is:
type Server struct {
listener net.Listener
quit chan interface{}
wg sync.WaitGroup
}
func NewServer(addr string) *Server {
s := &Server{
quit: make(chan interface{}),
}
l, err := net.Listen("tcp", addr)
if err != nil {
log.Fatal(err)
}
s.listener = l
s.wg.Add(1)
go s.serve()
return s
}
func (s *Server) Stop() {
close(s.quit)
s.listener.Close()
s.wg.Wait()
}
func (s *Server) serve() {
defer s.wg.Done()
for {
conn, err := s.listener.Accept()
if err != nil {
select {
case <-s.quit:
return
default:
log.Println("accept error", err)
}
} else {
s.wg.Add(1)
go func() {
s.handleConection(conn)
s.wg.Done()
}()
}
}
}
func (s *Server) handleConection(conn net.Conn) {
defer conn.Close()
buf := make([]byte, 2048)
for {
n, err := conn.Read(buf)
if err != nil && err != io.EOF {
log.Println("read error", err)
return
}
if n == 0 {
return
}
log.Printf("received from %v: %s", conn.RemoteAddr(), string(buf[:n]))
}
}
In your case you should call Stop when the context runs out.
If you look at the source code of TCPConn.Accept, you'll see it basically calls the underlying socket accept, and the context is not piped through there. But Accept is simple to cancel by closing the listener, so piping the context all the way isn't strictly necessary.
All,
I'm writing a program involving tcp traffic that has several points of failure, and
I'd like to be able to exit out of a goroutine smoothly in an error condition without incurring coding overhead.
Here's some pseudocode:
func main() {
l, err := net.Listen(CONN_TYPE, CONN_HOST+":"+ CONN_PORT)
for {
// Listen for an incoming connection.
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err.Error())
os.Exit(1)
}
done_flag := make(chan bool, 1)
// Handle connections in a new goroutine.
go func() {
conn.Write([]byte("string1\n"))
conn.Write([]byte("string2\n"))
...
}()
}
}
Now, what I'm trying to avoid is the following code with the connection statements, where I wrap the code in error handling inside the goroutine (something like the following):
go func() {
if (_err := _send_ack(conn, "string1\n"); _err != nil {
done_flag <- true
}
if (_err := _send_ack(conn, "string2\n"); _err != nil {
done_flag <- true
}
}()
Instead, if there was a connection issue, I'd rather short circuit the whole thing and just exit the goroutine with an error right then and there - and I'd rather not have to worry about how I structure the code. I could perhaps, further wrap _send_ack and send the channel as a function parameter - but that gets iffy if the program gets to be highly hierarchical. For example, I might have a goroutine composed of several funcs, each of which handles a different tcp conversation - and I don't want to litter my subroutines with an extra channel parameter to propogate the channel up and down the call stack just in case I have to set a done flag. Plus there is the question of what happens to the goroutine after the done flag is being set and how to handle it in the caller.
If I was working in python, or perl, or C++, i'd throw an exception which has attached to it a stack trace where the error occurred and then process this error in the caller. But since golang doesn't have exceptions, I was hoping for a way to just stop the goroutine cold without actually exiting the main program - ie: set a channel to have the relevant error and then just stop execution at that point.
I see the panic function, but i'm not sure of the side effects of this. Can you panic() out of a goroutine without affecting the main program, or is there a way to intelligently short-circuit a goroutine without side effects, perhaps returning back something akin to an exception, with stack trace and error? Or what is the suggested way to cleanly error handle a hierarchical program like this?
Thanks much for any help - I'm new to golang and it probably shows.
Ed
golang suggests using explicit error instead of using implicit exception.
// for code simplicity
func doSendACKImpl(conn net.Conn) error {
if err := _send_ack(conn, "string1\n"); err != nil {
return err
}
if err := _send_ack(conn, "string2\n"); err != nil {
return err
}
return nil
}
func main() {
l, err := net.Listen(CONN_TYPE, CONN_HOST+":"+ CONN_PORT)
for {
// Listen for an incoming connection.
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err.Error())
os.Exit(1)
}
// can change to self defined ResponseType, here use error for demo
workRes := make(chan error, 1)
go func() {
// return write back to channel
workRes <- doSendACKImpl(conn)
}()
select {
// read result back
case resError := <-workRes:
fmt.Printf("meet error %s", resError)
}
}
}
for more concurrent ability, use more channel buffer size, and move the processing result handler into another goroutine
func main() {
l, _ := net.Listen(CONN_TYPE, CONN_HOST+":"+CONN_PORT)
// more result buffer size
const workSize int = 100
// can change to self defined ResponseType, here use error for demo
workResBuffer := make(chan error, workSize)
// goroutine collect result
go func() {
// get all result from worker responses
for resError := range workResBuffer {
fmt.Printf("meet error %s", resError)
}
}()
for {
// Listen for an incoming connection.
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err.Error())
os.Exit(1)
}
// TODO: limit the goroutine number
go func() {
// return write back to channel
workResBuffer <- doSendACKImpl(conn)
}()
}
}
I am using the Mux library from Gorilla Web Toolkit along with the bundled Go http server.
The problem is that in my application the HTTP server is only one component and it is required to stop and start at my discretion.
When I call http.ListenAndServe(fmt.Sprintf(":%d", service.Port()), service.router) it blocks and I cannot seem to stop the server from running.
I am aware this has been a problem in the past, is that still the case? Are there any new solutions?
Regarding graceful shutdown (introduced in Go 1.8), a bit more concrete example:
package main
import (
"context"
"io"
"log"
"net/http"
"sync"
"time"
)
func startHttpServer(wg *sync.WaitGroup) *http.Server {
srv := &http.Server{Addr: ":8080"}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "hello world\n")
})
go func() {
defer wg.Done() // let main know we are done cleaning up
// always returns error. ErrServerClosed on graceful close
if err := srv.ListenAndServe(); err != http.ErrServerClosed {
// unexpected error. port in use?
log.Fatalf("ListenAndServe(): %v", err)
}
}()
// returning reference so caller can call Shutdown()
return srv
}
func main() {
log.Printf("main: starting HTTP server")
httpServerExitDone := &sync.WaitGroup{}
httpServerExitDone.Add(1)
srv := startHttpServer(httpServerExitDone)
log.Printf("main: serving for 10 seconds")
time.Sleep(10 * time.Second)
log.Printf("main: stopping HTTP server")
// now close the server gracefully ("shutdown")
// timeout could be given with a proper context
// (in real world you shouldn't use TODO()).
if err := srv.Shutdown(context.TODO()); err != nil {
panic(err) // failure/timeout shutting down the server gracefully
}
// wait for goroutine started in startHttpServer() to stop
httpServerExitDone.Wait()
log.Printf("main: done. exiting")
}
As mentioned in yo.ian.g's answer. Go 1.8 has included this functionality in the standard lib.
Minimal example for for Go 1.8+:
server := &http.Server{Addr: ":8080", Handler: handler}
go func() {
if err := server.ListenAndServe(); err != nil {
// handle err
}
}()
// Setting up signal capturing
stop := make(chan os.Signal, 1)
signal.Notify(stop, os.Interrupt)
// Waiting for SIGINT (kill -2)
<-stop
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := server.Shutdown(ctx); err != nil {
// handle err
}
// Wait for ListenAndServe goroutine to close.
You can kill the server gracefully using kill -2 <pid>
Original Answer - Pre Go 1.8 :
Building on Uvelichitel's answer.
You can create your own version of ListenAndServe which returns an io.Closer and does not block.
func ListenAndServeWithClose(addr string, handler http.Handler) (io.Closer,error) {
var (
listener net.Listener
srvCloser io.Closer
err error
)
srv := &http.Server{Addr: addr, Handler: handler}
if addr == "" {
addr = ":http"
}
listener, err = net.Listen("tcp", addr)
if err != nil {
return nil, err
}
go func() {
err := srv.Serve(tcpKeepAliveListener{listener.(*net.TCPListener)})
if err != nil {
log.Println("HTTP Server Error - ", err)
}
}()
srvCloser = listener
return srvCloser, nil
}
Full code available here.
The HTTP Server will close with the error
accept tcp [::]:8080: use of closed network connection
Go 1.8 will include graceful and forceful shutdown, available via Server::Shutdown(context.Context) and Server::Close() respectively.
go func() {
httpError := srv.ListenAndServe(address, handler)
if httpError != nil {
log.Println("While serving HTTP: ", httpError)
}
}()
srv.Shutdown(context)
The relevant commit can be found here
You can construct net.Listener
l, err := net.Listen("tcp", fmt.Sprintf(":%d", service.Port()))
if err != nil {
log.Fatal(err)
}
which you can Close()
go func(){
//...
l.Close()
}()
and http.Serve() on it
http.Serve(l, service.router)
Since none of the previous answers say why you can't do it if you use http.ListenAndServe(), I went into the v1.8 http source code and here is what it says:
func ListenAndServe(addr string, handler Handler) error {
server := &Server{Addr: addr, Handler: handler}
return server.ListenAndServe()
}
As you can see the http.ListenAndServe function does not return the server variable. This means you cannot get to 'server' to use the Shutdown command. Therefore, you need to create your own 'server' instance instead of using this function for the graceful shutdown to be implemented.
You can close the server by closing its context.
type ServeReqs func(ctx context.Context, cfg Config, deps ReqHandlersDependencies) error
var ServeReqsImpl = func(ctx context.Context, cfg Config, deps ReqHandlersDependencies) error {
http.Handle(pingRoute, decorateHttpRes(pingHandlerImpl(deps.pingRouteResponseMessage), addJsonHeader()))
server := &http.Server{Addr: fmt.Sprintf(":%d", cfg.port), Handler: nil}
go func() {
<-ctx.Done()
fmt.Println("Shutting down the HTTP server...")
server.Shutdown(ctx)
}()
err := server.ListenAndServeTLS(
cfg.certificatePemFilePath,
cfg.certificatePemPrivKeyFilePath,
)
// Shutting down the server is not something bad ffs Go...
if err == http.ErrServerClosed {
return nil
}
return err
}
And whenever you are ready to close it, call:
ctx, closeServer := context.WithCancel(context.Background())
err := ServeReqs(ctx, etc)
closeServer()
It is possible to solve this using a context.Context using a net.ListenConfig. In my case, I didn't want to use a sync.WaitGroup or http.Server's Shutdown() call, and instead rely on a context.Context (which was closed with a signal).
import (
"context"
"http"
"net"
"net/http/pprof"
)
func myListen(ctx context.Context, cancel context.CancelFunc) error {
lc := net.ListenConfig{}
ln, err := lc.Listen(ctx, "tcp4", "127.0.0.1:6060")
if err != nil {
// wrap the err or log why the listen failed
return err
}
mux := http.NewServeMux()
mux.Handle("/debug/pprof/", pprof.Index)
mux.Handle("/debug/pprof/cmdline", pprof.CmdLine)
mux.Handle("/debug/pprof/profile", pprof.Profile)
mux.Handle("/debug/pprof/symbol", pprof.Symbol)
mux.Handle("/debug/pprof/trace", pprof.Trace)
go func() {
if err := http.Serve(l, mux); err != nil {
cancel()
// log why we shut down the context
return err
}
}()
// If you want something semi-synchronous, sleep here for a fraction of a second
return nil
}
Reproducible example when you do not want your main server to be run in a separate goroutine:
main.go:
package main
import (
"context"
"log"
"net/http"
"os"
"os/signal"
"sync"
"time"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, _ *http.Request) {
// wait for 10 seconds before sending OK
time.Sleep(10 * time.Second)
_, _ = w.Write([]byte("OK\n"))
})
server := &http.Server{Addr: ":3333", Handler: nil}
// Creating a waiting group that waits until the graceful shutdown procedure is done
var wg sync.WaitGroup
wg.Add(1)
// This goroutine is running in parallels to the main one
go func() {
// creating a channel to listen for signals, like SIGINT
stop := make(chan os.Signal, 1)
// subscribing to interruption signals
signal.Notify(stop, os.Interrupt)
// this blocks until the signal is received
<-stop
// initiating the shutdown
err := server.Shutdown(context.Background())
// can't do much here except for logging any errors
if err != nil {
log.Printf("error during shutdown: %v\n", err)
}
// notifying the main goroutine that we are done
wg.Done()
}()
log.Println("listening on port 3333...")
err := server.ListenAndServe()
if err == http.ErrServerClosed { // graceful shutdown
log.Println("commencing server shutdown...")
wg.Wait()
log.Println("server was gracefully shut down.")
} else if err != nil {
log.Printf("server error: %v\n", err)
}
}
Open two terminals. In the first run the app, in the second one run curl localhost:3333, then quickly switch to the first one and try to stop the app with CTRL+C
The output should be:
2021/03/12 13:39:49 listening on port 3333...
2021/03/12 13:39:50 user initiated a request
2021/03/12 13:39:54 commencing server shutdown...
2021/03/12 13:40:00 user request is fulfilled
2021/03/12 13:40:01 server was gracefully shut down.
There exists a module which implements (graceful) stopping of Go HTTP servers:
https://github.com/pseidemann/finish
This removes the need of the boilerplate presented in the other answers.
What I've done for such cases where the application is just the server and performing no other function is install an http.HandleFunc for a pattern like /shutdown. Something like
http.HandleFunc("/shutdown", func(w http.ResponseWriter, r *http.Request) {
if <credentials check passes> {
// - Turn on mechanism to reject incoming requests.
// - Block until "in-flight" requests complete.
// - Release resources, both internal and external.
// - Perform all other cleanup procedures thought necessary
// for this to be called a "graceful shutdown".
fmt.Fprint(w, "Goodbye!\n")
os.Exit(0)
}
})
It does not require 1.8. But if 1.8 is available, then that solution can be embedded here instead of the os.Exit(0) call if desirable, I believe.
The code to perform all of that cleanup work is left as an exercise for the reader.
Extra credit if you can say where that cleanup code might be most reasonably be placed, for I would not recommend doing it here, and how this endpoint hit should cause the invocation that code.
More extra credit if you can say where that os.exit(0) call (or whatever process exit you choose to use), given here for illustrative purposes only, would be most reasonably placed.
Yet even more extra credit if you can explain why this mechanism of HTTP server process signaling should be considered above all other such mechanisms thought workable in this case.
If RPC does not have a timeout mechanism, how do I "kill" an RPC call if it is trying to call an RPC method of a server that is closed?
You can use channels to implement a timeout pattern:
import "time"
c := make(chan error, 1)
go func() { c <- client.Call("Service", args, &result) } ()
select {
case err := <-c:
// use err and result
case <-time.After(timeoutNanoseconds):
// call timed out
}
The select will block until either client.Call returns or timeoutNanoseconds elapsed.
if you want to implement a timeout (to prevent a call from taking too long), then you'll want to change rpc.Dial for net.DialTimeout (notice they're separate packages: rpc vs net). Also be aware that the returned type isn't a client any more (as it is in the previous example); instead it is a 'connection'.
conn, err := net.DialTimeout("tcp", "localhost:8080", time.Minute)
if err != nil {
log.Fatal("dialing:", err)
}
client := rpc.NewClient(conn)
It seems the only solution for net/rpc is to close the underlying connection when you notice stuck requests. Then the client should finish pending requests with "connection broken" errors.
An alternative way is to use https://github.com/valyala/gorpc , which supports timeout RPC calls out of the box.
func (client *Client) Call(serviceMethod string, args interface{}, reply interface{}) error
Call method may block goroutine forever
Change use Go method:
func (client *Client) Go(serviceMethod string, args interface{}, reply interface{}, done chan *Call) *Call
Client example:
call := rpcClient.Go(method, args, reply, make(chan *rpc.Call, 1))
select {
case <-time.After(timeout):
log.Printf("[WARN] rpc call timeout(%v) %v => %v", timeout, rpcClient, s.RpcServer)
rpcClient.Close()
return errors.New("timeout")
case resp := <-call.Done:
if resp != nil && resp.Error != nil {
rpcClient.Close()
return resp.Error
}
Or, anno now, someone might prefer to use context instead. This also takes care of returning a proper error when timed out. (context.DeadlineExceeded)
import (
"context"
"log"
"net/rpc"
)
type Client struct {
*rpc.Client
}
// CallEx is a context aware wrapper around rpc's Client.Call()
func (c *client) CallEx(ctx context.Context, serviceMethod string, args interface{}, reply interface{}) error {
ec := make(chan error, 1)
go func() {
ec <- c.Call(serviceMethod, args, reply)
}()
select {
case err := <-ec:
return err
case <-ctx.Done():
return ctx.Err()
}
}
Invoke this with a Deadlined context:
type Args struct {
A, B int
}
func main(){
rpc, err := rpc.DialHTTP("tcp", "host")
if err != nil {
t.Fatal(err)
}
c := client{rpc}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var i int
if err := c.CallEx(ctx, "Calc.Multiply", Args{2, 2}, &i); err != nil {
log.Fatal(err)
}
}