I need to make request to some URLs. It is possible, that someone of them is unavailable. It is OK for me, and I want just ignore these URLs.
My problem is I cannot catch error, which is occurs. I get a error message:
2018/01/13 18:46:24 Get http://fakesite.com: dial tcp [::1]:8084: connectex: No connection could be made because the target machine actively refused it.
My sample code is:
package main
import (
"fmt"
"log"
"net/http"
"io/ioutil"
)
func main() {
c := make(chan string, 1)
go func() {
defer func() {
if r := recover(); r != nil {
fmt.Println("try to catch error1", r)
}
}()
resp, err := http.Get("http://fakesite.com")
if err != nil {
log.Fatal(err)
c <- ""
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
c <- ""
return
}
c <- string(body)
}()
defer func() {
if r := recover(); r != nil {
fmt.Println("try to catch error2", r)
}
}()
msg := <-c
fmt.Println(msg)
}
The problem is:
log.Fatal(err)
This function performs two things. Firstly, it logs. But this function exits too.
// Fatal is equivalent to Print() followed by a call to os.Exit(1).
func Fatal(v ...interface{}) {
std.Output(2, fmt.Sprint(v...))
os.Exit(1)
}
Related
I'm implementing a TCP server application that accepts incoming TCP connections in an infinite loop.
I'm trying to use Context throughout the application to allow shutting down, which is generally working great.
The one thing I'm struggling with is cancelling a net.Listener that is waiting on Accept(). I'm using a ListenConfig which, I believe, has the advantage of taking a Context when then creating a Listener. However, cancelling this Context does not have the intended effect of aborting the Accept call.
Here's a small app that demonstrates the same problem:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
lc := net.ListenConfig{}
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(2*time.Second)
fmt.Println("cancelling context...")
cancel()
}()
ln, err := lc.Listen(ctx, "tcp", ":9801")
if err != nil {
fmt.Println("error creating listener:", err)
} else {
fmt.Println("listen returned without error")
defer ln.Close()
}
conn, err := ln.Accept()
if err != nil {
fmt.Println("accept returned error:", err)
} else {
fmt.Println("accept returned without error")
defer conn.Close()
}
}
I expect that, if no clients connect, when the Context is cancelled 2 seconds after startup, the Accept() should abort. However, it just sits there until you Ctrl-C out.
Is my expectation wrong? If so, what is the point of the Context passed to ListenConfig.Listen()?
Is there another way to achieve the same goal?
I believe you should be closing the listener when your timeout runs out. Then, when Accept returns an error, check that it's intentional (e.g. the timeout elapsed).
This blog post shows how to do a safe shutdown of a TCP server without a context. The interesting part of the code is:
type Server struct {
listener net.Listener
quit chan interface{}
wg sync.WaitGroup
}
func NewServer(addr string) *Server {
s := &Server{
quit: make(chan interface{}),
}
l, err := net.Listen("tcp", addr)
if err != nil {
log.Fatal(err)
}
s.listener = l
s.wg.Add(1)
go s.serve()
return s
}
func (s *Server) Stop() {
close(s.quit)
s.listener.Close()
s.wg.Wait()
}
func (s *Server) serve() {
defer s.wg.Done()
for {
conn, err := s.listener.Accept()
if err != nil {
select {
case <-s.quit:
return
default:
log.Println("accept error", err)
}
} else {
s.wg.Add(1)
go func() {
s.handleConection(conn)
s.wg.Done()
}()
}
}
}
func (s *Server) handleConection(conn net.Conn) {
defer conn.Close()
buf := make([]byte, 2048)
for {
n, err := conn.Read(buf)
if err != nil && err != io.EOF {
log.Println("read error", err)
return
}
if n == 0 {
return
}
log.Printf("received from %v: %s", conn.RemoteAddr(), string(buf[:n]))
}
}
In your case you should call Stop when the context runs out.
If you look at the source code of TCPConn.Accept, you'll see it basically calls the underlying socket accept, and the context is not piped through there. But Accept is simple to cancel by closing the listener, so piping the context all the way isn't strictly necessary.
I'm trying to create a program that will connect to several servers though gorilla web-sockets. I currently have a program that will iterate over a list of server addresses and create a new goroutine that will create its own Websocket.conn and handle reading and writing.
The problem is that every time a new goroutine is created the previous goroutines are blocked and only the last one can continue. I believe this is because the gorilla websocket library is blocking each gorotutine, but I might be mistaken.
I have tried putting a timer in the server list iterator and each goroutine will work perfectly but then the moment a new goroutine is made with another address the previous goroutine is blocked.
The relevant bits of my code:
In my main.go
for _, server := range servers {
go control(ctx, server, port)
}
In control()
func control(ctx context.Context, server, port string) {
url := url.URL{
Scheme: "ws",
Host: server + ":" + port,
Path: "",
}
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
defer conn.Close()
go sendHandler(ctx, conn)
go readHandler(ctx, conn)
}
readHandler(ctx context.Context, conn *websocket.Con) {
for {
_, p, err := conn.ReadMessage(); if err != nil {
panic(err)
}
select {
case <-ctx.Done():
goto TERM
default:
// do nothing
}
}
TERM:
// do termination
}
sendHandler(ctx context.Context, conn *websocket.Con) {
for _, msg := range msges {
err = conn.WriteMessage(websocket.TextMessage, msg)
if err != nil {
panic(err)
}
}
<-ctx.Done()
}
I removed the parts where I add waitgroups and other unnecessary pieces of code.
So what I expect is for there to be 3n goroutines running (where n is the number of servers) without blocking but right now I see only 3 goroutines running which are the ones called by the last iteration of the server list.
Thanks!
EDIT 14/06/2019:
I spent some time making a small working example and in the example the bug did not occur - none of the threads blocked each other. I'm still unsure what was causing it but here is my small working example:
main.go
package main
import (
"context"
"fmt"
"os"
"time"
"os/signal"
"syscall"
"sync"
"net/url"
"github.com/gorilla/websocket"
)
func main() {
servers := []string{"5555","5556", "5557"}
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
for _, server := range servers {
wg.Add(1)
go control(server,
ctx,
&wg)
}
<-comms
cancel()
wg.Wait()
}
func control(server string, ctx context.Context, wg *sync.WaitGroup) {
fmt.Printf("Started control for %s\n", server)
url := url.URL {
Scheme: "ws",
Host: "0.0.0.0" + ":" + server,
Path: "",
}
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
defer conn.Close()
var localwg sync.WaitGroup
localwg.Add(1)
go sendHandler(ctx, conn, &localwg, server)
localwg.Add(1)
go readHandler(ctx, conn, &localwg, server)
<- ctx.Done()
localwg.Wait()
wg.Done()
return
}
func sendHandler(ctx context.Context, conn *websocket.Conn, wg *sync.WaitGroup, server string) {
for i := 0; i < 50; i++ {
err := conn.WriteMessage(websocket.TextMessage, []byte("ping"))
if err != nil {
panic(err)
}
fmt.Printf("sent msg to %s\n", server)
time.Sleep(1 * time.Second)
}
<- ctx.Done()
wg.Done()
}
func readHandler(ctx context.Context, conn *websocket.Conn, wg *sync.WaitGroup, server string) {
for {
select {
case <- ctx.Done():
wg.Done()
return
default:
_, p, err := conn.ReadMessage()
if err != nil {
wg.Done()
fmt.Println("done")
}
fmt.Printf("Got [%s] from %s\n", string(p), server)
}
}
}
I tested it with dpallot's simple-websocket-server by a server on 5555, 5556 and 5557 respectively.
This part of your code is causing the problem:
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
defer conn.Close()
go sendHandler(ctx, conn)
go readHandler(ctx, conn)
You create the connection, defer the close of it, start two other goroutines and then end the function. The function end closes the socket due to your defer.
I only want to support HTTP/2 for a new project, the client is not a browser so it's not a problem if we don't support HTTP/1.x at all.
from what I see in golang.org/x/net/http2. I can use tls.Listen and pass the net.Conn to http2.Server.ServeConn.
But I'm bit confused about how to use http2.Transport here, can anyone give me an example?
Thanks
UPDATE:
This is the server part, pretty simple, it's an echo server
package main
import (
"fmt"
"io"
"net"
"net/http"
"golang.org/x/net/http2"
)
func main() {
l, err := net.Listen("tcp4", ":1234")
panicIfNotNil(err)
s := &http2.Server{}
sopt := &http2.ServeConnOpts{
BaseConfig: &http.Server{},
Handler: http.HandlerFunc(handler),
}
for {
c, err := l.Accept()
panicIfNotNil(err)
go serve(s, sopt, c)
}
}
func serve(s *http2.Server, sopt *http2.ServeConnOpts, c net.Conn) {
defer c.Close()
s.ServeConn(c, sopt)
}
func handler(w http.ResponseWriter, r *http.Request) {
if r.ProtoMajor != 2 {
w.WriteHeader(500)
fmt.Fprintln(w, "Not HTTP/2")
return
}
f, ok := w.(http.Flusher)
if !ok {
w.WriteHeader(500)
fmt.Fprintln(w, "Not Flusher")
return
}
w.Header().Set("Content-Type", "application/octet-stream")
fmt.Fprintln(w, "Hello World, Echo Server")
buf := [1024]byte{}
for {
n, err := r.Body.Read(buf[:])
if err == io.EOF {
break
}
panicIfNotNil(err)
_, err = w.Write(buf[:n])
f.Flush()
panicIfNotNil(err)
}
}
func panicIfNotNil(err error) {
if err != nil {
panic(err)
}
}
tested with curl --http2-prior-knowledge http://127.0.0.1:1234 -d a=b -d c=d -d e=f
for the client part, I'm still trying, I will update this post again when I got something.
UPDATE:
for the sake of simplicity, I don't use TLS here
UPDATE:
This is the client part
package main
import (
"crypto/tls"
"fmt"
"io"
"net"
"net/http"
"net/url"
"time"
"golang.org/x/net/http2"
)
func main() {
t := &http2.Transport{
DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, error) {
return net.Dial(network, addr)
},
AllowHTTP: true,
}
c := &http.Client{
Transport: t,
}
pr, pw := io.Pipe()
req := &http.Request{
Method: "POST",
URL: mustUrl("http://127.0.0.1:1234/"),
Body: pr,
}
resp, err := c.Do(req)
panicIfNotNil(err)
defer resp.Body.Close()
if resp.StatusCode != 200 {
panic(fmt.Errorf("Server return non 200, %d", resp.StatusCode))
}
wchan := make(chan struct{})
go func() {
buf := [1024]byte{}
for {
n, err := resp.Body.Read(buf[:])
if err == io.EOF {
break
}
panicIfNotNil(err)
fmt.Printf("GOT DATA %s\n", string(buf[:n]))
}
close(wchan)
}()
time.Sleep(1 * time.Second)
pw.Write([]byte("hai AAA"))
time.Sleep(1 * time.Second)
pw.Write([]byte("hai BBB"))
time.Sleep(1 * time.Second)
pw.Write([]byte("hai CCC"))
time.Sleep(1 * time.Second)
pw.Write([]byte("hai CCC"))
time.Sleep(1 * time.Second)
pw.Close()
<-wchan
}
func mustUrl(s string) *url.URL {
r, err := url.Parse(s)
panicIfNotNil(err)
return r
}
func panicIfNotNil(err error) {
if err != nil {
panic(err)
}
}
but somehow it doesn't work
You can see network traffic in https://imgur.com/EJV0uGI
After looking into Wireshark more closely I found the problem, it happens because the server didn't send any header frame, so the client cannot continue with more data. Just printing into http.ResponseWriter doesn't ensure its written into the network, it gets buffered instead, so we need to explicitly flush it.
This fixes the problem:
--- main.go 2018-07-25 22:31:44.092823590 +0700
+++ main2.go 2018-07-25 22:32:50.586179879 +0700
## -43,6 +43,9 ##
return
}
w.Header().Set("Content-Type", "application/octet-stream")
+ w.WriteHeader(200)
+ f.Flush()
+
fmt.Fprintln(w, "Hello World, Echo Server")
buf := [1024]byte{}
I encountered a strange behavior of the conn.Read:
let's presume that I have a couple of functions for testing net.Conn:
package example
import (
"io"
"log"
"net"
"os"
"time"
)
func CheckConn(conn net.Conn) (net.Conn, error) {
conn.SetReadDeadline(time.Now())
var one = []byte{}
_, err := conn.Read(one)
if err != nil {
log.Println("Net err: ", err)
}
if err == io.EOF {
return conn, err
}
var zero time.Time
conn.SetReadDeadline(zero)
return conn, nil
}
func CheckConnWithTimeout(conn net.Conn) (net.Conn, error) {
ch := make(chan bool, 1)
defer func() {
ch <- true
}()
go func() {
select {
case <-ch:
case <-time.After(1 * time.Second):
log.Println("It works too long")
os.Exit(1)
}
}()
return CheckConn(conn)
}
And I want to implement tests for it, lets start with this one:
package example
import (
"io"
"net"
"testing"
)
func TestClosedConn(t *testing.T) {
server, client := net.Pipe()
client.Close()
defer server.Close()
_, err := CheckConn(server)
if err != io.EOF {
t.Errorf("Not equal:\nExpected: %v\nactual: %v", io.EOF, err)
}
}
this works pretty well, we will receive io.EOF from CheckConn function, lets add one more test:
func TestClosedConnAfterWrite(t *testing.T) {
server, client := net.Pipe()
go func() {
client.Write([]byte{0xb})
}()
client.Close()
defer server.Close()
_, err := CheckConn(server)
err = nil
if err != io.EOF {
t.Errorf("Not equal:\nExpected: %v\nactual: %v", io.EOF, err)
}
}
looks like the first test, but we wrote to the client before(?) it was closed.
And this will not pass!
conn.Read will return &errors.errorString{s:"EOF"}, instead of io.EOF, so CheckConn will return error == nil,
It looks so weird!
But let's continue the tests, now I want to check unclosed connections:
func TestActiveConn(t *testing.T) {
server, client := net.Pipe()
defer client.Close()
defer server.Close()
_, err := CheckConnWithTimeout(server)
if err != nil {
t.Errorf("Not equal:\nExpected: %v\nactual: %v", nil, err)
}
}
I think you noticed that I use the function with a timeout just because SetReadDeadline will not work in this case(I have no idea why!)
So what is going wrong in last two test cases? Is there a normal way to test the connection? Why SetReadDeadline is not working in this case?
I'm trying to create a client and a server using Go but for some reason the server reports the connection as "closed". As the code is trivial I can't think of anything wrong with my code. Any help is appreciated.
package main
import (
log "github.com/golang/glog"
"net/http"
"golang.org/x/net/websocket"
"time"
"flag"
)
type server struct {
payload chan string
}
// srv pushes the messages received via ws into srv.payload
func (srv *server) serve(ws *websocket.Conn) {
go func() {
var msg string
if err := websocket.Message.Receive(ws, &msg); err != nil {
log.Exit(err)
}
srv.payload <- msg
}()
return
}
// This example demonstrates a trivial client/ server.
func main() {
flag.Parse()
srv := server{payload: make(chan string, 10)}
http.Handle("/echo", websocket.Handler(srv.serve))
go func() {
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Errorf("ListenAndServe: " + err.Error())
}
}()
// give the server some time to start listening
time.Sleep(3 *time.Second)
//dial and test the response.
ws, err := websocket.Dial("ws://localhost:12345/echo", "", "http://localhost/?x=45")
if err != nil {
log.Exit(err)
}
ms := "test"
if err := websocket.Message.Send(ws, ms); err != nil {
log.Exit(err)
}
msg := <-srv.payload
if msg != ms{
log.Errorf("msg %v is not %v", ms)
}
}
Error
t.go:21] read tcp 127.0.0.1:12345->127.0.0.1:43135:
Edit:
After some try and error I've found that if I remove the go routine from the serve method it works but it doesn't make sense to me. Any idea why it doesn't work when websocket.Message.Receive is in a separate go routine?
package main
import (
log "github.com/golang/glog"
"net/http"
"golang.org/x/net/websocket"
"time"
"flag"
)
type server struct {
payload chan string
}
// srv pushes the messages received via ws into srv.payload
func (srv *server) serve(ws *websocket.Conn) {
var msg string
if err := websocket.Message.Receive(ws, &msg); err != nil {
log.Exit(err)
}
srv.payload <- msg
return
}
// This example demonstrates a trivial client/ server.
func main() {
flag.Parse()
srv := server{payload: make(chan string, 10)}
go func() {
http.Handle("/echo", websocket.Handler(srv.serve))
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Errorf("ListenAndServe: " + err.Error())
}
}()
// give the server some time to start listening
time.Sleep(3 *time.Second)
//dial and test the response.
ws, err := websocket.Dial("ws://localhost:12345/echo", "", "http://localhost/?x=45")
if err != nil {
log.Exit(err)
}
ms := "test"
if err := websocket.Message.Send(ws, ms); err != nil {
log.Exit(err)
}
msg := <-srv.payload
if msg != ms{
log.Errorf("msg %v is not %v", ms)
}
}
The websocket server closes the connection when the handler returns.
Removing the Go routine is the correct fix.