......
resp, err := httplib.Get(url)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
......
Is it necessary to close the response body every time?
Quoting from the official documentation of the http package:
The client must close the response body when finished with it
Contrary to the top-voted answer: yes, it is necessary to close resp.Body whether you consume it or not.
This is a good question, and the docs are very misleading here. In this thread of the official Go forums, the diagnosis and conclusion -- which I have experienced for myself -- is:
I was leading to leaking open files on the server, so I confirm, you MUST close the body, even if you don't read it
What happens if we do not close the response body ?
It is a resource leak. It can remain open and client connection will not be
reused.
It is recommended to close immediately after checking the error.
client := http.DefaultClient
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
From the http.Client documentation:
If the returned error is nil, the Response will contain a non-nil Body
which the user is expected to close. If the Body is not both read to
EOF and closed, the Client's underlying RoundTripper (typically
Transport) may not be able to re-use a persistent TCP connection to
the server for a subsequent "keep-alive" request.
The request Body, if non-nil, will be closed by the underlying
Transport, even on errors.
What is RoundTripper ?
It is a Transport specifies the mechanism by which individual
HTTP requests are made.If nil, DefaultTransport is used.
Related
I'm experimenting with Protobuf and gRPC and was going through the Go basics tutorial: https://www.grpc.io/docs/languages/go/basics/
From what I could see in the documentation about the ClientConn it seems that it is safe to use the same ClientConn concurrently. But I can't anything about the stub/client that one creates...
Say I was receiving data via HTTP POST requests and I wanted to forward that data via gRPC as protobuf and then respond to the HTTP request. Should I create a client/stub every time I process a HTTP request, or can I create the client/stub when setting up the HTTP server and pass in the client/stub when setting up HTTP Handlers?
Example: I assume this is okay
var opts []grpc.DialOption
conn, err := grpc.Dial("127.0.0.1:1234", opts...)
if err != nil {
log.Panic(err)
}
defer conn.Close()
grpcService := service.Service{GRPC: conn}
http.HandleFunc("/", util.ValidateRequest(grpcService.HandleRoot))
// listen to port
http.ListenAndServe(fmt.Sprintf("%s:%d", viper.GetString("server.address"), viper.GetInt32("server.port")), nil)
but what about this
var opts []grpc.DialOption
conn, err := grpc.Dial("127.0.0.1:1234", opts...)
if err != nil {
log.Panic(err)
}
defer conn.Close()
client := pb.NewEventShipperClient(conn)
grpcService := service.Service{GRPC: conn, Client: client}
http.HandleFunc("/", util.ValidateRequest(grpcService.HandleRoot))
// listen to port
http.ListenAndServe(fmt.Sprintf("%s:%d", viper.GetString("server.address"), viper.GetInt32("server.port")), nil)
Thread safety ("is it safe to run this concurrently") is covered in the Generated-code reference:
Thread-safety: note that client-side RPC invocations and server-side RPC handlers are thread-safe and are meant to be run on concurrent goroutines. But also note that for individual streams, incoming and outgoing data is bi-directional but serial; so e.g. individual streams do not support concurrent reads or concurrent writes (but reads are safely concurrent with writes).
So, yes, methods on the struct returned by pb.NewEventShipperClient(conn) can be called concurrently.
You can also check this yourself by looking at the implementation of pb.NewEventShipperClient. Currently (I guess this may change) the only member variable will be an interface (cc grpc.ClientConnInterface - implemented by *ClientConn) with each method using the connection to Invoke a gRPC call.
Please note, however, that you need to be careful in your implementation of service.Service to ensure it is also threadsafe and note the caveat regarding streams.
I try to create TCP client to receive data from TCP server,
but after server sending data only I receive data one even if server send many data, and I want to receive data forever, and I don't know what is my problem,and
Client:
func main() {
tcpAddr := "localhost:3333"
conn, err := net.DialTimeout("tcp", tcpAddr, time.Second*7)
if err != nil {
log.Println(err)
}
defer conn.Close()
// conn.Write([]byte("Hello World"))
connBuf := bufio.NewReader(conn)
for {
bytes, err := connBuf.ReadBytes('\n')
if err != nil {
log.Println("Rrecv Error:", err)
}
if len(bytes) > 0 {
fmt.Println(string(bytes))
}
time.Sleep(time.Second * 2)
}
}
I'm following this example to create TCP test server
Server:
// Handles incoming requests.
func handleRequest(conn net.Conn) {
// Make a buffer to hold incoming data.
buf := make([]byte, 1024)
// Read the incoming connection into the buffer.
_, err := conn.Read(buf)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
fmt.Println(buf)
// Send a response back to person contacting us.
var msg string
fmt.Scanln(&msg)
conn.Write([]byte(msg))
// Close the connection when you're done with it.
conn.Close()
}
Read requires a Write on the other side of the connection
want to receive data forever
Then you have to send data forever. There's a for loop on the receiving end, but no looping on the sending end. The server writes its message once and closes the connection.
Server expects to get msg from client but client doesn't send it
// conn.Write([]byte("Hello World"))
That's supposed to provide the msg value to the server
_, err := conn.Read(buf)
So those two lines don't match.
Client expects a newline but server isn't sending one
fmt.Scanln expects to put each whitespace separated value into the corresponding argument. It does not capture the whitespace. So:
Only up to the first whitespace of what you type into server's stdin will be stored in msg
Newline will not be stored in msg.
But your client is doing
bytes, err := connBuf.ReadBytes('\n')
The \n never comes. The client never gets done reading that first msg.
bufio.NewScanner would be a better way to collect data from stdin, since you're likely to want to capture whitespace as well. Don't forget to append the newline to each line of text you send, because the client expects it!
Working code
I put these changes together into a working example on the playground. To get it working in that context, I had to make a few other changes too.
Running server and client in the same process
Hard coded 3 clients so the program ended in limited amount of time
Hard coded 10 receives in the client so program can end
Hard coded 3 server connections handled so program can end
Removed fmt.Scanln and have server just return the original message sent (because playground provides no stdin mechanism)
Should be enough to get you started.
I have a very strange problem with a simple HTTP Get Request in Golang.
Every request in Golang to https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook needs about 6-8 seconds (!)
If same request fired in Chrome, with Postman or with Powershell it needs less than a second.
Does somebody has a clue why this happens?
My Code:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
client := &http.Client{}
req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)
response, err := client.Do(req)
if err != nil && response == nil {
log.Fatalf("Error on request. %v", err)
}
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalf("Couldn't get response body. %v", err)
}
fmt.Print(string(body))
}
The site you are trying to access is behind the Akamai CDN:
$ dig www.alltron.ch
...
www.alltron.ch. 152 IN CNAME competec.botmanager.edgekey.net.
competec.botmanager.edgekey.net. 7052 IN CNAME e9179.f.akamaiedge.net.
e9179.f.akamaiedge.net. 162 IN A 2.20.176.40
Akamai offers its customers a detection of web clients which are not browsers so that the customers can keep bots away or slowing bots down.
As can be seen from Strange CURL issue with a particular website SSL certificate and Scraping attempts getting 403 error this kind of detection mainly cares about having a Accept-Language header, having a Connection header with the value Keep-Alive and having a User-Agent which matches Mozilla/....
This means the following code changes result in an immediate response:
req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)
req.Header.Set("Connection","Keep-Alive")
req.Header.Set("Accept-Language","en-US")
req.Header.Set("User-Agent","Mozilla/5.0")
Still, the site obviously does not like bots and you should adhere to these wishes and not stress the site too much (like doing lots of information scraping). And, the bot detection done by Akamai might change without notice, i.e. even if this code fixes the problem now it might no longer work in the future. Such changes will be especially true if many clients bypass the bot detection.
try to disable cache in your chrome and compare to golang
Snippet from WebSocket RFC:
To Start the WebSocket Closing Handshake with a status code (Section 7.4) /code/ and an optional close reason (Section 7.1.6) /reason/, an endpoint MUST send a Close control frame, as described in Section 5.5.1, whose status code is set to /code/ and whose close reason is set to /reason/. Once an endpoint has both sent and received a Close control frame, that endpoint SHOULD Close the WebSocket Connection as defined in Section 7.1.1.
I am trying to do the Close Handshake using Gorilla WebSocket package with the following code:
Server:
// Create upgrader function
conn, err := upgrader.Upgrade(w, r, nil)
// If there is an error stop everything.
if err != nil {
fmt.Println(err)
return
}
for {
// Read Messages
_, _, err := conn.ReadMessage()
// Client is programmed to send a close frame immediately...
// When reading close frame resend close frame with same
// reason and code
conn.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(1000, "woops"))
fmt.Println(err)
break
}
Client:
d := &websocket.Dialer{}
conn, _, err := d.Dial("ws://localhost:8080", nil)
if err != nil {
fmt.Println(err)
return
}
go func() {
for {
// Read Messages
_, _, err := conn.ReadMessage()
if c, k := err.(*websocket.CloseError); k {
if(c.Code == 1000) {
// Never entering since c.Code == 1005
fmt.Println(err)
break
}
}
}
}()
conn.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(1000, "woops"))
for {}
The server is reading the Close Frame as expected outputting the following:
websocket: close 1000 (normal): woops
However the client is like its stopping to read once it sends a close message. The ReadMessage continue to return error 1005. What am I doing wrong?
The server responds to a close frame with the code:
c.WriteControl(CloseMessage, []byte{}, time.Now().Add(writeWait))
This is translated to close code 1005 (no status received) by the client.
The 1000 oops close frame written by the server is not seen by the client application because the websocket connection stops reading from network after receiving the first close frame.
The client application should exit the loop when an error is returned from ReadMessage. There's no need to check for specific close codes.
for {
// Read Messages
_, _, err := conn.ReadMessage()
if err != nil {
break
}
}
Unrelated to the issue in the question, the server application should close the websocket connection after sending the close frame.
Also unrelated to the issue in the question, use select {} instead of for {} to block the main goroutine. The former simply blocks the goroutine. The latter spins using CPU time.
I have a function that makes a call to an external API using a Go http.Client, parses the result, and uses the result in the template executed afterwards. Occasionally, the external API will respond slowly (~20s), and the template execution will fail citing "i/o timeout", or more specifically,
template: :1:0: executing "page.html" at <"\n\t\t\t\t\t\t\t\t\...>: write tcp 127.0.0.1:35107: i/o timeout
This always coincides with a slow API response, but there is always a valid response in the JSON object, so the http.Client is receiving a proper response. I am just wondering if anyone could point me towards what could be causing the i/o timeout in the ExecuteTemplate call.
I have tried ResponseHeaderTimeout and DisableKeepAlives in the client transport (both with and without those options) to no avail. I've also tried setting the request's auto-close value to true to no avail. A stripped-down version of the template generation code is below:
func viewPage(w http.ResponseWriter, r *http.Request) {
tmpl := pageTemplate{}
duration, _ := time.ParseDuration("120s")
tr := &http.Transport{
ResponseHeaderTimeout: duration,
DisableKeepAlives: true,
}
client := &http.Client{Transport: tr}
req, _ := http.NewRequest("GET", "http://example.com/some_function", nil)
req.Close = true
resp, _ := client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
var res api_response // some struct that matches the JSON response
err = json.Unmarshal(body, &res)
t, _ := template.New("page.html")
err = t.ExecuteTemplate(w, "page.html", tmpl)
}
The timeout on this line:
err = t.ExecuteTemplate(w, "page.html", tmpl)
means that the outgoing response is timing out when being written into, so nothing you change in the locally created client should affect it. It also does make sense that a slow response from that client increases the chance of the timeout on w, since the deadline is set when the response is created, before your handler is called, so a slow activity from your handler will increase the chances of a timeout.
There's no write timeout on the http.Server instance used by http.ListenAndServe, so you must be setting the Server.WriteTimeout field explicitly on the created server.
As a side note, there are errors being ignored in that handler, which is a strongly discouraged practice.