os.File is a ReadCloser, and the defer fd.Close() return a error: file is already closed. My question is that will Request.Body be closed no matter when request finished?
If I Assign a ReadCloser to http.Request.Body, do I need to close the ReadCloser myself?
package main
import (
"log"
"net/http"
"os"
)
func main() {
fd, err := os.Open("test")
if err != nil {
log.Fatal(err)
}
// return err: file is already closed
defer fd.Close()
//req, err := http.NewRequest("PUT", "http://localhost:8080/test.txt", fd)
if err != nil {
log.Fatal(err)
}
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
}
Quoting from Client.Do():
The request Body, if non-nil, will be closed by the underlying Transport, even on errors.
So yes, do you not need to close it. Some ReadCloser has an idempotent Close() method which means closing it multiple times does no harm, but as you experienced, there are cases when calling Close() multiple times results in an error. So when using it for the request body, do not close it.
Related
I have the following code in GoLang
package main
import (
"bytes"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
var transport http.RoundTripper = &http.Transport{
DisableKeepAlives: false,
}
client := &http.Client{Timeout: 60 * time.Second, Transport: transport}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
values := map[string]string{"foo": "baz"}
jsonData, err := json.Marshal(values)
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer(jsonData))
if err != nil {
log.Fatalf("Error Occurred. %+v", err)
}
resp, err:= client.Do(req)
if err != nil {
defer resp.Body.Close()
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
// c should be re-used for further calls
c := httpClient()
for i := 1; i <= 60; i++ {
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
response = nil
time.Sleep(time.Millisecond * 1000)
}
}
When executed, it keeps the memory size increasing and the growth goes to as much as 90mb in one hour. is the gc not working properly. Even though i am using same httpclient for multiple requests but it still looks like theres something thats increasing the size of memory footprint.
I advice you to use tools like pprof, these are very useful at troubleshooting precisely this kind of issues.
You have set DisableKeepAlives field to false, which means that it will keep open connections even after the requests have been made, leading to further memory leaks. You should also call defer resp.Body.Close() after calling ioutil.ReadAll(resp.Body). This is precisely the purpose of the defer keyword - preventing memory leaks. GC does not mean absolute memory safety.
Also, outside of main avoid using log.Fatal. Use leveled logger, like zap or zerolog instead, since log.Fatal calls os.Exit(1) with an immediate effect, which means your defer statements will take no effect, or call plain panic. See Should a Go package ever use log.Fatal and when?
I'm trying to read and write some commands from a TCP client. I want to close a connection after the last function has been executed but for some reason, it seems like the server disconnects the connection in the middle of the function even when explicitly placed afterward.
package main
import (
"bufio"
"fmt"
"io"
"log"
"net"
"strconv"
"strings"
"time"
)
func main() {
listener, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
if err != nil {
log.Print(err)
}
go handleConn(conn)
conn.Close()
}
}
func handleConn(someconnection net.Conn) {
func1(someconnection)
func2(someconnection) //connection drops in the middle of executing this part
}
func func2(someconnection net.Conn) {
//send message(a string)
_, err := io.WriteString(someconnection, dosomething)
if err != nil {
log.Fatal(err)
}
//await reply
//send another message
_, err = io.WriteString(someconnection, dosomething)
if err != nil {
log.Fatal(err)
}
//await reply
//send another message, connection tends to close somewhere here
_, err = io.WriteString(someconnection, dosomething)
if err != nil {
log.Fatal(err)
}
//await,send
_, err = io.WriteString(someconnection, do something)
if err != nil {
log.Fatal(err)
}
//await, read and print message
c := bufio.NewReader(someconnection)
buff1 := make([]byte, maxclientmessagelength)
buff1, err = c.ReadBytes(delimiter)
fmt.Printf("\n%s\n", buff1)
_, err = io.WriteString(someconnection, dosomething)
if err != nil {
log.Fatal(err)
}
}
That means the client trying to communicate backward simply isn't able to communicate but the program runs to the end.
Update 1:
Made some progress by placing the deferred close statement to when the connection was first acquired.
func main() {
listener, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
if err != nil {
log.Print(err)
}
defer conn.Close()
go handleConn(conn)
}}
Now it doesn't necessarily close within the second I hope it to close but at least it now runs all the way through.
Goroutines are asynchronous so after calling handleConn here:
go handleConn(conn)
conn.Close()
the main function continues to execute and closes the connection.
Try just calling the handleConn function regularly (without the go).
The conn.Close needs to de done AFTER handleConn has done its work. You could communicate the back to the main thread using channels but that would be too complex (and also block execution of main thread). This is how it should be done
func main() {
listener, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
if err != nil {
log.Print(err)
}
go handleConn(conn)
// REMOVE BELOW LINE
// conn.Close()
}
}
Add conn.Close inside handleConn
func handleConn(someconnection net.Conn) {
// ADD BELOW LINE
defer someconnection.Close()
func1(someconnection)
func2(someconnection)
}
This makes sure conn.Close is called AFTER func1 and func2 are done executing
I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}
I use https://github.com/NeowayLabs/wabbit/
When amqp.Channel is closing after wrong try channel.Consume, we have a not listened chan and function is not responding.
My code:
package main
import (
"fmt"
"github.com/NeowayLabs/wabbit"
"github.com/NeowayLabs/wabbit/amqptest"
"github.com/NeowayLabs/wabbit/amqptest/server"
)
func someFunc(amqpURL string) error {
conn, err := amqptest.Dial(amqpURL)
defer conn.Close()
channel, err := conn.Channel()
defer channel.Close()
consumer, err := channel.Consume(
"queue",
"consumer",
wabbit.Option{},
)
if err != nil {
return err // err = "Unknown queue 'queue'", but we never response it
}
fmt.Println(<-consumer)
return nil
}
func main() {
amqpURL := "127.0.0.1:32773"
fakeServer := server.NewServer(amqpURL)
err := fakeServer.Start()
defer fakeServer.Stop()
err = someFunc(amqpURL)
if err != nil {
panic(err)
}
fmt.Println("Happy end")
}
someFunc never responding with error, but I want to handle consumer errors.
someFunc never responds with an error because it gets hung up in the defer code.
When someFunc gets to the return err line, then it tries to run the defer statements that you set up in the beginning of the function. The first one that it tries is defer channel.Close().
The problem seems to be with this block in the wabbit library: https://github.com/NeowayLabs/wabbit/blob/d8bc549279ecd80204a8a83a868a14fdd81d1a1b/amqptest/server/channel.go#L315-L317
I think, although I am not sure, that writing to the consumer.done channel is a blocking operation because the channel is not buffered and does not have a receiver. See this: https://gobyexample.com/non-blocking-channel-operations for more information.
I commented that block of code out when running this locally and found that the rest of the code ran as you expected that it would, finally resulting in a panic: Unknown queue 'queue'
I wrote a little web crawler and had known that the Response is a zip file.
In my limited experience with golang programing, I only know how to unzip a existing file.
Can I unzip the Response.Body in memory without saving it in hard disk in advance?
Updating answer for handling Zip file response body in-memory.
Note: Ensure you have sufficient memory for handling zip file.
package main
import (
"archive/zip"
"bytes"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
resp, err := http.Get("zip file url")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
zipReader, err := zip.NewReader(bytes.NewReader(body), int64(len(body)))
if err != nil {
log.Fatal(err)
}
// Read all the files from zip archive
for _, zipFile := range zipReader.File {
fmt.Println("Reading file:", zipFile.Name)
unzippedFileBytes, err := readZipFile(zipFile)
if err != nil {
log.Println(err)
continue
}
_ = unzippedFileBytes // this is unzipped file bytes
}
}
func readZipFile(zf *zip.File) ([]byte, error) {
f, err := zf.Open()
if err != nil {
return nil, err
}
defer f.Close()
return ioutil.ReadAll(f)
}
By default Go HTTP client handles Gzip response automatically. So do typical read and close of response body.
However there is a catch in it.
// Reference https://github.com/golang/go/blob/master/src/net/http/transport.go
//
// DisableCompression, if true, prevents the Transport from
// requesting compression with an "Accept-Encoding: gzip"
// request header when the Request contains no existing
// Accept-Encoding value. If the Transport requests gzip on
// its own and gets a gzipped response, it's transparently
// decoded in the Response.Body. However, if the user
// explicitly requested gzip it is not automatically
// uncompressed.
DisableCompression bool
What it means is; If you add a header Accept-Encoding: gzip manually in the request then you have to handle Gzip response body by yourself.
For Example -
reader, err := gzip.NewReader(resp.Body)
if err != nil {
log.Fatal(err)
}
defer reader.Close()
body, err := ioutil.ReadAll(reader)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(body))