I usually find my way with Reader and Writer in Golang but I came to a situation new to me.
I am using "golang.org/x/net/html" Render. It outputs to a Writer w. I want to use that output and create a new request from that. NewRequest uses a Reader r.
err := html.Render(w, msg)
...
req, err := http.NewRequest("Post", url, r)
io.Copy(w, r)
My question is "what is the best/ideomatic solution for binding the two calls using w and r?". I could not find an example for a similar situation on the web. I am thinking about creating both Reader and Writer and using io.Copy(w, r) on them. I am not sure since this appears a little complicated for something that apparently is used often.
A simple approach is to use a bytes.Buffer:
var buf bytes.Buffer
err := html.Render(&buf, msg)
...
req, err := http.NewRequest("POST", url, &buf)
This buffers the entire request in memory. An alternate approach that does not buffer everything in memory is to use io.Pipe. This approach is more complicated because it introduces concurrency in the program. Also, the http client starts to write the request to the wire before possible errors are detected in Render.
r, w := io.Pipe()
go func() {
w.CloseWithError(html.Render(w, msg))
}()
req, err := http.NewRequest("POST", url, r)
Related
This is a heavily edited post.
I’m working on a client/server setup to practice golang where I use gocv to grab an image from my webcam and process it on a remote server using gocv again. Both processes are written in Go. What would be the best way to transmit the mat to the server process?
I have already tried to use gob like so:
func svr(){
ln, _ := net.Listen("tcp", ":8090")
conn, _ := ln.Accept()
dec := gob.NewDecoder(conn)
var p gocv.Mat
dec.Decode(&p)
gocv.IMWrite("copy.jpg", p)
conn.Close()
}
func clnt(){
time.Sleep(19999)
conn, _ := net.Dial("tcp", "localhost:8090")
encoder := gob.NewEncoder(conn)
var chip gocv.Mat
chip = gocv.IMRead("test.jpg", 0)
encoder.Encode(chip)
}
func main(){
go svr()
clnt()
time.Sleep(10000)
}
The program doesn't return any errors but doesn't create a file, either. I'm not sure what's wrong here. What am I doing wrong?
I am learning go and I have the following code which works fine:
resp, err := http.Get(url) // get the html
...
doc, err := html.Parse(resp.Body) // parse the html page
Now I want to print out the html first then do the parsing:
resp, err := http.Get(url)
...
b, err := ioutil.ReadAll(resp.Body) // this line is added, not working now...
doc, err := html.Parse(resp.Body)
I guess the reason is resp.Body is a reader, I can not call the read twice? Any idea how can I do this correctly? Copy the resp.Body?
Because the client streams the response body from the network, it's not possible to read the body twice.
Read the response to a []byte as you are already doing. Create a io.Reader on the bytes for the HTML parser using bytes.NewReader.
resp, err := http.Get(url)
...
b, err := io.ReadAll(resp.Body)
doc, err := html.Parse(bytes.NewReader(b))
Given an io.ReadCloser, from the response of an HTTP request for example, what is the most efficient way both in memory overhead and code readability to stream the response to a File?
io.Copy is undoubtedly the most efficient in terms of code; you only need to
outFile, err := os.Create(filename)
// handle err
defer outFile.Close()
_, err = io.Copy(outFile, res.Body)
// handle err
it's also likely to be pretty efficient in terms of CPU and memory as well. You can peek at the implementation of io.Copy if you want; assuming that the body doesn't implement WriteTo and the file doesn't implement ReadFrom (a quick glance says that they don't), Copy will copy chunks of up to 32kB at a time. A bigger chunk would probably use a bit less CPU but more memory; the value they picked seems like a good tradeoff.
Another option is File.ReadFrom:
package main
import (
"net/http"
"os"
)
func main() {
r, e := http.Get("https://stackoverflow.com")
if e != nil {
panic(e)
}
defer r.Body.Close()
f, e := os.Create("index.html")
if e != nil {
panic(e)
}
defer f.Close()
f.ReadFrom(r.Body)
}
https://golang.org/pkg/os#File.ReadFrom
I am building a Go application that takes an http.Response object and saves it (response headers and body) to a redis hash. When the application receives an http.Response.Body that is not gzipped, I want to gzip it before saving it to the cache.
My confusion stems from my inability to make clear sense of Go's io interfaces, and how to negotiate between http.Response.Body's io.ReadCloser and the gzip Writer. I imagine there is an elegant, streaming solution here, but I can't quite get it to work.
If you've already determined the body is uncompressed, and if you need a []byte of the compressed data (instead of for example already having an io.Writer you could write to, e.g. if you wanted to save the body to a file then you'd want to stream into the file not into a buffer) then something like this should work:
func getCompressedBody(r *http.Response) ([]byte, error) {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
if _, err := io.Copy(gz, r.Body); err != nil {
return nil, err
}
err := gz.Close()
return buf.Bytes(), err
}
(this is just an example and would probably be in-line instead of as a function; if you wanted it as a fuction then it should probably take an io.Reader instead of an *http.Response).
I have a function that makes a call to an external API using a Go http.Client, parses the result, and uses the result in the template executed afterwards. Occasionally, the external API will respond slowly (~20s), and the template execution will fail citing "i/o timeout", or more specifically,
template: :1:0: executing "page.html" at <"\n\t\t\t\t\t\t\t\t\...>: write tcp 127.0.0.1:35107: i/o timeout
This always coincides with a slow API response, but there is always a valid response in the JSON object, so the http.Client is receiving a proper response. I am just wondering if anyone could point me towards what could be causing the i/o timeout in the ExecuteTemplate call.
I have tried ResponseHeaderTimeout and DisableKeepAlives in the client transport (both with and without those options) to no avail. I've also tried setting the request's auto-close value to true to no avail. A stripped-down version of the template generation code is below:
func viewPage(w http.ResponseWriter, r *http.Request) {
tmpl := pageTemplate{}
duration, _ := time.ParseDuration("120s")
tr := &http.Transport{
ResponseHeaderTimeout: duration,
DisableKeepAlives: true,
}
client := &http.Client{Transport: tr}
req, _ := http.NewRequest("GET", "http://example.com/some_function", nil)
req.Close = true
resp, _ := client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
var res api_response // some struct that matches the JSON response
err = json.Unmarshal(body, &res)
t, _ := template.New("page.html")
err = t.ExecuteTemplate(w, "page.html", tmpl)
}
The timeout on this line:
err = t.ExecuteTemplate(w, "page.html", tmpl)
means that the outgoing response is timing out when being written into, so nothing you change in the locally created client should affect it. It also does make sense that a slow response from that client increases the chance of the timeout on w, since the deadline is set when the response is created, before your handler is called, so a slow activity from your handler will increase the chances of a timeout.
There's no write timeout on the http.Server instance used by http.ListenAndServe, so you must be setting the Server.WriteTimeout field explicitly on the created server.
As a side note, there are errors being ignored in that handler, which is a strongly discouraged practice.