net/http: http: ContentLength=222 with Body length 0 - go

Question

I'm trying to retry a request if there is a connection/proxy error. For some reasons I keep getting this error which doesn't seem to recover regardless the attepts to retry the request:
Post https://m.somewebsite.co.uk/api/di/34433: http: ContentLength=222 with Body length 0
Am I doing something wrong? My first suspicion is that the http.Request is consumed somehow so on the next attempts it's no longer good. Should I manage a copy?
func Post(URL string, form url.Values, cl *http.Client) ([]byte, error) {
req, err := http.NewRequest("POST", URL, strings.NewReader(form.Encode()))
if err != nil {
log.Error(err)
return nil, err
}
req.Header.Set("User-Agent", ua)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rsp, err := do(cl, req)
if err != nil {
return nil, err
}
defer rsp.Body.Close()
b, err := ioutil.ReadAll(rsp.Body)
if err != nil {
log.Error(err)
return nil, err
}
return b, nil
}
func do(cl *http.Client, req *http.Request)(*http.Response, error){
rsp, err := cl.Do(req)
for i := 0; IsErrProxy(err); i++ {
log.Errorf("Proxy is slow or down ")
time.Sleep(6 * time.Second)
5t rsp, err = cl.Do(&ncp)
if err == nil{
return rsp, nil
}
if i > 10 {
return nil, fmt.Errorf("after %v tries error: %v", i, err)
}
}
return rsp, err
}

Answer 1

The problem is that the request body is read to the end on the first call to Do(). On subsequent calls to Do(), no data is read from the response body.
The fix is to move the creation of the body reader inside the for loop. This requires that the request also be created inside the for loop.
func Post(URL string, form url.Values, cl *http.Client) ([]byte, error) {
body := form.Encode()
for i := 0; i < 10; i++ {
req, err := http.NewRequest("POST", URL, strings.NewReader(body))
if err != nil {
log.Error(err)
return nil, err
}
req.Header.Set("User-Agent", ua)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rsp, err := cl.Do(req)
if err == nil {
defer rsp.Body.Close()
b, err := ioutil.ReadAll(rsp.Body)
if err != nil {
log.Error(err)
return nil, err
}
return b, nil
}
if !IsErrorProxy(err) {
return nil, err
}
log.Errorf("Proxy is slow or down ")
time.Sleep(6 * time.Second)
}
return nil, fmt.Errorf("after 10 tries error: %v", err)
}

Answer 2

Let's see what does http.NewRequest do
rc, ok := body.(io.ReadCloser)
if !ok && body != nil {
rc = ioutil.NopCloser(body)
}
// The host's colon:port should be normalized. See Issue 14836.
u.Host = removeEmptyPort(u.Host)
req := &Request{
Method: method,
URL: u,
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(Header),
Body: rc,
Host: u.Host,
}
body is type of io.Reader, and convert to io.ReaderCloser by ioutil.NopCloser. As #Cerise Limón said, the Request.Body had been read and the stream is closer, so when you Do() again, the Body Length is 0.
So, we could reset the Request.Body before invoke Do.
func Post(URL string, form url.Values, cl *http.Client) ([]byte, error) {
requestBodyString := form.Encode()
req, err := http.NewRequest("POST", URL, strings.NewReader(requestBodyString))
// ...
rsp, err := do(cl, req, requestBodyString)
//...
return b, nil
}
func do(cl *http.Client, req *http.Request, requestBodyString string)(*http.Response, error){
rsp, err := cl.Do(req)
for i := 0; IsErrProxy(err); i++ {
log.Errorf("Proxy is slow or down ")
time.Sleep(6 * time.Second)
// reset Request.Body
req.Body = ioutil.NopCloser(strings.NewReader(requestBodyString))
rsp, err = cl.Do(&req)
if err == nil{
return rsp, nil
}
if i > 10 {
return nil, fmt.Errorf("after %v tries error: %v", i, err)
}
}
return rsp, err
}

Answer 3

I found a way to do this without re-creating the request every time. here's a sample code, which is a very slight modification of #Cerise Limón and is similar to #Rambo's code in that it creates the request only once:
func Post(URL string, data *bytes.Buffer, cl *http.Client) ([]byte, error) {
var err error
req, err := http.NewRequest("POST", URL, ioutil.NopCloser(data))
if err != nil {
log.Errorf("Unable to create the request: %v", err)
return nil, err
}
req.Header.Set("User-Agent", ua)
for i := 0; i < 10; i++ {
rsp, err := cl.Do(req)
if err == nil && rsp.StatusCode == 200 {
defer rsp.Body.Close()
b, err := ioutil.ReadAll(rsp.Body)
if err != nil {
fmt.Printf("Error: %v", err)
return nil, err
}
return b, nil
}
log.Errorf("Proxy is slow or down %v", err)
time.Sleep(1 * time.Second)
}
return nil, fmt.Errorf("after 10 tries error: %v", err)
}
func main() {
client := http.Client{}
data := []byte{0, 1, 2, 3, 4}
Post("http://server/my/api/resource/", bytes.NewBuffer(data), &client)
}