golang httputil.NewSingleHostReverseProxy how to read response and modify the response? - go

I've a reverse proxy like this:
Iam using RoundTrip but this proxy server don't work correctly.
How to correctly read and modify response?
and somebody create proxy server via NewSingleHostReverseProxy.
Please Help.
package main
import (
"bytes"
"fmt"
"io/ioutil"
"net/http"
"net/http/httputil"
"net/url"
)
type transport struct {
http.RoundTripper
}
func (t *transport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
resp, err = t.RoundTripper.RoundTrip(req)
if err != nil {
return nil, err
}
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
err = resp.Body.Close()
if err != nil {
return nil, err
}
b = bytes.Replace(b, []byte("Google"), []byte("GOOGLE"), -1)
body := ioutil.NopCloser(bytes.NewReader(b))
resp.Body = body
return resp, nil
}
func sameHost(handler http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Host = r.URL.Host
handler.ServeHTTP(w, r)
})
}
func main() {
u, _ := url.Parse("http://habrahabr.ru")
reverseProxy := httputil.NewSingleHostReverseProxy(u)
reverseProxy.Transport = &transport{http.DefaultTransport}
// wrap that proxy with our sameHost function
singleHosted := sameHost(reverseProxy)
http.ListenAndServe(":3000", singleHosted)
}

When you are going to http:// for most good sites (for example your habrahabr.ru) there is a redirect to https://, so request to http will return something like 301 Moved Permanently and you will not find content that you seek for. Also, after correct to https, make sure that site does not use javascript to load content, you can easily check this by curl:
curl localhost:3000
Also use some logging to determine what's wrong.

Related

http: read on closed response body - httptest.NewServer

I am trying to get to grips with testing using the httptest.NewServer and I am hitting a roadblock.
In my code I am making a GET request to an external API and I want to write a test for this using httptest.NewServer.
Here is my code making the request (main.go):
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
)
type HTTPClient interface {
Do(req *http.Request) (*http.Response, error)
}
type NewRequest interface {
NewRequest(method string, url string, body io.Reader) (*http.Request, error)
}
var (
Client HTTPClient
)
func init() {
Client = &http.Client{}
}
func main() {
url := "https://httpbin.org/get"
GetData(url)
}
func GetData(url string) (*http.Response, error) {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatalln(err)
return nil, err
}
resp, err := Client.Do(req)
if err != nil {
log.Fatalln(err)
return nil, err
}
defer resp.Body.Close()
responseBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
return nil, err
}
fmt.Println(resp.Status)
fmt.Println(string(responseBody))
return resp, nil
}
When I run this it works fine.
Here is my test file:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
"testing"
)
func TestYourHTTPGet(t *testing.T){
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, `response from the mock server goes here`)
}))
defer ts.Close()
mockServerURL := ts.URL
resp, err := GetData(mockServerURL)
if err != nil {
fmt.Println("Error 1: ", err)
}
defer resp.Body.Close()
responseBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal("Error 2: ", err)
}
fmt.Println(resp.Status)
fmt.Println(string(responseBody))
}
When I run go test I receive the error: http: read on closed response body. If I remove defer resp.Body.Close() from main.go the test passes correctly.
I am not sure why this is happening and was hoping that someone could explain what is going on here?
As #Cerise Limón says you call resp.Body.Close() twice and then try to read closed body. To fix yor code you can remove body processing from GetData function and do it outside GetData or return the body and do not read it in test.
main.go:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
var Client = &http.Client{}
func main() {
url := "https://httpbin.org/get"
status, data, err := GetData(url)
if err != nil {
log.Fatalln(err)
}
fmt.Println(status)
fmt.Println(string(data))
}
func GetData(url string) (status string, body []byte, err error) {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return
}
resp, err := Client.Do(req)
if err != nil {
return
}
defer resp.Body.Close()
body, err = ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
return resp.Status, body, nil
}
main_test.go:
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"testing"
)
func TestYourHTTPGet(t *testing.T){
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, `response from the mock server goes here`)
}))
defer ts.Close()
mockServerURL := ts.URL
status, data, err := GetData(mockServerURL)
if err != nil {
fmt.Println("Error 1: ", err)
}
fmt.Println(status)
fmt.Println(string(data))
}
Your GetData()'s return is a pointer. You run GetData() in main.go, when retun, it will close the resp.body. And if you read it again, it cause http: read on closed response body
So if you want read the body again, you should not return *http.Response, you should clone the resp.body to return

How to simulate multiple different HTTP responses using Go's httptest?

I have created some Go functions that make HTTP GET calls to services that are out there on the internet and parse the results.
I am now working on writing test-cases for these functions.
In my test cases, I'm using the go package httptest to simulate calls to these external services. Below is my code. Error checking is purposefully removed for brevity. Here is the go-playground.
package main
import (
"fmt"
"io"
"context"
"net/http"
"net/http/httptest"
)
func handlerResponse() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"A":"B"}`))
})
}
func buildMyRequest(ctx context.Context, url string) *http.Request {
request, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
return request
}
func myPrint(response *http.Response) {
b := make([]byte, 60000)
for {
_, err := response.Body.Read(b)
if err == io.EOF {
break
}
}
fmt.Println(string(b))
}
func main() {
srv := httptest.NewServer(handlerResponse())
client := http.Client{}
myResponse1, _ := client.Do(buildMyRequest(context.Background(), srv.URL))
fmt.Println("myResponse1:")
myPrint(myResponse1)
myResponse2, _ := client.Do(buildMyRequest(context.Background(), srv.URL))
fmt.Println("myResponse2:")
myPrint(myResponse2)
}
This is the output it produces:
myResponse1:
{"A":"B"}
myResponse2:
{"A":"B"}
As you can see, I have created some dummy HTTP response data {"A":"B"} and when you send an HTTP request to srv.URL, it actually hits an ephemeral HTTP server which responds with the dummy data. Cool!
When you send the second HTTP request to srv.URL, it again responds with the same dummy data. But this is where my problem arises. I want the ephemeral HTTP server to return some different data the second time {"C":"D"} and third time {"E":"F"} it receives a request.
How can I change the first line of the main() function so that the server responds with my desired data on subsequent HTTP calls?
you could use a hack like follows ( playground : here)
package main
import (
"fmt"
"io"
"context"
"net/http"
"net/http/httptest"
"sync"
)
type responseWriter struct{
resp map[int]string
count int
lock *sync.Mutex
}
func NewResponseWriter()*responseWriter{
r := new(responseWriter)
r.lock = new(sync.Mutex)
r.resp = map[int]string{
0: `{"E":"F"}`,
1: `{"A":"B"}`,
2: `{"C":"D"}`,
}
r.count = 0
return r
}
func (r *responseWriter)GetResp()string{
r.lock.Lock()
defer r.lock.Unlock()
r.count ++
return r.resp[r.count%3]
}
func handlerResponse(rr *responseWriter) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte(rr.GetResp()))
})
}
func buildMyRequest(ctx context.Context, url string) *http.Request {
request, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
return request
}
func myPrint(response *http.Response) {
b := make([]byte, 60000)
for {
_, err := response.Body.Read(b)
if err == io.EOF {
break
}
}
fmt.Println(string(b))
}
func main() {
rr := NewResponseWriter()
srv := httptest.NewServer(handlerResponse(rr))
client := http.Client{}
myResponse1, err := client.Do(buildMyRequest(context.Background(), srv.URL))
if err != nil{
fmt.Println(err)
return
}
defer myResponse1.Body.Close()
fmt.Println("myResponse1:")
myPrint(myResponse1)
myResponse2, err := client.Do(buildMyRequest(context.Background(), srv.URL))
if err != nil{
fmt.Println(err)
return
}
defer myResponse2.Body.Close()
fmt.Println("myResponse2:")
myPrint(myResponse2)
}

Go equivalent of Python's requests.Session for making many requests with the same basic authentication?

Consider this example for making an HTTP request in Go with basic authentication:
package main
import (
"encoding/base64"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"strings"
)
var userName = "myUserName"
var password = "myPassword"
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !checkAuth(w, r) {
http.Error(w, "You're not authorized!", http.StatusUnauthorized)
return
}
w.Write([]byte("You're authorized!"))
}))
defer ts.Close()
req, err := http.NewRequest("GET", ts.URL, nil)
check(err)
req.SetBasicAuth(userName, password+"foo")
resp, err := http.DefaultClient.Do(req)
check(err)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
check(err)
fmt.Println(string(body))
}
// checkAuth checks authentication (cf. https://stackoverflow.com/questions/21936332/idiomatic-way-of-requiring-http-basic-auth-in-go/21937924#21937924)
func checkAuth(w http.ResponseWriter, r *http.Request) bool {
s := strings.SplitN(r.Header.Get("Authorization"), " ", 2)
if len(s) != 2 {
return false
}
b, err := base64.StdEncoding.DecodeString(s[1])
if err != nil {
return false
}
pair := strings.SplitN(string(b), ":", 2)
if len(pair) != 2 {
return false
}
return pair[0] == userName && pair[1] == password
}
func check(err error) {
if err != nil {
panic(err)
}
}
Note that SetBasicAuth is a method of an *http.Request, so if I want to make many requests, I would have to call this method on each request.
In Python, you can define a requests.Session like in this example (from https://requests.readthedocs.io/en/master/user/advanced/#session-objects):
s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})
# both 'x-test' and 'x-test2' are sent
s.get('https://httpbin.org/headers', headers={'x-test2': 'true'})
Is there an idiomatic way of defining the equivalent of a requests.Session in Go (preferably using the standard library)? All I can think of is defining a custom client struct with its own Do() method:
type MyClient struct {
UserName, Password string
}
func (client *MyClient) Do(req *http.Request) (*http.Response, error) {
req.SetBasicAuth(client.UserName, client.Password)
return http.DefaultClient.Do(req)
}
and invoking it in the above script like
client := MyClient{UserName: userName, Password: password}
resp, err := client.Do(req)
Would this be an idiomatic way to avoid multiple calls to SetBasicAuth()?

Unable to redirect in a golang web app. It sticks to one page

This is code snippet from a file called upload.go.
I tried a lot of ways to redirect to another pages. I want to redirect to another page when the statements in POST are completed running.
package main
import (
"fmt"
"io"
"net/http"
"os"
"text/template"
)
func upload(w http.ResponseWriter, r *http.Request) {
if r.Method == "GET" {
// GET
t, _ := template.ParseFiles("upload.gtpl")
t.Execute(w, nil)
} else if r.Method == "POST" {
// Post
file, handler, err := r.FormFile("uploadfile")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
fmt.Fprintf(w, "%v", handler.Header)
f, err := os.OpenFile("./test/"+handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Println(err)
return
}
defer f.Close()
io.Copy(f, file)
img, err := imgio.Open("./test/" + handler.Filename)
if err != nil {
panic(err)
}
inverted := effect.Invert(img)
if err := imgio.Save("filename.png", inverted, imgio.PNGEncoder()); err != nil {
panic(err)
}
fmt.Fprintf(w, "%v", handler.Header)
http.Redirect(w, r, "www.google.com", http.StatusMovedPermanently)
} else {
fmt.Println("Unknown HTTP " + r.Method + " Method")
}
}
func main() {
http.HandleFunc("/upload", upload)
http.HandleFunc("/hi", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi")
http.Redirect(w, r, "www.google.com", http.StatusMovedPermanently)
})
http.ListenAndServe(":9090", nil) // setting listening port
}
It stays on the upload page what ever I do. Can anyone help me debug this?
Your code is writing to the ResponseWriter before trying to send a redirect.
Upon the first write to the ResponseWriter, the status code (200 OK) and headers are sent, if they haven't already been sent, and then the data you passed to the writer.
If you intend to send an HTTP redirect, you can't write any response body to the ResponseWriter. From reading your code, it doesn't make much sense why you are writing to it in the first place. They look like debugging print statements, which you probably ought to send to os.Stderr or a logger instead of the web page response body.
If you need to redirect after posting a form, you need to set the status to http.StatusSeeOther (303)
For example:
http.Redirect(w, r, "/index", http.StatusSeeOther)

How can I inject a specific IP address in the test server ? Golang

I'm trying to test an application which provides information based on ip address. However I can't find how to set the Ip address manually . Any idea ?
func TestClientData(t *testing.T) {
URL := "http://home.com/hotel/lmx=100"
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
t.Fatal(err)
}
req.RemoveAddr := "0.0.0.0" ??
w := httptest.NewRecorder()
handler(w, req)
b := w.Body.String()
t.Log(b)
}
The correct line would be:
req.RemoteAddr = "0.0.0.0"
You don't need the :=. It won't work because you don't create a new variable.
Like this (on playground http://play.golang.org/p/_6Z8wTrJsE):
package main
import (
"io"
"log"
"net/http"
"net/http/httptest"
)
func handler(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "Got request from ")
io.WriteString(w, r.RemoteAddr)
}
func main() {
url := "http://home.com/hotel/lmx=100"
req, err := http.NewRequest("GET", url, nil)
if err != nil {
log.Fatal(err)
}
// can't use := here, because RemoteAddr is a field on a struct
// and not a variable
req.RemoteAddr = "127.0.0.1"
w := httptest.NewRecorder()
handler(w, req)
log.Print(w.Body.String())
}

Resources