I am looking to make a proxy gateway in Go.
Almost done ! One thing is still missing : send the entire client response to the server request.
I've got my own HTTP handler :
func (f HttpHandlerFunc) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if rurl, err := getOriginurl(r.RequestURI); err == nil {
[...]
client := &Http.Client{}
r.URL = rurl
r.RequestURI = ""
resp, err := client.Do(r)
if err == nil {
for k, vs := range resp.Header {
for _, v := range vs {
w.Header().Set(k, v)
}
}
w.WriteHeader(resp.StatusCode)
if responseData,err := ioutil.ReadAll(resp.Body); err == nil {
w.Write(responseData)
}
}
}
}
func getOriginurl(request string) *url.URL {
{...}
// Would return an *url.URL with : http://127.0.0.1:8080/{requestURI}
}
I am looking for a way to optimize the way to parse Client response to ResponseWriter.
Actually my question would be : How to parse Response type to ResponseWriter exhaustively ?
You can use httputil.NewSingleHostReverseProxy instead of your own HTTP client logic.
httputil.NewSingleHostReverseProxy(rurl).ServeHTTP(w, r)
Related
I will like to pass a list of forwarding proxy servers for POST request
Currently i am able to do it with just single forwarding proxy
serverProxy := "http://user:password#123.45.67.89:3128"
request, error := http.NewRequest("POST", httpposturl, bytes.NewBuffer(requestJSON))
request.Header.Set("Content-Type", "application/json; charset=UTF-8")
proxyURL, _ := url.Parse(serverProxy)
proxy := http.ProxyURL(proxyURL)
transport := &http.Transport{Proxy: proxy}
client := &http.Client{Transport: transport}
what i will like to do is pass a list to url.Parse and want it to use them using round robin balancing
so something like this
serverProxy := "http://user:password#123.45.67.89:3128, http://user:password#223.45.67.89:3128"
and then it will select which of the proxy servers to use and rotate them within requests
Is this possible?
UPDATE:
I want to be able to pass the rotated proxy server like this
proxyServer := roundRobin("http://round:robin#123.45.67.89:3128, http://robin:round#223.45.67.89:3128")
fmt.Println("proxy server used", proxyServer, "\n")
transport := &http.Transport{Proxy: proxyServer}
client := &http.Client{Transport: transport}
Create a proxy function that round-robins through your proxy URLs. Use that function in your transport:
func roundRobin(urls []*url.URL) func(*http.Request) (*url.URL, error) {
var mu sync.Mutex
var i int
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % len(urls)
u := urls[i]
mu.Unlock()
return u, nil
}
}
transport := &http.Transport{Proxy: roundRobin(yourProxyURLs)}
client := &http.Client{Transport: transport}
Here's the Montage's answer with explanation:
The requirement is to forward request through proxies in round-robin fashion
Since we are using http.Client to make request we can look at http.Client documentation to see if it provide any support for forwarding request through proxy when we look at the documentation we can see that it does support passing proxy which we can pass through http.Transport type which will be passed to http.Client. http.Transport takes proxy through Proxy field which takes in func that return *url.URL and error there are existing methods like http.ProxyURL and http.ProxyFromEnvironment provided within http package that we can use to pass proxies to http.Transport but the problem with these methods is that they only take a single proxy server which does not solve our problem at hand and hence we would require to create our own function which takes in multiple proxy servers urls and round-robin between them.
If we look at one of the existing method implemention as our base for creating our own method lets go with http.ProxyURL for our case the implementation can be found here. I have copied the implementation below
func ProxyURL(fixedURL *url.URL) func(*Request) (*url.URL, error) {
return func(*Request) (*url.URL, error) {
return fixedURL, nil
}
}
we can see that its a simple closure which takes in single url and return a closure function which then intern return the url passed in as parameter. so we can take it base and create our own round-robin clouse function
func roundRobin(proxies ...string) func(*http.Request) (*url.URL, error) {
var urls []*url.URL
for _, proxy := range proxies {
u, err := url.Parse(proxy)
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
var mu sync.Mutex
var i, lenUrls int = 0, len(urls)
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % lenUrls
u := urls[i]
mu.Unlock()
return u, nil
}
}
Lets go over the roundRobin function implementation it is a variadic function which takes in proxy url(s) in string format as argument, which internally gets converted to url.URL by parsing the string using url.Parse then using the parsed url.URL to create slice of urls []*url.URL which then being used to forward request in round-robin fashion
Complete working example can be found below:
package main
import (
"fmt"
"log"
"net/url"
"net/http"
"sync"
)
func roundRobin(proxies ...string) func(*http.Request) (*url.URL, error) {
var urls []*url.URL
for _, proxy := range proxies {
u, err := url.Parse(proxy)
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
var mu sync.Mutex
var i, lenUrls int = 0, len(urls)
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % lenUrls
u := urls[i]
mu.Unlock()
return u, nil
}
}
func main() {
proxyFn := roundRobin("http://user:password#123.45.67.89:3128", "http://user:password#223.45.67.89:3128")
transport := &http.Transport{Proxy: proxyFn}
client := &http.Client{Transport: transport}
req, err := http.NewRequest("POST", "http://example.com", nil)
req.Header.Set("Content-Type", "application/json; charset=UTF-8")
resp, err := client.Do(req)
if err != nil {
fmt.Println(resp)
}
fmt.Println(proxyFn(nil))
fmt.Println(proxyFn(nil))
}
Playground
Another version
package main
import (
"fmt"
"log"
"net/http"
"net/url"
"sync"
)
func praseUrls(proxies ...string) (urls []*url.URL) {
for _, proxy := range proxies {
u, err := url.Parse(proxy)
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
return
}
func roundRobin(max int) func() int {
var i int
return func() int {
i = (i + 1) % max
return i
}
}
func proxyFn(urls []*url.URL) func(*http.Request) (*url.URL, error) {
var m sync.Mutex
fn := roundRobin(len(urls))
return func(*http.Request) (*url.URL, error) {
m.Lock()
u := urls[fn()]
m.Unlock()
return u, nil
}
}
func main() {
proxies := []string{"http://user:password#123.45.67.89:3128", "http://user:password#223.45.67.89:3128"}
urls := praseUrls(proxies...)
transport := &http.Transport{Proxy: proxyFn(urls)}
client := &http.Client{Transport: transport}
req, err := http.NewRequest("POST", "http://example.com", nil)
req.Header.Set("Content-Type", "application/json; charset=UTF-8")
resp, err := client.Do(req)
if err != nil {
fmt.Println(resp)
}
}
Playground
Note: It would be better to pass proxy urls from env variable which would help in case any proxy server changes or new are added
Here's the Montage's answer with code to parse a string.
func roundRobin(serverProxy string) func(*http.Request) (*url.URL, error) {
parts := strings.Split(serverProxy, ",")
var urls []*url.URL
for _, part := range parts {
u, err := url.Parse(strings.TrimSpace(part))
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
var mu sync.Mutex
var i int
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % len(urls)
u := urls[i]
mu.Unlock()
return u, nil
}
}
serverProxy := "http://user:password#123.45.67.89:3128, http://user:password#223.45.67.89:3128"
transport := &http.Transport{Proxy: roundRobin(serverProxy)}
client := &http.Client{Transport: transport}
As part of my first project I am creating a tiny library to send an SMS to any user. I have added the logic of waiting and retrying if it doesn't receive a positive status on first go. It's a basic HTTP call to am SMS sending service. My algorithm looks like this (comments would explain the flow of the code):
for {
//send request
resp, err := HTTPClient.Do(req)
checkOK, checkSuccessUrl, checkErr := CheckSuccessStatus(resp, err)
//if successful don't continue
if !checkOK and checkErr != nil {
err = checkErr
return resp, SUCCESS, int8(RetryMax-remain+1), err
}
remain := remain - 1
if remain == 0 {
break
}
//calculate wait time
wait := Backoff(RetryWaitMin, RetryWaitMax, RetryMax-remain, resp)
//wait for time calculated in backoff above
time.Sleep(wait)
//check the status of last call, if unsuccessful then continue the loop
if checkSuccessUrl != "" {
req, err := GetNotificationStatusCheckRequest(checkSuccessUrl)
resp, err := HTTPClient.Do(req)
checkOK, _, checkErr = CheckSuccessStatusBeforeRetry(resp, err)
if !checkOK {
if checkErr != nil {
err = checkErr
}
return resp,SUCCESS, int8(RetryMax-remain), err
}
}
}
Now I want to test this logic using any HTTP mock framework available. The best I've got is https://github.com/jarcoal/httpmock
But this one does not provide functionality to mock the response of first and second URL separately. Hence I cannot test the success in second or third retry. I can either test success in first go or failure altogether.
Is there a package out there which suits my needs of testing this particular feature? If no, How can I achieve this using current tools?
This can easily be achieved using the test server that comes in the standard library's httptest package. With a slight modification to the example contained within it you can set up functions for each of the responses you want up front by doing this:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
)
func main() {
responseCounter := 0
responses := []func(w http.ResponseWriter, r *http.Request){
func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "First response")
},
func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Second response")
},
}
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
responses[responseCounter](w, r)
responseCounter++
}))
defer ts.Close()
printBody(ts.URL)
printBody(ts.URL)
}
func printBody(url string) {
res, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
resBody, err := ioutil.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", resBody)
}
Which outputs:
First response
Second response
Executable code here:
https://play.golang.org/p/YcPe5hOSxlZ
Not sure you still need an answer, but github.com/jarcoal/httpmock provides a way to do this using ResponderFromMultipleResponses.
I'm trying to make a proxy by golang.
The origin version is written by lua, nginx like this:
location / {
keepalive_timeout 3600s;
keepalive_requests 30000;
rewrite_by_lua_file ./test.lua;
proxy_pass http://www.example.com/bd/news/home;
}
and lua file like this:
local req_params = ngx.req.get_uri_args()
local args = {
media = 24,
submedia = 46,
os = req_params.os,
osv = req_params.osv,
make = req_params.make,
model = req_params.model,
devicetype = req_params.devicetype,
conn = req_params.conn,
carrier = req_params.carrier,
sw = req_params.w,
sh = req_params.h,
}
if tonumber(req_params.os) == 1 then
args.imei = req_params.imei
args.adid = req_params.android_id
end
ngx.req.set_uri_args(args)
I try to do the same thing by golang, and my code is like this:
const newsTargetURL = "http://www.example.com/bd/news/home"
func GetNews(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only get allowed", http.StatusMethodNotAllowed)
return
}
// deal params
rq := r.URL.Query()
os := rq.Get("os")
osv := rq.Get("osv")
imei := rq.Get("imei")
androidID := rq.Get("android_id")
deviceMake := rq.Get("make")
model := rq.Get("model")
deviceType := rq.Get("devicetype")
sw := rq.Get("w")
sh := rq.Get("h")
conn := rq.Get("conn")
carrier := rq.Get("carrier")
uv := make(url.Values)
uv.Set("media", "24")
uv.Set("submedia", "46")
uv.Set("os", os)
uv.Set("osv", osv)
if os == "1" {
uv.Set("imei", imei)
uv.Set("anid", androidID)
}
uv.Set("make", deviceMake)
uv.Set("model", model)
uv.Set("sw", sw)
uv.Set("sh", sh)
uv.Set("devicetype", deviceType)
uv.Set("ip", ip)
uv.Set("ua", ua)
uv.Set("conn", conn)
uv.Set("carrier", carrier)
t := newsTargetURL + "?" + uv.Encode()
// make a director
director := func(req *http.Request) {
u, err := url.Parse(t)
if err != nil {
panic(err)
}
req.URL = u
}
// make a proxy
proxy := &httputil.ReverseProxy{Director: director}
proxy.ServeHTTP(w, r)
}
func main() {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(GetNews))
srv := &http.Server{
Addr: ":2222",
Handler: mux,
}
srv.ListenAndServe()
}
I put this go version to the same server where lua version locate, but it does not work as lua file do. I read the httputil document but found nothing that can help. What do I need to do?
I wrote together a simple proxy for GET requests. Hope this helps.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
const newsTargetURL = "http://www.example.com/bd/news/home"
func main() {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(GetNews))
srv := &http.Server{
Addr: ":2222",
Handler: mux,
}
// output error and quit if ListenAndServe fails
log.Fatal(srv.ListenAndServe())
}
func GetNews(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only get allowed", http.StatusMethodNotAllowed)
return
}
// build proxy url
urlstr := fmt.Sprintf("%s?%s", newsTargetURL, r.URL.RawQuery)
// request the proxy url
resp, err := http.Get(urlstr)
if err != nil {
http.Error(w, fmt.Sprintf("error creating request to %s", urlstr), http.StatusInternalServerError)
return
}
// make sure body gets closed when this function exits
defer resp.Body.Close()
// read entire response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
http.Error(w, "error reading response body", http.StatusInternalServerError)
return
}
// write status code and body from proxy request into the answer
w.WriteHeader(resp.StatusCode)
w.Write(body)
}
You can try it as is. It will work and show the content of example.com.
It uses a single handler GetNews for all requests. It skips all of the request parameter parsing and building by simply using r.url.RawQuery and newsTargetURL to build the new url.
Then we make a request to the new url (the main part missing in your question). From the response we read resp.StatusCode and resp.body to use in our response to the original request.
The rest is error handling.
The sample does not forward any additional information like cookies, headers, etc. That can be added as needed.
I have this unit test for a proxy i'm writing. I cannot for the life of me see why the environment get ignored and my test make a direct access to the target server, acap.
func TestHandleHTTPS(t *testing.T) {
successfulCalls := 0
proxyPassed := 0
acap := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
successfulCalls++
}))
defer acap.Close()
testproxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxyPassed++
}))
defer testproxy.Close()
os.Setenv("https_proxy", testproxy.URL)
defer os.Setenv("https_proxy", "")
client := acap.Client()
tmp := client.Transport.(*http.Transport)
tmp.Proxy = http.ProxyFromEnvironment // <--- This should make client use env vars!
req, err := http.NewRequest("GET", acap.URL, nil)
if err != nil {
t.Errorf("Unable to create request: %s", err.Error())
}
resp, err := client.Do(req)
if err != nil {
t.Errorf("Something is wrong with the test: %s", err.Error())
return
}
if resp.StatusCode != 200 {
t.Errorf("Unexpected status code: %d", resp.StatusCode)
body, _ := ioutil.ReadAll(resp.Body)
t.Errorf("Body: %s", string(body))
}
if successfulCalls == 0 {
t.Errorf("No successful call over HTTPS occurred")
}
if proxyPassed == 0 {
t.Errorf("Proxy got ignored")
}
}
The only failure i get is Proxy got ignored. I use Go v1.10, everything compiles.
Edit 1:
I do the tmp.Proxy dance because the client already have certificates and stuff configured in the Transport. I don't want to mess that up by replacing the entire Transport struct
If you take a look at doc for ProxyFromEnvironment you will find there a special case:
As a special case, if req.URL.Host is "localhost" (with or without a port number), then a nil URL and nil error will be returned.
That means that no proxy will be used. I would suggest you to use ProxyURL instead
proxyURL, _ := url.Parse(testproxy.URL)
tmp.Proxy = http.ProxyURL(proxyURL)
It will take into account your Proxy, but won't work, because you are trying to make an https call throw http proxy...
Here my use case
We have one services "foobar" which has two version legacy and version_2_of_doom (both in go)
In order to make the transition from legacy to version_2_of_doom , we would like in a first time, to have the two version alongside, and have the POST request (as there's only one POST api call in this ) received on both.
The way I see how to do it. Would be
modifying the code of legacy at the beginning of the handler, in order to duplicate the request to version_2_of_doom
func(w http.ResponseWriter, req *http.Request) {
req.URL.Host = "v2ofdoom.local:8081"
req.Host = "v2ofdoom.local:8081"
client := &http.Client{}
client.Do(req)
// legacy code
but it seems to not be as straightforward as this
it fails with http: Request.RequestURI can't be set in client requests.
Is there a well-known method to do this kind of action (i.e transfering without touching) a http.Request to an other server ?
You need to copy the values you want into a new request. Since this is very similar to what a reverse proxy does, you may want to look at what "net/http/httputil" does for ReverseProxy.
Create a new request, and copy only the parts of the request you want to send to the next server. You will also need to read and buffer the request body if you intend to use it both places:
func handler(w http.ResponseWriter, req *http.Request) {
// we need to buffer the body if we want to read it here and send it
// in the request.
body, err := ioutil.ReadAll(req.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// you can reassign the body if you need to parse it as multipart
req.Body = ioutil.NopCloser(bytes.NewReader(body))
// create a new url from the raw RequestURI sent by the client
url := fmt.Sprintf("%s://%s%s", proxyScheme, proxyHost, req.RequestURI)
proxyReq, err := http.NewRequest(req.Method, url, bytes.NewReader(body))
// We may want to filter some headers, otherwise we could just use a shallow copy
// proxyReq.Header = req.Header
proxyReq.Header = make(http.Header)
for h, val := range req.Header {
proxyReq.Header[h] = val
}
resp, err := httpClient.Do(proxyReq)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
defer resp.Body.Close()
// legacy code
}
In my experience, the easiest way to achieve this was to simply create a new request and copy all request attributes that you need into the new request object:
func(rw http.ResponseWriter, req *http.Request) {
url := req.URL
url.Host = "v2ofdoom.local:8081"
proxyReq, err := http.NewRequest(req.Method, url.String(), req.Body)
if err != nil {
// handle error
}
proxyReq.Header.Set("Host", req.Host)
proxyReq.Header.Set("X-Forwarded-For", req.RemoteAddr)
for header, values := range req.Header {
for _, value := range values {
proxyReq.Header.Add(header, value)
}
}
client := &http.Client{}
proxyRes, err := client.Do(proxyReq)
// and so on...
This approach has the benefit of not modifying the original request object (maybe your handler function or any middleware functions that are living in your stack still need the original object?).
Using original request (copy or duplicate only if original request still need):
func handler(w http.ResponseWriter, r *http.Request) {
// Step 1: rewrite URL
URL, _ := url.Parse("https://full_generic_url:123/x/y")
r.URL.Scheme = URL.Scheme
r.URL.Host = URL.Host
r.URL.Path = singleJoiningSlash(URL.Path, r.URL.Path)
r.RequestURI = ""
// Step 2: adjust Header
r.Header.Set("X-Forwarded-For", r.RemoteAddr)
// note: client should be created outside the current handler()
client := &http.Client{}
// Step 3: execute request
resp, err := client.Do(r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Step 4: copy payload to response writer
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
resp.Body.Close()
}
// copyHeader and singleJoiningSlash are copy from "/net/http/httputil/reverseproxy.go"
func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}
func singleJoiningSlash(a, b string) string {
aslash := strings.HasSuffix(a, "/")
bslash := strings.HasPrefix(b, "/")
switch {
case aslash && bslash:
return a + b[1:]
case !aslash && !bslash:
return a + "/" + b
}
return a + b
}
I've seen the accepted anwser, but I would like to say that I dont like this. I've used this code for months with it working, but after some time you encounter requests that break (POST requests in my case). My preferred solution is the following:
r.URL.Host = "example.com"
r.RequestURI = ""
client := &http.Client{}
delete(r.Header, "Accept-Encoding")
delete(r.Headers, "Content-Length")
resp, err := client.Do(r.WithContext(context.Background())
if err != nil {
return nil, err
}
return resp, nil