I'm using httputil.ReverseProxy to proxy Amazon s3 files to my clients. I'd like to hide all headers coming from Amazon - is that possible without having to reimplement Reverse Proxy?
proxy := httputil.ReverseProxy{Director: func(r *http.Request) {
r.Header = http.Header{} // Don't send client's request headers to Amazon.
r.URL = proxyURL
r.Host = proxyURL.Host
}}
proxy.ServeHTTP(w, r) // How do I remove w.Headers ?
You can implement ReverseProxy.Transport
type MyTransport struct{
header http.Header
}
func (t MyTransport) RoundTrip(r *Request) (*Response, error){
resp, err := http.DefaultTransport.RoundTrip(r)
resp.Header = t.header
return resp, err
}
mytransport := MyTransport{
//construct Header
}
proxy := httputil.ReverseProxy{Director: func(r *http.Request) {
r.Header = http.Header{} // Don't send client's request headers to Amazon.
r.URL = proxyURL
r.Host = proxyURL.Host
},
Transport: mytransport,
}
This was my solution to remove/replace all the http.ReverseProxy response headers:
type responseHeadersTransport http.Header
func (t responseHeadersTransport) RoundTrip(r *http.Request) (*http.Response, error) {
resp, err := http.DefaultTransport.RoundTrip(r)
if err != nil {
return nil, err
}
resp.Header = http.Header(t)
return resp, nil
}
func ProxyFile(w http.ResponseWriter, r *http.Request) {
// ...
headers := http.Header{}
headers.Set("Content-Type", file.ContentType)
headers.Set("Content-Length", fmt.Sprintf("%d", file.Filesize))
headers.Set("Content-Disposition", fmt.Sprintf("inline; filename=\"%s\"", file.Filename))
proxy := httputil.ReverseProxy{
Director: func(r *http.Request) { // Remove request headers.
r.Header = http.Header{}
r.URL = proxyURL
r.Host = proxyURL.Host
},
Transport: responseHeadersTransport(headers), // Replace response headers.
}
proxy.ServeHTTP(w, r)
}
Related
I will like to pass a list of forwarding proxy servers for POST request
Currently i am able to do it with just single forwarding proxy
serverProxy := "http://user:password#123.45.67.89:3128"
request, error := http.NewRequest("POST", httpposturl, bytes.NewBuffer(requestJSON))
request.Header.Set("Content-Type", "application/json; charset=UTF-8")
proxyURL, _ := url.Parse(serverProxy)
proxy := http.ProxyURL(proxyURL)
transport := &http.Transport{Proxy: proxy}
client := &http.Client{Transport: transport}
what i will like to do is pass a list to url.Parse and want it to use them using round robin balancing
so something like this
serverProxy := "http://user:password#123.45.67.89:3128, http://user:password#223.45.67.89:3128"
and then it will select which of the proxy servers to use and rotate them within requests
Is this possible?
UPDATE:
I want to be able to pass the rotated proxy server like this
proxyServer := roundRobin("http://round:robin#123.45.67.89:3128, http://robin:round#223.45.67.89:3128")
fmt.Println("proxy server used", proxyServer, "\n")
transport := &http.Transport{Proxy: proxyServer}
client := &http.Client{Transport: transport}
Create a proxy function that round-robins through your proxy URLs. Use that function in your transport:
func roundRobin(urls []*url.URL) func(*http.Request) (*url.URL, error) {
var mu sync.Mutex
var i int
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % len(urls)
u := urls[i]
mu.Unlock()
return u, nil
}
}
transport := &http.Transport{Proxy: roundRobin(yourProxyURLs)}
client := &http.Client{Transport: transport}
Here's the Montage's answer with explanation:
The requirement is to forward request through proxies in round-robin fashion
Since we are using http.Client to make request we can look at http.Client documentation to see if it provide any support for forwarding request through proxy when we look at the documentation we can see that it does support passing proxy which we can pass through http.Transport type which will be passed to http.Client. http.Transport takes proxy through Proxy field which takes in func that return *url.URL and error there are existing methods like http.ProxyURL and http.ProxyFromEnvironment provided within http package that we can use to pass proxies to http.Transport but the problem with these methods is that they only take a single proxy server which does not solve our problem at hand and hence we would require to create our own function which takes in multiple proxy servers urls and round-robin between them.
If we look at one of the existing method implemention as our base for creating our own method lets go with http.ProxyURL for our case the implementation can be found here. I have copied the implementation below
func ProxyURL(fixedURL *url.URL) func(*Request) (*url.URL, error) {
return func(*Request) (*url.URL, error) {
return fixedURL, nil
}
}
we can see that its a simple closure which takes in single url and return a closure function which then intern return the url passed in as parameter. so we can take it base and create our own round-robin clouse function
func roundRobin(proxies ...string) func(*http.Request) (*url.URL, error) {
var urls []*url.URL
for _, proxy := range proxies {
u, err := url.Parse(proxy)
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
var mu sync.Mutex
var i, lenUrls int = 0, len(urls)
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % lenUrls
u := urls[i]
mu.Unlock()
return u, nil
}
}
Lets go over the roundRobin function implementation it is a variadic function which takes in proxy url(s) in string format as argument, which internally gets converted to url.URL by parsing the string using url.Parse then using the parsed url.URL to create slice of urls []*url.URL which then being used to forward request in round-robin fashion
Complete working example can be found below:
package main
import (
"fmt"
"log"
"net/url"
"net/http"
"sync"
)
func roundRobin(proxies ...string) func(*http.Request) (*url.URL, error) {
var urls []*url.URL
for _, proxy := range proxies {
u, err := url.Parse(proxy)
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
var mu sync.Mutex
var i, lenUrls int = 0, len(urls)
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % lenUrls
u := urls[i]
mu.Unlock()
return u, nil
}
}
func main() {
proxyFn := roundRobin("http://user:password#123.45.67.89:3128", "http://user:password#223.45.67.89:3128")
transport := &http.Transport{Proxy: proxyFn}
client := &http.Client{Transport: transport}
req, err := http.NewRequest("POST", "http://example.com", nil)
req.Header.Set("Content-Type", "application/json; charset=UTF-8")
resp, err := client.Do(req)
if err != nil {
fmt.Println(resp)
}
fmt.Println(proxyFn(nil))
fmt.Println(proxyFn(nil))
}
Playground
Another version
package main
import (
"fmt"
"log"
"net/http"
"net/url"
"sync"
)
func praseUrls(proxies ...string) (urls []*url.URL) {
for _, proxy := range proxies {
u, err := url.Parse(proxy)
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
return
}
func roundRobin(max int) func() int {
var i int
return func() int {
i = (i + 1) % max
return i
}
}
func proxyFn(urls []*url.URL) func(*http.Request) (*url.URL, error) {
var m sync.Mutex
fn := roundRobin(len(urls))
return func(*http.Request) (*url.URL, error) {
m.Lock()
u := urls[fn()]
m.Unlock()
return u, nil
}
}
func main() {
proxies := []string{"http://user:password#123.45.67.89:3128", "http://user:password#223.45.67.89:3128"}
urls := praseUrls(proxies...)
transport := &http.Transport{Proxy: proxyFn(urls)}
client := &http.Client{Transport: transport}
req, err := http.NewRequest("POST", "http://example.com", nil)
req.Header.Set("Content-Type", "application/json; charset=UTF-8")
resp, err := client.Do(req)
if err != nil {
fmt.Println(resp)
}
}
Playground
Note: It would be better to pass proxy urls from env variable which would help in case any proxy server changes or new are added
Here's the Montage's answer with code to parse a string.
func roundRobin(serverProxy string) func(*http.Request) (*url.URL, error) {
parts := strings.Split(serverProxy, ",")
var urls []*url.URL
for _, part := range parts {
u, err := url.Parse(strings.TrimSpace(part))
if err != nil {
log.Fatal(err)
}
urls = append(urls, u)
}
var mu sync.Mutex
var i int
return func(r *http.Request) (*url.URL, error) {
mu.Lock()
i = (i + 1) % len(urls)
u := urls[i]
mu.Unlock()
return u, nil
}
}
serverProxy := "http://user:password#123.45.67.89:3128, http://user:password#223.45.67.89:3128"
transport := &http.Transport{Proxy: roundRobin(serverProxy)}
client := &http.Client{Transport: transport}
I have krakend apigateway which is using opentracing and sending req headers as X-B3-... for tracing and my service is using opentelemetry.
This is what I'm having right now on jaeger.
enter image description here
enter image description here
I want the service spans to come under the apigateways'.
Workaround I have done:
This is the exact handler for the request using chi router.
func (env *Env) getCommentForBlogRouter(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
keys := []string{"X-B3-Traceid", "X-B3-Spanid", "X-B3-Sampled"}
var carrier propagation.HeaderCarrier = propagation.HeaderCarrier{}
for _, k := range keys {
carrier.Set(strings.ToLower(k), r.Header[k][0])
}
var propagator propagation.TextMapPropagator = otel.GetTextMapPropagator()
ctx = propagator.Extract(ctx, carrier)
// fmt.Println(ctx)
tr := otel.Tracer("Handler: blog-comments")
ctx, span := tr.Start(ctx, "handler span")
defer span.End()
blogId := r.URL.Query().Get("blog-id")
span.SetAttributes(attribute.Key("blog-id").String(fmt.Sprint(blogId)))
var spanDB trace.Span
ctx, spanDB = tr.Start(ctx, "Select row")
comments, err := env.comments.GetForBlog(blogId)
spanDB.End()
var spanRes trace.Span
_, spanRes = tr.Start(ctx, "Sending Response")
defer spanRes.End()
if err != nil {
fmt.Println(err)
SendError(w, http.StatusInternalServerError, "Something went wrong")
return
}
if comments == nil {
comments = []models.Comment{}
}
SendResponse(w, http.StatusOK, map[string]interface{}{
"data": comments,
})
}
Ok, I figured out how to make it work.
I added this middleware and it will sync the request context with the headers traceID, spanID and traceFlags.
After this we are good to create tracer and spans as we want.
func Tracing(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
traceID, _ := trace.TraceIDFromHex(r.Header["X-B3-Traceid"][0])
spanID, _ := trace.SpanIDFromHex(r.Header["X-B3-Spanid"][0])
var traceFlags trace.TraceFlags
if r.Header["X-B3-Sampled"][0] == "1" {
traceFlags = trace.FlagsSampled
}
spanContext := trace.NewSpanContext(trace.SpanContextConfig{
TraceID: traceID,
SpanID: spanID,
TraceFlags: traceFlags,
})
ctx := trace.ContextWithSpanContext(r.Context(), spanContext)
r = r.WithContext(ctx)
next.ServeHTTP(w, r)
})
}
I'm developing a service rest in Go using go-kit. I need send a header response. This header response should have the same value of request header.
This is a part of my transport.go:
func MakeHandler(m **http.ServeMux) http.Handler {
const URL = "..."
var serverOptions []kithttp.ServerOption
logger := log.NewLogfmtLogger(os.Stderr)
var svc repositories.SignDocumentRepository
impl := persistence.NewSignerDocumentRepository()
svc = middleware.LoggingMiddlewareSignDocument{Logger: logger, Next: impl}
registerHandler := kithttp.NewServer(
makeSignDocumentEndpoint(svc),
decodeRequest,
encodeResponse,
serverOptions...,
)
r := mux.NewRouter()
r.Handle(URL, handlers.LoggingHandler(os.Stdout, registerHandler))
(*m).Handle(URL, r)
return nil
}
func decodeRequest(_ context.Context, r *http.Request) (interface{}, error) {
return r, nil
}
func encodeResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
w.Header().Set("headerA", "val1")
w.Header().Set("headerB", "") // This header should be equal that a header request
switch response.(type) {
case model.MsgRsHdr:
w.WriteHeader(http.StatusPartialContent)
default:
w.WriteHeader(http.StatusAccepted)
}
if response != nil {
return json.NewEncoder(w).Encode(response)
}
return nil
}
How do I get a request header in encodeResponse method?
You can use ServerBefore to put *http.Request in the context, and can get it in encodeResponse to read request headers.
type ctxRequestKey struct{}
func putRequestInCtx(ctx context.Context, r *http.Request, _ Request) context.Context {
return context.WithValue(ctx, ctxRequestKey{}, r)
}
func encodeResponse(ctx context.Context, w http.ResponseWriter, response interface{}) error {
r := ctx.Value(ctxRequestKey{}).(*http.Request)
// can use r.Header.Get here to read request here.
}
serverOptions := []kithttp.ServerOptions{
kithttp.ServerBefore(putRequestInCtx),
}
registerHandler := kithttp.NewServer(
makeSignDocumentEndpoint(svc),
decodeRequest,
encodeResponse,
serverOptions...,
)
I'm trying to make a proxy by golang.
The origin version is written by lua, nginx like this:
location / {
keepalive_timeout 3600s;
keepalive_requests 30000;
rewrite_by_lua_file ./test.lua;
proxy_pass http://www.example.com/bd/news/home;
}
and lua file like this:
local req_params = ngx.req.get_uri_args()
local args = {
media = 24,
submedia = 46,
os = req_params.os,
osv = req_params.osv,
make = req_params.make,
model = req_params.model,
devicetype = req_params.devicetype,
conn = req_params.conn,
carrier = req_params.carrier,
sw = req_params.w,
sh = req_params.h,
}
if tonumber(req_params.os) == 1 then
args.imei = req_params.imei
args.adid = req_params.android_id
end
ngx.req.set_uri_args(args)
I try to do the same thing by golang, and my code is like this:
const newsTargetURL = "http://www.example.com/bd/news/home"
func GetNews(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only get allowed", http.StatusMethodNotAllowed)
return
}
// deal params
rq := r.URL.Query()
os := rq.Get("os")
osv := rq.Get("osv")
imei := rq.Get("imei")
androidID := rq.Get("android_id")
deviceMake := rq.Get("make")
model := rq.Get("model")
deviceType := rq.Get("devicetype")
sw := rq.Get("w")
sh := rq.Get("h")
conn := rq.Get("conn")
carrier := rq.Get("carrier")
uv := make(url.Values)
uv.Set("media", "24")
uv.Set("submedia", "46")
uv.Set("os", os)
uv.Set("osv", osv)
if os == "1" {
uv.Set("imei", imei)
uv.Set("anid", androidID)
}
uv.Set("make", deviceMake)
uv.Set("model", model)
uv.Set("sw", sw)
uv.Set("sh", sh)
uv.Set("devicetype", deviceType)
uv.Set("ip", ip)
uv.Set("ua", ua)
uv.Set("conn", conn)
uv.Set("carrier", carrier)
t := newsTargetURL + "?" + uv.Encode()
// make a director
director := func(req *http.Request) {
u, err := url.Parse(t)
if err != nil {
panic(err)
}
req.URL = u
}
// make a proxy
proxy := &httputil.ReverseProxy{Director: director}
proxy.ServeHTTP(w, r)
}
func main() {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(GetNews))
srv := &http.Server{
Addr: ":2222",
Handler: mux,
}
srv.ListenAndServe()
}
I put this go version to the same server where lua version locate, but it does not work as lua file do. I read the httputil document but found nothing that can help. What do I need to do?
I wrote together a simple proxy for GET requests. Hope this helps.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
const newsTargetURL = "http://www.example.com/bd/news/home"
func main() {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(GetNews))
srv := &http.Server{
Addr: ":2222",
Handler: mux,
}
// output error and quit if ListenAndServe fails
log.Fatal(srv.ListenAndServe())
}
func GetNews(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only get allowed", http.StatusMethodNotAllowed)
return
}
// build proxy url
urlstr := fmt.Sprintf("%s?%s", newsTargetURL, r.URL.RawQuery)
// request the proxy url
resp, err := http.Get(urlstr)
if err != nil {
http.Error(w, fmt.Sprintf("error creating request to %s", urlstr), http.StatusInternalServerError)
return
}
// make sure body gets closed when this function exits
defer resp.Body.Close()
// read entire response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
http.Error(w, "error reading response body", http.StatusInternalServerError)
return
}
// write status code and body from proxy request into the answer
w.WriteHeader(resp.StatusCode)
w.Write(body)
}
You can try it as is. It will work and show the content of example.com.
It uses a single handler GetNews for all requests. It skips all of the request parameter parsing and building by simply using r.url.RawQuery and newsTargetURL to build the new url.
Then we make a request to the new url (the main part missing in your question). From the response we read resp.StatusCode and resp.body to use in our response to the original request.
The rest is error handling.
The sample does not forward any additional information like cookies, headers, etc. That can be added as needed.
I'm trying out this small Go example https://github.com/jakecoffman/golang-rest-bootstrap, and so far so good.
I'm trying to add CORS to allow my front-end app access.
Here is my Main.go
func main() {
var err error
session, err = r.Connect(r.ConnectOpts{
Address: "localhost:28015",
Database: "demo",
MaxOpen: 40,
})
if err != nil {
log.Fatalln(err.Error())
}
r := mux.NewRouter()
users.Init(r, session)
accounts.Init(r, session)
r.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
})
port := "9999" // os.Getenv("PORT")
log.Println("Serving on", ":"+port)
http.ListenAndServe(":"+port, context.ClearHandler(r))
}
It allows CORS at the root url, but since the other routes are handled in the controllers I just can't seem to get CORS to work there as well.
Here is part of the AccountController
func NewAccountController(r *mux.Router, s AccountService) *AccountController {
cont := AccountController{s}
r.Handle("/accounts", cont)
r.Handle("/accounts/{id}", cont)
return &cont
}
And
func (a AccountController) ServeHTTP(w http.ResponseWriter, r *http.Request) {
code := http.StatusMethodNotAllowed
var data interface{}
defer func(c int) {
log.Println(r.URL, "-", r.Method, "-", code, r.RemoteAddr)
}(code)
if r.URL.Path == "/accounts" {
switch r.Method {
case "GET":
code, data = a.List(w, r)
case "POST":
code, data = a.Add(w, r)
default:
return
}
} else {
switch r.Method {
case "GET":
code, data = a.Get(w, r)
case "PUT":
code, data = a.Update(w, r)
case "DELETE":
code, data = a.Delete(w, r)
default:
return
}
}
w.WriteHeader(code)
w.Header().Set("Content-Type", "application/json")
err := json.NewEncoder(w).Encode(data)
if err != nil {
log.Println("Failed to write data: ", err)
code = http.StatusInternalServerError
}
}
Any ideas or pointers would be great.
Thanks,
JB
You can make a simple middleware for that:
type CORSMiddleware struct {
http.Handler
}
func (cm CORSMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
cm.Handler.ServeHTTP(w, r)
}
Then, you can use it like that:
var h http.Handler = CORSMiddleware{cont}
r.Handle("/accounts", h)
r.Handle("/accounts/{id}", h)