I'm trying to use goproxy as an alternative to fiddler on mac OSX,
I wrote the following code, and succeeded to run it:
package main
import (
"flag"
"log"
"net/http"
"gopkg.in/elazarl/goproxy.v1"
)
func main() {
verbose := flag.Bool("v", false, "should every proxy request be logged
to stdout")
addr := flag.String("addr", ":8080", "proxy listen address")
proxy := goproxy.NewProxyHttpServer()
proxy.Verbose = *verbose
proxy.OnResponse().DoFunc(func(resp *http.Response, ctx
*goproxy.ProxyCtx) *http.Response {
contentType := resp.Header.Get("Content-Type")
if contentType == "application/javascript" || contentType == "application/x-javascript" {
// Do something...
}
return resp
})
log.Fatal(http.ListenAndServe(*addr, proxy))
}
According to goproxy documentation I need to configure web proxy & secure web proxy:
I have put multiple breakpoints and run the code in debug, tried to access a website but I can't get it to stop on the breakpoint..
I think I'm missing something in the proxy settings maybe.
Related
I'm trying to dial using credentials and maintain a connection with a socks5 proxy server in Go.
This works nicely if I have IP authorisation set up with the proxy provider, however there is no way pass any auth credentials using net.Dial function in Go:
package main
import (
"io"
"net"
)
func main() {
dst, err := net.Dial("tcp", "11.22.33.44:1111")
if err != nil {
panic("Dial Error:" + err.Error())
}
dst.Close()
}
Go has a useful proxy library and allows authenticated forward requests via proxy using this:
package main
import (
"io"
"net"
)
func main() {
var proxyAuth *proxy.Auth
if conf.Username != "" {
proxyAuth = new(proxy.Auth)
proxyAuth.User = conf.Username
proxyAuth.Password = conf.Password
}
proxyconn, _ := proxy.SOCKS5("tcp", "11.11.11.11:1111", proxyAuth, nil) //returns a Dialer with proxy that can be invoked to connect to another address
dst := proxyconn.Dial("tcp", "22.33.44.55:6666") //connects to an address via proxy
dst.Close()
}
However it returns a Dialer that then asks to connect a target/ultimate address through this authenticated proxy rather the proxy server itself:
My objective here is to return a net.conn connection with a credentials-authenticated proxy server - something like this:
package main
import (
"io"
"net"
)
func main() {
//net.Dial does not have a way to pass Auth creds
dst := net.Dial("tcp", "22.33.44.55:6666", proxyAuth)
dst.Close()
}
The net.Dial() method doesn't concerned with proxy authentication. However, If you want proxy authentication you can set it in header of the request before the call. Please refer this link
dst := net.Dial("tcp", "22.33.44.55:6666")
I've been working on finding a way to try and override the default DNS server for a Go program for a while but still with no success sadly.
The current solution which I had thought would work was:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
DNS := "1.1.1.1"
net.DefaultResolver = &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
d := net.Dialer{
Timeout: time.Millisecond * time.Duration(3000),
}
return d.DialContext(ctx, "udp", fmt.Sprintf("%s:53", DNS))
},
}
resp, err := net.LookupHost("tsdfsdf.com")
if err != nil {
fmt.Printf(err.Error())
}
fmt.Printf("%+v", resp)
}
But the response from this is:
lookup tsdfsdf.com on 192.168.0.1:53: no such host[]
Which is sadly my default DNS server set in my reslov.conf
I have tried forcing to use the Go Resolver by setting export GODEBUG=netdns=go
The long term solution is to be able to over the default resolver for the HTTP Client which would, in turn be consumed by some AWS SDK stuff.
Has any faced this or knows how I can get around this?
FYI, I know "tsdfsdf.com" is not a real domain I'm just using it to spit a No such host error to see what DNS it asked.
What I want to achieve:
An HTTPS server designed specifically to serve binaries to around 1000 devices, sometimes in the same time (clients will fetch the binaries via wget, curl, browser download, etc).
Key functionality features:
client won't be able to download the binary without a certificate
server will allow the client directory browsing/download via browser(if client has certificate)
server is optimized for stability and security, then for speed
server must use high security ciphers and TLS1.2
What I managed to achieve
package main
import (
"log"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
http.ServeFile(w, req, "/")
})
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(http.ListenAndServeTLS(":9900", "cert.crt", "priv.key", http.FileServer(http.Dir("/"))))
}
Now, this works fine although it doesn't check all the features and its not very flexible and somehow I wanted to make it more future proof, as I wish to both learn by creating this project and also expand on it in the future as I'm interested in learning more about servers.
After a bit of research I found several code pieces on GitHub and in tutorials, which led me to put together the following piece of code:
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
http.ServeFile(w, req, "/")
})
cfg := &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
srv := &http.Server{
Addr: ":9900",
Handler: mux,
TLSConfig: cfg,
TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
}
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(srv.ListenAndServeTLS("cert.crt", "priv.key"), http.FileServer(http.Dir("/")))
}
Problem is that when the server starts and I connect to it via browser, I'm presented with the root directory but every time I click on a folder the URL increments the address but the page just refreshes in the "/" directory.
To be exact:
I connect initially to the server and I'm presented the root directory , I'm shown Dir1, Dir2, Dir3
I click Dir1
The URL modifies from https://localhost:9900 to https://localhost:9900/Dir1
But I'm still in the root directory
From what I'm able to see...I think I'm creating a loop somewhere.
If anyone knows what I need to do to make this functional, help would be much appreciated.
NOTE
The above behavior is on Firefox, on Chrome I get one of the 2 errors in the server error log, depending on changes that I make:
2019/09/29 19:59:37 http: TLS handshake error from [::1]:53287: EOF
2019/09/29 19:15:59 http: TLS handshake error from [::1]:50457: tls: client doesn't support selected certificate
There are several examples elsewhere on how to do this (as has been commented).
Here's a worked example in which the fileserver handler is an enhanced version of that in the standard library that supports more cache headers and locally-compressed files. The standard fileserver handler can be dropped in here instead if that's what you need.
package main
import (
"crypto/tls"
"flag"
"fmt"
"log"
"net/http"
"time"
"github.com/rickb777/servefiles/v3"
)
var path = flag.String("path", "..", "directory for the files tp be served")
var cert = flag.String("cert", "", "file containing the certificate (optional)")
var key = flag.String("key", "", "file containing the private key (optional)")
var port = flag.Int("port", 8080, "TCP port to listen on")
var maxAge = flag.String("maxage", "", "Maximum age of assets sent in response headers - causes client caching")
var verbose = flag.Bool("v", false, "Enable verbose messages")
func main() {
flag.Parse()
if *verbose {
servefiles.Debugf = log.Printf
}
if (*cert != "" && *key == "") ||
(*cert == "" && *key != "") {
log.Fatal("Both certificate file (-cert) and private key file (-key) are required.")
}
h := servefiles.NewAssetHandler(*path)
if *maxAge != "" {
d, err := time.ParseDuration(*maxAge)
log.Printf("MaxAge: %s %v\n", d, err)
h = h.WithMaxAge(d)
}
srv := &http.Server{
Addr: fmt.Sprintf(":%d", *port),
Handler: h,
}
if *cert != "" {
srv.TLSConfig = &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
log.Printf("Access the server via: https://localhost:%d/", *port)
log.Fatal(srv.ListenAndServeTLS(*cert, *key))
} else {
log.Printf("Access the server via: http://localhost:%d/", *port)
log.Fatal(srv.ListenAndServe())
}
}
Source code https://github.com/rickb777/servefiles/blob/master/v3/webserver/example.go
I have just cloned the example code from micro/go-grpc and tried to build a grpc server in localhost.
proto file
syntax = "proto3";
package go.micro.srv.greeter;
service Say {
rpc Hello(Request) returns (Response) {}
}
message Request {
string name = 1;
}
message Response {
string msg = 1;
}
server/main.go
package main
import (
"context"
"log"
"github.com/micro/go-micro"
"github.com/micro/go-grpc"
hello "github.com/micro/go-grpc/examples/greeter/server/proto/hello"
)
type Say struct{}
func (s *Say) Hello(ctx context.Context, req *hello.Request, rsp
*hello.Response) error {
log.Print("Received Say.Hello request")
rsp.Msg = "Hello " + req.Name
return nil
}
func main() {
service := grpc.NewService(
micro.Name("go.micro.srv.greeter"),
)
// optionally setup command line usage
service.Init()
// Register Handlers
hello.RegisterSayHandler(service.Server(), new(Say))
// Run server
if err := service.Run(); err != nil {
log.Fatal(err)
}
}
client/main.go
package main
import (
"context"
"fmt"
"github.com/micro/cli"
"github.com/micro/go-grpc"
hello "github.com/micro/go-grpc/examples/greeter/server/proto/hello"
"github.com/micro/go-micro"
)
var (
// service to call
serviceName string
)
func main() {
service := grpc.NewService()
service.Init(
micro.Flags(cli.StringFlag{
Name: "service_name",
Value: "go.micro.srv.greeter",
Destination: &serviceName,
}),
)
cl := hello.NewSayService(serviceName, service.Client())
rsp, err := cl.Hello(context.TODO(), &hello.Request{
Name: "John",
})
if err != nil {
fmt.Println(err)
return
}
fmt.Println(rsp.Msg)
}
My OS is MacOsX, go version is 1.11.1.
When i run the server side example code, everything looks fine:
$ go run ./main.go --server_address:localhost:9999
2018/11/18 20:08:05 Listening on 127.0.0.1:9999
2018/11/18 20:08:05 Broker Listening on [::]:62739
2018/11/18 20:08:05 Registering node: go.micro.srv.greeter-9b2818b0-eb2a-11e8-bfef-720008acb800
But if I run the client side example code, always received:
{"id":"","code":0,"detail":"transport: received the unexpected content-type "text/plain"","status":""}
I tried to remove the --server_address and still the same. I tried to add the mdns registry, not working either. I tried to use $ micro health go.micro.srv.greeter, it returned the same result.
Wonder whats wrong with my setup?
I just got this same error because I was pointing to the wrong port. I'm sure my setup is different from yours and it's a different issue, but the problem was that I was trying to make a GRPC request to an http server, which returned a 404 not found with a content-type of "text/plain" for html. If you have the same problem when removing your server address, it's likely either you aren't reading the param correctly, or the value you have set is still pointing to a place where there is an http server and not a GRPC server.
I don't know why this reverse proxy is not working. I've seen several examples and I can't find anything wrong with it.
package main
import (
"log"
"net/url"
"net/http"
"net/http/httputil"
)
func report(r *http.Request){
log.Print("URL: " + r.URL.Path)
log.Print("Scheme: " + r.URL.Scheme)
log.Print("Host: " + r.URL.Host)
//r.URL.Scheme = "http"
//r.URL.Host = "stackoverflow.com"
//r.Header.Set("Host", "stackoverflow.com")
//log.Print("Header Host: " + r.Header.Get("Host"))
}
func main() {
proxy := httputil.NewSingleHostReverseProxy( &url.URL{Scheme:"http",Host:"myrealserver.com"})
proxy.Director = report
// http.Handle("/", proxy)
error := http.ListenAndServe("mylocalhost.com:8080", proxy)
if(error != nil) {
log.Fatal(error)
}
}
It logs:
2014/04/18 21:32:50 URL: /arg/es
2014/04/18 21:32:50 Scheme:
2014/04/18 21:32:50 Host:
2014/04/18 21:32:50 http: proxy error: unsupported protocol scheme ""
2014/04/18 21:32:51 URL: /favicon.ico
2014/04/18 21:32:51 Scheme:
2014/04/18 21:32:51 Host:
2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme ""
If I uncomment the line that redefines the Schema the error message becomes:
2014/04/18 21:38:05 http: proxy error: http: no Host in request URL
If I uncomment the line that redefines the host also, then the target server becomes stackoverflow.com (I mean, it never uses "myrealserver.com").
If I ask for mylocalhost.com:8080/somepath (or even /) then I get a 404 from Stackoverflow, no matter if stackoverflow.com/somepath exists or not. It says:
Couldn't find mylocalhost.com:8080
The Q&A site mylocalhost.com:8080 doesn't seem to exist... yet
It does not translate the Host header automatically.
If then I uncomment the line that sets (and the other one that prints) the Header "Host". Then I can read "stackoverflow.com" in the log, but I still get the same 404 page reporting that I am trying to access "mylocalhost.com".
I'm using go1.2.1 linux/amd64
How is it that I am supposed to make the program work as a proxy?
Thanks to Alex from Golang-nuts, I have the answer now.
This is what Alex said:
Just need to set http.Request.Host [and scheme] in the Director to get this
working: http://play.golang.org/p/I17ZSM6LQb
If you read the source for SingleHostReverseProxy
(http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L61), it
sets its own Director which you are overriding. So you need to
reimplement what it already does plus the extra Host change.
Anyway, that didn't solve de Header part of the problem: the target server was still receiving "localhost:8080" as the HTTP Host name, so I did it without the ReverseProxy package, just with http and a RoundTripper, plus a helper function that copies all the headers:
package main
import (
"flag"
"fmt"
"os"
"log"
"net/http"
"io/ioutil"
)
var target *string
func main() {
target = flag.String("target", "http://stackoverflow.com", "target URL for reverse proxy")
flag.Parse()
http.HandleFunc("/", report)
log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}
func report(w http.ResponseWriter, r *http.Request){
uri := *target+r.RequestURI
fmt.Println(r.Method + ": " + uri)
if r.Method == "POST" {
body, err := ioutil.ReadAll(r.Body)
fatal(err)
fmt.Printf("Body: %v\n", string(body));
}
rr, err := http.NewRequest(r.Method, uri, r.Body)
fatal(err)
copyHeader(r.Header, &rr.Header)
// Create a client and query the target
var transport http.Transport
resp, err := transport.RoundTrip(rr)
fatal(err)
fmt.Printf("Resp-Headers: %v\n", resp.Header);
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fatal(err)
dH := w.Header()
copyHeader(resp.Header, &dH)
dH.Add("Requested-Host", rr.Host)
w.Write(body)
}
func fatal(err error) {
if err != nil {
log.Fatal(err)
os.Exit(1)
}
}
func copyHeader(source http.Header, dest *http.Header){
for n, v := range source {
for _, vv := range v {
dest.Add(n, vv)
}
}
}
Now I'm able to see StackOverflow or any other site how it's supposed to be.
I'm still working on POST calls, though, so this is a work in progress.
A little late to the party, but ReverseProxy isn't broken, it's just a little confusing because it doesn't work how you'd expect (at the least, I expected it to work the way you did, so that makes two of us).
From the docs:
// For server requests Host specifies the host on which the
// URL is sought. Per RFC 2616, this is either the value of
// the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
//
// For client requests Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
Since under the hood ReverseProxy is using this Request to make a client request (after ReverseProxy.Director optionally modifies it), if the Host is set it will override the Host header. This will always be set, because as the first part of the doc comment states "For server requests, Host specifies the host on which the URL is sought".
So in addition to Sebastián's answer, you also need to set req.Host. For example, to proxy to example.com:
proxy := ReverseProxy{
Director: func(req *http.Request) {
req.URL.Scheme = "http"
req.URL.Host = "example.com"
req.Host = "example.com"
}
}
Alternatively you can set req.Host to "" and it will use the value of req.URL.Host.
I figured this out by reading: https://github.com/golang/go/issues/14413