I'm using toxiproxy. I'm having this exact issue. Both of the curl solutions mentioned in the issue work (solution a, solution b), but I can't use curl. I need to use the go standard net/http library.
Is there any way to use net/http in such a fashion that I can explicitly tell it what host the proxy is using so it can see that the certificate is valid?
I've tried setting Host and Authority headers on the net/http.Request, but that didn't work.
Details
Toxiproxy output:
proxy=[::]:22002 upstream=maps.googleapis.com:443
My code:
url := "https://localhost:22002/maps/api/geocode/json..."
req, err := http.NewRequest("GET", url, nil)
req.Host = "maps.googleapis.com"
req.Header.Set("Host", "maps.googleapis.com")
res, err := httpClient.Do(req)
Error:
x509: certificate is valid for *.googleapis.com,
*.clients6.google.com, *.cloudendpointsapis.com, cloudendpointsapis.com, googleapis.com, not localhost
You need to set the http.Request.Host field so that the http request has the correct header
req.Host = "maps.googleapis.com"
An you also need to set the hostname in the tls.Config.ServerName field for SNI and host name verification.
If you've already configured a transport for your httpClient, you can set it like so:
httpClient.Transport.(*http.Transport).TLSClientConfig = &tls.Config{
ServerName: "maps.googleapis.com",
}
Or for small programs you can override the default:
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{
ServerName: "maps.googleapis.com",
}
Or create a custom transport for your program
var httpTransport = &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
TLSClientConfig: &tls.Config{
ServerName: "maps.googleapis.com",
},
}
Just make sure you reuse the Transport, and don't create a new one for each request.
Related
DB connection config:
c := sqlutil.DBConfig{
UserName: "user",
UserPassword: "DbPwd",
Host: "DbHost",
Port: "DbPort",
DatabaseName: "DbName",
RdsPemFile: "DbPemFile",
UseSsl: false,
MaxIdleConns: 16,
MaxOpenConns: 128,
InsecureSkipVerify: true,
CharsetType: "utf8mb4",
}
nRetries := 10
delay := 30
Http rest client
&RestClient{
logger: logger,
scheme: "https://",
apiRootURL: apiRootURL,
client: http.Client{
Transport: &http.Transport{
DialContext: (&net.Dialer{
Timeout: 30 * time.Second, //TCP Connect Timeout 30 Secs
}).DialContext,
MaxIdleConnsPerHost: 8,
MaxIdleConns: 16,
},
Timeout: 600 * time.Second}, //Total Read Timeout 10 mins
}
Lambda config file
resource "aws_lambda_function" "lambda_function" {
function_name = var.function_name
role = data.aws_iam_role.lambda.arn
handler = var.handler
timeout = var.timeout
runtime = var.runtime
memory_size = var.memory_size
kms_key_arn = var.kms_key_arn
}
variable "timeout" {
default = 300
}
variable "runtime" {
default = "go1.x"
}
variable "memory_size" {
default = 128
}
variable "reserved_concurrent_executions" {
default = -1
}
Consumer :
resp, err = r.client.Do(req) //req *http.Request, r *RestClient
if err != nil {
<return errorcode>
}
body, err = ioutil.ReadAll(resp.Body)
defer resp.Body.Close()
if resp.StatusCode != 200 { }
Above is my db connection config and https connection config.
I am making an api call (with the https rest client) sequentially for multiple times (2000-3000) using this connection and updating my db (created with the dbconfig).
but after making some api calls, we get an error of net: dns lookup sometimes fails with "no such host" when there are too many open files
The rest client connection is also closed after the required operation. [Do we need more info into this?]
I tried isolating the two calls and found that http client is the problem. after around 1000-2000 calls, we get the fd error.
I understand that each open connection is 1 file descriptor (henceforth mentioned as fd) and the max fd limit of aws lamda is intrinsically 1024.
How do I get around this issue with the confined parameters of the aws lambda ??
lambda memory footprint
I just recently started coding in go and thus I am not so skilled yet.
I have a use case where I want to implement a reverse proxy as a gin route.
So my route looks like this:
server.router.POST("/console", server.proxyConsoleUrl)
And my handler function something like this:
func (server *Server) proxyConsoleUrl(ctx *gin.Context) {
director := func(req *http.Request) {
r := ctx.Request
// This is not working, scheme wss is not supported
req.URL.Scheme = "wss"
req.URL.Host = "192.168.******:8006"
// The path which gets proxied should be overriden
req.URL.RawPath = "/api2/json/nodes/something/qemu/123/vncwebsocket?port=5900&vncticket=something"
req.Header["my-header"] = []string{r.Header.Get("my-header")}
// Golang camelcases headers
delete(req.Header, "My-Header")
// This header has to be added to every request which gets proxied
req.Header["Authorization"] = []string{"MYCUSTOMHEADER"}
}
proxy := &httputil.ReverseProxy{Director: director, Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}}
proxy.ServeHTTP(ctx.Writer, ctx.Request)
}
So my first problem is, that httputil.ReverseProxy doesn't support web socket as far as I know and noticed when running my code:
httputil: unsupported protocol scheme "wss"
The Second problem is, that I want to override the backend url as well as add custom headers which are added by the proxy.
Maybe someone has an idea hot to implement this, if it's even possible. -Thanks in advance
WebSocket support was added to httputil.ReverseProxy in Go version 1.12.
Use the result of url.Parse("https://192.168.******:8006/api2/json/nodes/something/qemu/123/vncwebsocket?port=5900&vncticket=something") to set the target URL. This fixes the following issues:
The WebSocket protocol uses "http" or "https" on the wire, not "wss".
RawPath is ignored when RawPath is a not a valid escaping of Path. See EscapedPath for the details. Because the RawPath in the question includes a query string, it will never be a valid escaping of Path. The client's path is alway used as is. Ditto for the client's query string.
Create the proxy once and reuse it. The important point is to create an reuse a single transport per the Transport documentation. Reusing the proxy accomplishes that goal.
func createProxy() *httputil.ReverseProxy {
target, _ := url.Parse("https://192.168.******:8006/api2/json/nodes/something/qemu/123/vncwebsocket?port=5900&vncticket=something")
director := func(req *http.Request) {
req.URL = target
req.Header["my-header"] = []string{req.Header.Get("my-header")}
delete(req.Header, "My-Header")
req.Header["Authorization"] = []string{"MYCUSTOMHEADER"}
}
return &httputil.ReverseProxy{Director: director, Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}}
}
var proxy = createProxy()
func (server *Server) proxyConsoleUrl(ctx *gin.Context) {
proxy.ServeHTTP(ctx.Writer, ctx.Request)
}
I have an IP address and want to check if I can send requests to the internet from it.
I want to check this both for IPv4 addresses and IPv6 addresses.
The check should work for any IP address (local, public, docker's, lo, etc.).
From another angle, I want to see if I can bind to the address. For example, I have IPs set on my machine to which I can't bind (Tested via curl --interface) and want to filter those from my available addresses.
Usage: user inputs an IP address to send (http get) requests from, and I want to check if the IP is valid/usable.
EDIT on trying to bind and seeing if I can:
I've tried this:
tcpAddr, _ := net.ResolveTCPAddr("tcp", ipString)
client := &http.Client{
Transport: &http.Transport{
DialContext: (&net.Dialer{
Timeout: 10 * time.Second,
LocalAddr: tcpAddr,
}).DialContext,
},
}
_, err = client.Get("http://www.google.com/")
if err != nil {
fmt.Println("Unusable")
} else {
fmt.Println("Usable")
}
It shows all available addresses (even loopback) as usable (Even checked response bodies; All were valid). Many of those aren't bindable through curl --interface.
Try binding it and see if you get an error:
l,err := net.Listen("tcp", address)
if err != nil {
// address is not bindable
} else {
l.Close()
}
I dont know if it's possible as the std lib does not state anything about the current address being used:
http://golang.org/pkg/net/http/
resp, err := http.Get("http://example.com/")
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
What I'm trying to do is set the source address for that http request, why? because I don't want to use my primary ip address for that kind of stuff...
You can set a custom Dialer in the Client's Transport.
// Create a transport like http.DefaultTransport, but with a specified localAddr
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
LocalAddr: localAddr,
DualStack: true,
}).DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
client := &http.Client{
Transport: transport,
}
Setting up a custom dialer with the specify IP works sometimes, but it is weird since sometimes it failed to work, while curl works normally.
After checking the source code of curl, I figured out why curl is able to specify what network interface to be used: SO_BINDTODEVICE. Unfortunately this is a Linux thing.
So compared to JimB's answer, my method has:
Pro: Behaves as stable as curl, which is irrelevant from routing tables
Con: Only support Linux. So you probably want to learn more about the build tags to write platform-specific code.
dialer := &net.Dialer{
Control: func(network, address string, conn syscall.RawConn) error {
var operr error
if err := conn.Control(func(fd uintptr) {
operr = unix.BindToDevice(int(fd), forceNetworkInterface)
}); err != nil {
return err
}
return operr
},
}
client = http.Client{
Transport = &http.Transport{
DialContext: dialer.DialContext,
},
}
In addition, curl performs SO_BINDTODEVICE which behaves like the above code. And for non-Linux platforms, or when SO_BINDTODEVICE fails, curl sets the local IP address just as JimB's answer does. So you can first try my code and then use JimB's answer as a fallback.
See the source code of curl for details.
I have customer http service:
s := &http.Server{
Addr: config.Port,
Handler: Controller.Log(http.DefaultServeMux),
ReadTimeout: 3 * time.Second,
WriteTimeout: 3 * time.Second,
}
http.HandleFunc("/exapmle/router/", exampleFunc)
err := s.ListenAndServe()
if err != nil {
log.Critical(err)
os.Exit(1)
}
It's does't work:
go tool pprof http://localhost:8201/debug/pprof/profile
return error:
Failed to fetch http://localhost:8201/debug/pprof/profile?seconds=30
Thanks.
edit:
I think the problem is i rewrite default http server, net/http/pprof package inject http handler:
func init() {
http.Handle("/debug/pprof/", http.HandlerFunc(Index))
http.Handle("/debug/pprof/cmdline", http.HandlerFunc(Cmdline))
http.Handle("/debug/pprof/profile", http.HandlerFunc(Profile))
http.Handle("/debug/pprof/symbol", http.HandlerFunc(Symbol))
}
the handler does not work in my code.
You set "WriteTimeout" less then profile write time.
on pprof.StartCPUProfile() execute, It already start write, see:
http://golang.org/src/pkg/runtime/pprof/pprof.go#L565
http://golang.org/src/pkg/runtime/pprof/pprof.go#L594
So http.WriteTimeout must be greater than profile write time.
Sorry for my poor English.