I want to make https calls via a third party proxy using Go.
The third party proxy accepts a username and password.
My code works fine when I call the proxy using http. But I get a runtime error when I change the proxy to https.
The error message is:
proxyconnect tcp: tls: first record does not look like a TLS handshake
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1248521]
package main
import (
"log"
"net/http"
"net/url"
)
func main() {
// proxyURL - change to https get runtime error
proxyStr := "http://username:password#ip:port"
proxyURL, err := url.Parse(proxyStr)
if err != nil { log.Println(err) }
// targetURL
targetStr := "https://httpbin.org/get"
targetURL, err := url.Parse(targetStr)
if err != nil { log.Println(err) }
//config Transport w Proxy
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
//config Client w Transport
client := &http.Client{
Transport: transport,
}
//config Request
request, err := http.NewRequest("GET", targetURL.String(), nil)
if err != nil { log.Println(err) }
log.Println("Requesting", targetStr)
//try it
response, err := client.Do(request)
if err != nil { log.Println(err) }
log.Println(response.Status)
}
Related
My PC has multiple IP addresses(ex: 10.1.1.20, 192.168.123.30, ...).
Can I extract the connected local ip address when connecting to remote server using http.Client?
If this is not possible with http.Client, is there any other possible way?
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
req, err := http.NewRequest("GET", "https://www.google.com", nil)
if err != nil {
panic(err)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
// extract the local ip address???
// getsockname(?????)
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
fmt.Printf("StatusCode=%v\n", resp.StatusCode)
fmt.Printf("%v\n", string(data))
}
You can either:
loop through all network interfaces
or retrieve the preferred outbound ip address
But in both case, the fact that you are in the middle of using an http.Client and making a GET would not matter: you could get those IP addresses independently.
You can provide your own Transport implementation that extracts the outgoing local IP address right after establishing the TCP connection, e.g. like this:
client := &http.Client{
Transport: &http.Transport{
Dial: func(network, addr string) (net.Conn, error) {
conn, err := net.Dial(network, addr)
if err == nil {
localAddr := conn.LocalAddr().(*net.TCPAddr)
fmt.Println("LOCAL IP:", localAddr.IP)
}
return conn, err
},
},
}
I have a secure websocket server running on localhost:443/server-demo ( jetty websocket server).
Now I am writing a go client that can communicate with the websocket server. I am able to connect to the websocket server using right certificates. Here is the sample code.
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"log"
)
func main() {
cert, err := tls.LoadX509KeyPair("nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin.pem", "nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin-key.pem")
if err != nil {
log.Fatalf("server: loadkeys: %s", err)
}
config := tls.Config{Certificates: []tls.Certificate{cert}, InsecureSkipVerify: true}
conn, err := tls.Dial("tcp", "127.0.0.1:443", &config)
if err != nil {
log.Fatalf("client: dial: %s", err)
}
defer conn.Close()
log.Println("client: connected to: ", conn.RemoteAddr())
state := conn.ConnectionState()
for _, v := range state.PeerCertificates {
fmt.Println(x509.MarshalPKIXPublicKey(v.PublicKey))
fmt.Println(v.Subject)
}
log.Println("client: handshake: ", state.HandshakeComplete)
log.Println("client: mutual: ", state.NegotiatedProtocolIsMutual)
message := "Hello\n"
n, err := io.WriteString(conn, message)
if err != nil {
log.Fatalf("client: write: %s", err)
}
log.Printf("client: wrote %q (%d bytes)", message, n)
reply := make([]byte, 256)
n, err = conn.Read(reply)
log.Printf("client: read %q (%d bytes)", string(reply[:n]), n)
log.Print("client: exiting")
}
The above code throws this error:
"HTTP/1.1 400 No URI\r\nContent-Type: text/html;charset=iso-8859-1\r\nContent-Length: 49\r\nConnection: close\r\nServer: Jetty(9.4.19.v20190610)\r\n\r\n<h1>Bad Message 400</h1><pre>reason: No URI</pre>" (188 bytes)
My question is after making the connection how can I send message to particular URI? i.e I want to send a message to wss://localhost:443/server-demo.
The code in a question does not establish a WebSocket connection to the server.
To establish the WebSocket connection, the application must write a WebSocket handshake to conn and receive the handshake response. See the RFC for the details.
Most applications use a websocket package than handles all of these details. The gorilla/websocket package is a popular choice.
This code should get you started with gorilla:
cert, err := tls.LoadX509KeyPair("nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin.pem", "nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin-key.pem")
if err != nil {
log.Fatalf("server: loadkeys: %s", err)
}
config := tls.Config{Certificates: []tls.Certificate{cert}, InsecureSkipVerify: true}
d := websocket.Dialer{
TLSClientConfig: &config,
}
c, _, err := d.Dial("wss://localhost:443/server-demo", nil)
if err != nil {
log.Fatal(err)
}
defer c.Close()
// Use `c` to send and receive messages
How can I force a simple Go client to use HTTP/2 and prevent it from falling back to HTTP 1.1 ?
I have a simple HTTP/2 server running on "localhost" and it returns details of the request in its reply. Here is the output using Google Chrome for this URL: https://localhost:40443/bananas
I like bananas!
Method = GET
URL = /bananas
Proto = HTTP/2.0
Host = localhost:40443
RequestURI = /bananas
But here is what I get for my Go client code. You can see it falls back to HTTP 1.1
I like monkeys!
Method = GET
URL = /monkeys
Proto = HTTP/1.1
Host = localhost:40443
RequestURI = /monkeys
Below is the source code of my best attempt to contact the same server using HTTP/2, but it always falls back to HTTP 1.1
// simple http/2 client
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"log"
"net/http"
)
const (
certFile = "client-cert.pem"
keyFile = "client-key.pem"
caFile = "server-cert.pem"
)
func main() {
// Load client certificate
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
log.Fatal(err)
}
// Load CA cert
caCert, err := ioutil.ReadFile(caFile)
if err != nil {
log.Fatal(err)
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
// Setup HTTPS client
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
RootCAs: caCertPool,
}
tlsConfig.BuildNameToCertificate()
transport := &http.Transport{TLSClientConfig: tlsConfig}
client := &http.Client{Transport: transport}
response, err := client.Get("https://localhost:40443/monkeys")
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
// dump response
text, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Body:\n%s", text)
}
Any suggestions would be appreciated, including pointers to other working examples that illustrate how to make HTTP/2 client requests in Go.
First import "golang.org/x/net/http2" package. And then change
transport := &http.Transport{TLSClientConfig: tlsConfig}
to
transport := &http2.Transport{TLSClientConfig: tlsConfig}
I currently am working on vendor go balancer code. I need to remove the tcp dial call and emulate a successful connection without the call. In the code below, there is
this line:
ds, err := net.Dial("tcp", backend.String());if err != nil {
log.Printf("failed to dial %s: %s", backend, err)
us.Close()
return
}
What this does is make a dial to the tcp server and then return connection
response in ds, which is defined here: https://golang.org/pkg/net/#Dial
What i need is to obtain the ds without doing the tcp dialer. I'm
trying to test the load balancer without any actual tcp calls. So,
essentially,
when we enter handleConnection wed create a net connection prior to the tcp dial and use this net conn, which should emulate 100% net connection before the tcp dialing begins.
func copy(wc io.WriteCloser, r io.Reader) { defer wc.Close()
io.Copy(wc, r)
}
func handleConnection(us net.Conn, backend BA.Backend) {
if backend == nil {
log.Printf("no backend available for connection from %s",
us.RemoteAddr())
us.Close()
return
}
host, _, _ := net.SplitHostPort(us.RemoteAddr().String())
_, ok := dbAuthTokenData[host]; if !ok {
w := bufio.NewWriter(us)
w.WriteString("InvalidCredentials")
w.Flush()
us.Close()
return
}
ds, err := net.Dial("tcp", backend.String());if err != nil {
log.Printf("failed to dial %s: %s", backend, err)
us.Close()
return
}
// Ignore errors
go copy(ds, us)
go copy(us, ds)
}
func tcpBalance(bind string, backends BA.Backends) error {
log.Println("using tcp balancing")
ln, err := net.Listen("tcp", bind)
if err != nil {
return fmt.Errorf("failed to bind: %s", err)
}
log.Printf("listening on %s, balancing %d backends", bind, backends.Len())
for {
conn, err := ln.Accept()
if err != nil {
log.Printf("failed to accept: %s", err)
continue
}
go handleConnection(conn, backends.Choose())
}
return err
}
I tried commenting out go handleConnection(conn, backends.Choose()) but that failed.
The pattern you could refactor your code is to create a Dialer interface. In your code example you are using the returned ds just as an io.ReadWriteCloser. So you don't need to implement the whole net.Conn interface. As net.Conn has the read and write method inside everything works
type Dialer interface{
Dial(network, address string) (io.ReadWriteCloser, error)
}
Now let's change your function:
func handleConnection(us net.Conn, backend BA.Backend, d Dialer) {
// ...
// Code here stays
ds, err := d.Dial("tcp", backend.String());if err != nil {
log.Printf("failed to dial %s: %s", backend, err)
us.Close()
return
}
// ...
}
That your production code works you now need to define a type netDialer which wraps the net.Dial() function. In your test you can use a testDialer which uses a bytes.Buffer.
This answers your question:
Emulate net connection without entering net dial
I have a go service that makes REST requests to an HTTP server that I don't control. A customer asked my to "confirm" that my service is connecting via TLS 1.2. Is that something that I can do in code?
Current code looks something like this:
request, _ := http.NewRequest("PUT",
"https://example.com/path/to/endpoint",
bytes.NewReader(json))
client := &http.Client{}
response, _ := client.Do(request)
defer response.Body.Close()
str, err := ioutil.ReadAll(response.Body)
Based on a quick read of the docs I believe I need to use a Transport and build my client using that transport. Something like this:
tr := &http.Transport{
... some options here ...
}
client := &http.Client{Transport: tr}
But I'm not sure what options I should set.
At the time of writing, Go will speak TLS 1.2 automatically if the server supports it.
tls.ConnectionState reports various negotiated TLS parameters of a connection, including the protocol version.
To get the underlying TLS connection for an HTTP client it is easiest to set the DialTLS field of the Transport to a function that establishes and remembers the connection. Once the response arrived (but before you close the response body!), call tls.Conn.ConnectionState:
package main
import (
"crypto/tls"
"fmt"
"log"
"net"
"net/http"
)
func main() {
var (
conn *tls.Conn
err error
)
tlsConfig := http.DefaultTransport.(*http.Transport).TLSClientConfig
c := &http.Client{
Transport: &http.Transport{
DialTLS: func(network, addr string) (net.Conn, error) {
conn, err = tls.Dial(network, addr, tlsConfig)
return conn, err
},
},
}
res, err := c.Get("https://example.com")
if err != nil {
log.Fatal(err)
}
versions := map[uint16]string{
tls.VersionSSL30: "SSL",
tls.VersionTLS10: "TLS 1.0",
tls.VersionTLS11: "TLS 1.1",
tls.VersionTLS12: "TLS 1.2",
}
fmt.Println(res.Request.URL)
fmt.Println(res.Status)
v := conn.ConnectionState().Version
fmt.Println(versions[v])
res.Body.Close()
}
// Output:
// https://example.com
// 200 OK
// TLS 1.2
From the docs
Package tls partially implements TLS 1.2, as specified in RFC 5246.
That beeing said I keep this function as a snippet to create the necessary configuration:
func NewTLSConfig(clientCertFile, clientKeyFile, caCertFile string) (*tls.Config, error) {
tlsConfig := tls.Config{}
// Load client cert
cert, err := tls.LoadX509KeyPair(clientCertFile, clientKeyFile)
if err != nil {
return &tlsConfig, err
}
tlsConfig.Certificates = []tls.Certificate{cert}
// Load CA cert
caCert, err := ioutil.ReadFile(caCertFile)
if err != nil {
return &tlsConfig, err
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
tlsConfig.RootCAs = caCertPool
tlsConfig.BuildNameToCertificate()
return &tlsConfig, err
}
After that you just need to initialize the transport:
transport := &http.Transport{TLSClientConfig: tlsConfig}
client := &http.Client{Transport: transport}