I am trying to get DNS requests to tunnel through my company's proxy by using the HTTP CONNECT method.
The library I am using to make the DNS requests supports setting a custom net.Dialer on an instance of dns.Client. I found another library that claims to act as a drop-in replacement for net.Dialer that supports initiating TCP connections through a proxy using the HTTP CONNECT method.
However, I can't figure out how to get this to work with these two libraries.
I have tried setting the Dialer field of dns.Client, but it complains about incompatible types:
client := &dns.Client{
Net: "tcp",
Timeout: time.Duration(forwarder.Timeout) * time.Second,
}
if forwarder.Proxy != nil {
client.Dialer = http_dialer.New(forwarder.Proxy, http_dialer.WithDialer(client.Dialer))
}
Yields:
cannot use http_dialer.New(forwarder.Proxy, http_dialer.WithDialer(client.Dialer)) (type *http_dialer.HttpTunnel) as type *net.Dialer in assignment
So I tried casting it:
client := &dns.Client{
Net: "tcp",
Timeout: time.Duration(forwarder.Timeout) * time.Second,
}
if forwarder.Proxy != nil {
client.Dialer = net.Dialer(*http_dialer.New(forwarder.Proxy, http_dialer.WithDialer(client.Dialer)))
}
But that yields:
cannot convert *http_dialer.New(forwarder.Proxy, http_dialer.WithDialer(client.Dialer)) (type http_dialer.HttpTunnel) to type net.Dialer
Finally, I tried setting the Dial function of dns.Client.Dialer to the Dial function in the http_dialer.HttpTunnel returned by http_dialer#New:
client := &dns.Client{
Net: "tcp",
Timeout: time.Duration(forwarder.Timeout) * time.Second,
}
if forwarder.Proxy != nil {
client.Dialer.Dial = http_dialer.New(forwarder.Proxy, http_dialer.WithDialer(client.Dialer)).Dial
}
But that yielded:
cannot assign to client.Dialer.Dial
So I how do I set the Dialer of my DNS client?
Related
I'm using Fiber as an HTTP client to make some requests to an http server, however I'm being rate limited. On my vm I configured 5 different IP addresses (public/private) and have confirmed that they are indeed connected to the internet.
curl --interface 10.0.0.4 ipinfo.io/json
curl --interface 10.0.0.5 ipinfo.io/json
...
curl --interface 10.0.0.8 ipinfo.io/json
each one returns a different public facing ip address.
Now I'm interested in making round-robin requests using these local addresses but I'm not so sure how to go about it.
Is there some sort of property or function I can set/call to change where the outgoing request is coming from?
I've looked around at fasthttp.HostClient which fiber.Agent extends but I didn't see anything useful.
Thanks guys.
a := fiber.AcquireAgent()
req := a.Request()
req.Header.SetMethod(fiber.MethodGet)
req.SetRequestURI(fmt.Sprintf(formatUrl, args...))
if err := a.Parse(); err != nil {
h.Logger.Error("%v", err)
return fiber.StatusInternalServerError, nil, []error{err}
}
customDialer := fasthttp.TCPDialer{
Concurrency: 1000,
LocalAddr: &net.TCPAddr{
IP: h.IPPool[atomic.AddUint32(&h.IpIdx, 1)%uint32(len(h.IPPool))],
},
}
a.HostClient.Dial = func(addr string) (net.Conn, error) {
return customDialer.Dial(addr)
}
Creating a custom dialer and dial func allows you to change the local address associated with the http request.
I am using Colly to scrape a website and I am trying to also get the TLS certificate that the site is presenting during the TLS handshake. I looked through the documentation and the response object but did not find what I was looking for.
According to the docs, I can customize some http options by changing the default HTTP roundtripper. I tried setting custom GetCertificate and GetClientCertificate functions, assuming that these functions would be used during the TLS handshake, but the print statements are never called.
// Instantiate default collector
c := colly.NewCollector(
// Visit only domains: hackerspaces.org, wiki.hackerspaces.org
colly.AllowedDomains("pkg.go.dev"),
)
c.WithTransport(&http.Transport{
TLSClientConfig: &tls.Config{
GetCertificate: func(ch *tls.ClientHelloInfo) (*tls.Certificate, error) {
fmt.Println("~~~GETCERT CALLED~~")
return nil, nil
},
GetClientCertificate: func(cri *tls.CertificateRequestInfo) (*tls.Certificate, error) {
fmt.Println("~~~GETCLIENTCERT CALLED~~")
return nil, nil
},
},
})
Please help me scrape TLS certificates using Colly.
This is a snippet to get leaf certificate from raw http.Response in case you give up getting certificate using Colly.
tls := ""
if res.TLS != nil && len(res.TLS.PeerCertificates) > 0 {
cert := res.TLS.PeerCertificates[0]
tls = base64.StdEncoding.EncodeToString(cert.Raw)
}
I am a new golang learner, I have created a simple program that sends many requests per second, in fact it works very well until I decide to use proxies, I get many errors but one of them confuses me
proxyconnect tcp: dial tcp 103.126.217.129:8080: connectex: No connection could be made because the target machine actively refused it. I read that something might be blocking the proxy connection, so I turned off my firewall and unfortunately that didn't make a difference.
I didn't know what I should do to fix this and I hope to get some help from here
MyTransport
MyClient = &http.Transport{
MaxIdleConnsPerHost: 9216,
MaxIdleConns: 0,
MaxConnsPerHost: 0,
DisableKeepAlives: true,
IdleConnTimeout: 20 * time.Second,
TLSHandshakeTimeout: 20 * time.Second,
DialContext: (&net.Dialer{
Timeout: 20 * time.Second,
KeepAlive: 20 * time.Second,
DualStack: true,
}).DialContext,
}
MyClient.Proxy = http.ProxyURL(&url.URL{
Host: hostport,
Scheme: "http",
})
MyClient.Dial = func(network, addr string) (net.Conn, error) {
TCPConnection, err := net.Dial(network, addr)
if err != nil {
return nil, err
}
TCPConnection.(*net.TCPConn).SetKeepAlive(false)
return TCPConnection, err
}
So your issue is going to be on the side you are subscribing to, most likely server side. The script is clearly working otherwise it wouldn't be coming back with an error that it is being refused from the server.
When I shut off my websocket server, I get the same error on my client side.
Consider connecting to the server via a different script to validate or use a programming language your more familiar with until you know your server is working as intended.
Also make sure your server is not HTTPS or wanting encryption headers or something.
I wrote a sample gRPC client a server in Go, both configured for server-authenticated TLS.
The client gRPC call succeeds, giving me the impression the TLS is configured properly, otherwise if the TLS handshake had failed, I would expect the client to fail and not make the gRPC request (i.e. not default to plaintext).
Yet I am puzzled by a result I obtain when I attach Wireshark to that network to sniff TCP packets. I do not see any packet with TLS, for e.g. I do not see the TLS CLIENT HELLO packet.
So is this because I'm misinterpreting what I see in Wireshark, or is my gRPC client actually doing plaintext gRPC?
The client code looks like this, note the grpc.withTransportCredentials which I think means it will use TLS or fail, but never plaintext:
// block the dial until connection is successful or 3 sec timeout
dialOptions := []grpc.DialOption{
grpc.WithBlock(),
grpc.WithTimeout(3 * time.Second),
}
// Load TLS Configuration
tlsCredentials, err := LoadTLSCredentials()
if err != nil {
log.Fatalf("Failed to load TLS credentials: %v", err)
}
dialOptions = append(dialOptions, grpc.WithTransportCredentials(tlsCredentials))
// Dial the gRPC server
log.Printf("Dialing %v", *address)
conn, err := grpc.Dial(*address, dialOptions...)
if err != nil {
log.Fatalf("Failed to connect to the server: %v", err)
}
defer conn.Close()
// then this application sets up a gRPC request, and logs the response to stdout,
// in my testing stdout shows the expected gRPC response, so I'd assume TLS is working.
func LoadTLSCredentials() (credentials.TransportCredentials, error) {
rootCA, err := ioutil.ReadFile("ca.cert")
if err != nil {
return nil, err
}
certPool := x509.NewCertPool()
if !certPool.AppendCertsFromPEM(rootCA) {
return nil, fmt.Errorf("Failed to add rootCA to x509 certificate pool")
}
config := &tls.Config{
MinVersion: tls.VersionTLS12,
RootCAs: certPool,
}
return credentials.NewTLS(config), nil
}
And here's a screenshot of Wireshark showing no TLS packet
whereas I would expect something similar to the following which clearly shows some TLS activity (not my app, image is from the web for illustration purposes)
I'm running Wireshark v2.6.10 on Ubuntu 16.04. The source and destination IPs match my gRPC client and server IPs (both are docker containers on the same docker network).
Not that it really matters, but as can be seen in my client code, I'm sharing a root CA certificate on the client (self signed). I can do this because I deploy both the client and the server.
As #steffanUllrich explained in the comments, this was a case of Wireshark can be better configured to show TLS. I confirmed the gRPC exchange is indeed TLS protected.
You should right click the packet list, and select 'decode as..' menu item, then select 'tls' to force wireshark dissect traffic in this tcp port as TLS.
I have been trying to set up a WebSocket connection over TLS (so with encryption). I use Golang with Gorilla. A WebSocket connection is implemented as an initial HTTP connection that gets upgraded to the WebSocket protocol connection. The code is like this:
func wsEndpoint(w http.ResponseWriter, r *http.Request) {
// upgrade
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
//...
}
log.Println("Client Connected")
err = ws.WriteMessage(1, []byte("Hi Client!"))
if err != nil {
//...
}
// listen indefinitely for new messages coming
}
Then we set up the routing:
func main() {
//...
setupRoutes()
log.Fatal(http.ListenAndServe(":8080", nil))
}
Does it suffice to change the last line to:
...http.ListenAndServeTLS(...)
so in other words to use TLS to establish the first connection?
Does this approach suffice to secure the entire communication over WebSocket from start till the end? Should I be certain that all packet transmission within the connection duration is also protected by TLS? If not, how to set it up in Golang / Gorilla framework?
Use http.ListenAndServeTLS to encrypt the underlying network connections used for the HTTP protocol and the WebSocket protocol.
The approach secures the entire communication on the underlying network connection including all WebSocket traffic.
The Gorilla server code uses the network connection provided by the net/http server. The Gorilla server code does create new network connections.
http.ListenAndServeTLS is a helper function that calls lower-level functions and methods. It also works to call those lower-level functions and methods directly.
One approach to this problem would be to set up a reverse proxy like nginx with certbot to generate certificates.
Here's how it would work
[ Client ] ----------> [ nginx ] --------------> [ golang server ]
Encrypted Not encrypted