I've been working on finding a way to try and override the default DNS server for a Go program for a while but still with no success sadly.
The current solution which I had thought would work was:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
DNS := "1.1.1.1"
net.DefaultResolver = &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
d := net.Dialer{
Timeout: time.Millisecond * time.Duration(3000),
}
return d.DialContext(ctx, "udp", fmt.Sprintf("%s:53", DNS))
},
}
resp, err := net.LookupHost("tsdfsdf.com")
if err != nil {
fmt.Printf(err.Error())
}
fmt.Printf("%+v", resp)
}
But the response from this is:
lookup tsdfsdf.com on 192.168.0.1:53: no such host[]
Which is sadly my default DNS server set in my reslov.conf
I have tried forcing to use the Go Resolver by setting export GODEBUG=netdns=go
The long term solution is to be able to over the default resolver for the HTTP Client which would, in turn be consumed by some AWS SDK stuff.
Has any faced this or knows how I can get around this?
FYI, I know "tsdfsdf.com" is not a real domain I'm just using it to spit a No such host error to see what DNS it asked.
Related
I'm trying to dial using credentials and maintain a connection with a socks5 proxy server in Go.
This works nicely if I have IP authorisation set up with the proxy provider, however there is no way pass any auth credentials using net.Dial function in Go:
package main
import (
"io"
"net"
)
func main() {
dst, err := net.Dial("tcp", "11.22.33.44:1111")
if err != nil {
panic("Dial Error:" + err.Error())
}
dst.Close()
}
Go has a useful proxy library and allows authenticated forward requests via proxy using this:
package main
import (
"io"
"net"
)
func main() {
var proxyAuth *proxy.Auth
if conf.Username != "" {
proxyAuth = new(proxy.Auth)
proxyAuth.User = conf.Username
proxyAuth.Password = conf.Password
}
proxyconn, _ := proxy.SOCKS5("tcp", "11.11.11.11:1111", proxyAuth, nil) //returns a Dialer with proxy that can be invoked to connect to another address
dst := proxyconn.Dial("tcp", "22.33.44.55:6666") //connects to an address via proxy
dst.Close()
}
However it returns a Dialer that then asks to connect a target/ultimate address through this authenticated proxy rather the proxy server itself:
My objective here is to return a net.conn connection with a credentials-authenticated proxy server - something like this:
package main
import (
"io"
"net"
)
func main() {
//net.Dial does not have a way to pass Auth creds
dst := net.Dial("tcp", "22.33.44.55:6666", proxyAuth)
dst.Close()
}
The net.Dial() method doesn't concerned with proxy authentication. However, If you want proxy authentication you can set it in header of the request before the call. Please refer this link
dst := net.Dial("tcp", "22.33.44.55:6666")
go version: go1.13.5 linux/amd64
I am using "x/net/proxy" to connect with the "http_proxy".
I have referred following proxy page:
https://godoc.org/golang.org/x/net/proxy
To get proxy information I have set environment variable "all_proxy" to the the desired proxy "http://192.130.0.10:3200", and performed the tcp connection, but following error is raised:
[Network Error : socks connect tcp 192.130.0.10:3200->mx.eu1.mico.io:8883: read tcp 172.17.0.2:48118->192.130.0.10:3200: read: connection reset by peer]
I have looked "x/net/proxy", It seems "http_proxy" support is not available instead of "SOCKS5" proxy is supported. I have similar implementation for "http_proxy", but unfortunately it does not worked.
I have created a sample code (with port 1883) which is working for the non proxy environment, Please suggest how I can enable "http_proxy" or "https_proxy" support?
package main
import (
"fmt"
"os"
"golang.org/x/net/proxy"
)
//The host address which we want to connect with the proxy
var host = "google.com:80"
func main() {
fmt.Println("Inside main...")
//Setting the proxy before starting the application
if os.Getenv("http_proxy") == "" {
os.Setenv("http_proxy", "http://192.130.0.10:3200")
}
os.Setenv("all_proxy", os.Getenv("http_proxy"))
if os.Getenv("all_proxy") != os.Getenv("http_proxy") {
fmt.Println("Environment variables are not matching...")
return
}
fmt.Println("System proxy is:", os.Getenv("all_proxy"))
proxyDialer := proxy.FromEnvironment()
fmt.Println("Connecting to...", host)
conn, err := proxyDialer.Dial("tcp", host)
if err != nil {
fmt.Println("Unable to dial...", err)
return
}
fmt.Println("Connected...", conn)
}
Output:
Inside main...
System proxy is: http://192.130.0.10:3200
Connecting to... google.com:80
Unable to dial... dial tcp 172.217.23.174:80: connect: connection timed out
What's your purpose?
If you need to use an http-proxy server for http requests, you can just configure your http-client without another package usage:
package main
import (
"fmt"
"net/http"
"net/url"
"time"
)
func main() {
proxyUrl, err := url.Parse("http://192.130.0.10:3200")
if err != nil {
// TODO handle me
panic(err)
}
cl := http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
},
Timeout: 3000 * time.Millisecond,
}
resp, err := cl.Get("http://google.com")
if err != nil {
// TODO handle me
panic(err)
}
// TODO work with the response
fmt.Println(resp)
}
What I want to achieve:
An HTTPS server designed specifically to serve binaries to around 1000 devices, sometimes in the same time (clients will fetch the binaries via wget, curl, browser download, etc).
Key functionality features:
client won't be able to download the binary without a certificate
server will allow the client directory browsing/download via browser(if client has certificate)
server is optimized for stability and security, then for speed
server must use high security ciphers and TLS1.2
What I managed to achieve
package main
import (
"log"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
http.ServeFile(w, req, "/")
})
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(http.ListenAndServeTLS(":9900", "cert.crt", "priv.key", http.FileServer(http.Dir("/"))))
}
Now, this works fine although it doesn't check all the features and its not very flexible and somehow I wanted to make it more future proof, as I wish to both learn by creating this project and also expand on it in the future as I'm interested in learning more about servers.
After a bit of research I found several code pieces on GitHub and in tutorials, which led me to put together the following piece of code:
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
http.ServeFile(w, req, "/")
})
cfg := &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
srv := &http.Server{
Addr: ":9900",
Handler: mux,
TLSConfig: cfg,
TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
}
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(srv.ListenAndServeTLS("cert.crt", "priv.key"), http.FileServer(http.Dir("/")))
}
Problem is that when the server starts and I connect to it via browser, I'm presented with the root directory but every time I click on a folder the URL increments the address but the page just refreshes in the "/" directory.
To be exact:
I connect initially to the server and I'm presented the root directory , I'm shown Dir1, Dir2, Dir3
I click Dir1
The URL modifies from https://localhost:9900 to https://localhost:9900/Dir1
But I'm still in the root directory
From what I'm able to see...I think I'm creating a loop somewhere.
If anyone knows what I need to do to make this functional, help would be much appreciated.
NOTE
The above behavior is on Firefox, on Chrome I get one of the 2 errors in the server error log, depending on changes that I make:
2019/09/29 19:59:37 http: TLS handshake error from [::1]:53287: EOF
2019/09/29 19:15:59 http: TLS handshake error from [::1]:50457: tls: client doesn't support selected certificate
There are several examples elsewhere on how to do this (as has been commented).
Here's a worked example in which the fileserver handler is an enhanced version of that in the standard library that supports more cache headers and locally-compressed files. The standard fileserver handler can be dropped in here instead if that's what you need.
package main
import (
"crypto/tls"
"flag"
"fmt"
"log"
"net/http"
"time"
"github.com/rickb777/servefiles/v3"
)
var path = flag.String("path", "..", "directory for the files tp be served")
var cert = flag.String("cert", "", "file containing the certificate (optional)")
var key = flag.String("key", "", "file containing the private key (optional)")
var port = flag.Int("port", 8080, "TCP port to listen on")
var maxAge = flag.String("maxage", "", "Maximum age of assets sent in response headers - causes client caching")
var verbose = flag.Bool("v", false, "Enable verbose messages")
func main() {
flag.Parse()
if *verbose {
servefiles.Debugf = log.Printf
}
if (*cert != "" && *key == "") ||
(*cert == "" && *key != "") {
log.Fatal("Both certificate file (-cert) and private key file (-key) are required.")
}
h := servefiles.NewAssetHandler(*path)
if *maxAge != "" {
d, err := time.ParseDuration(*maxAge)
log.Printf("MaxAge: %s %v\n", d, err)
h = h.WithMaxAge(d)
}
srv := &http.Server{
Addr: fmt.Sprintf(":%d", *port),
Handler: h,
}
if *cert != "" {
srv.TLSConfig = &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
log.Printf("Access the server via: https://localhost:%d/", *port)
log.Fatal(srv.ListenAndServeTLS(*cert, *key))
} else {
log.Printf("Access the server via: http://localhost:%d/", *port)
log.Fatal(srv.ListenAndServe())
}
}
Source code https://github.com/rickb777/servefiles/blob/master/v3/webserver/example.go
I tried looking it up, but couldn't find an already answered question.
How do I get the host part of the URL in Go?
For eg.
if the user enters http://localhost:8080 in the address bar, I wanted to extract "localhost" from the URL.
If you are talking about extracting the host from a *http.Request you can do the following:
func ExampleHander(w http.ResponseWriter, r *http.Request) {
host := r.Host // This is your host (and will include port if specified)
}
If you just want to access the host part without the port part you can do the following:
func ExampleHandler(w http.ResponseWriter, r *http.Request) {
host, port, _ := net.SplitHostPort(r.Host)
}
For the last one to work you also have to import net
Go has wonderful documentation, I would also recommend taking a look at that: net/http
Go has built in library that can do it for you.
package main
import "fmt"
import "net"
import "net/url"
func main() {
s := "http://localhost:8080"
u, err := url.Parse(s)
if err != nil {
panic(err)
}
host, _, _ := net.SplitHostPort(u.Host)
fmt.Println(host)
}
https://golang.org/pkg/net/url/#
I'm developing an api for blog or online publishing website to develop a recommendation engine for their content.
Since my api returns same json for same url request, I decided to use Redis as a cache for high traffic websites by passing the url as key and json as value. I am developing this api in go-lang recently and have been using redigo to talk to our Redis instance. The way I decided to architect my system is to check the url of the query sent by the client (blog) and search for it in redis. If however, the url response in not cached I do a 301 redirect to another api that applied the logic to generate the json response for that particular url and also set the redis cache. However, while I'm testing if my Redis is working properly, I realised that it is missing cache far too often than what I would like. It's definitely caching the json response mapped to the url as confirmed by doing a simple GET in Redis-cli but after 3-4 hits I could see Redis missing cache again. I'm still very new to go-lang and caching world so I'm not sure if I'm missing something in my implementation. Also, I would like to know under what circumstances can Redis instance miss caches ? It can't be timeout because Redis docs says "By default recent versions of Redis don't close the connection with the client if the client is idle for many seconds: the connection will remain open forever." so I'm not sure what exactly is happening with my setup. Relevant part of my code is below:
package main
import (
"flag"
"fmt"
"github.com/garyburd/redigo/redis"
"log"
"net/http"
"time"
)
var (
port int
folder string
pool *redis.Pool
redisServer = flag.String("redisServer", "redisip:22121", "")
redisPassword = flag.String("redisPassword", "", "")
)
func init() {
flag.IntVar(&port, "port", 80, "HTTP Server Port")
flag.StringVar(&folder, "folder", "www", "Serve this folder")
}
func newPool(server, password string) *redis.Pool {
return &redis.Pool{
MaxIdle: 3,
MaxActive: 25000,
IdleTimeout: 30 * time.Second,
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", server)
if err != nil {
return nil, err
}
return c, err
},
TestOnBorrow: func(c redis.Conn, t time.Time) error {
_, err := c.Do("PING")
return err
},
}
}
func main() {
flag.Parse()
pool = newPool(*redisServer, *redisPassword)
httpAddr := fmt.Sprintf(":%v", port)
log.Printf("Listening to %v", httpAddr)
http.HandleFunc("/api", api)
http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir(folder))))
log.Fatal(http.ListenAndServe(httpAddr, nil))
}
func api(w http.ResponseWriter, r *http.Request) {
link := r.URL.Query().Get("url")
fmt.Println(link)
heading := r.URL.Query().Get("heading")
conn := pool.Get()
reply, err := redis.String(conn.Do("GET", link))
defer conn.Close()
if err != nil {
fmt.Println("Error for link %v:%v", heading, err)
http.Redirect(w, r, "json-producing-api", 301)
}
fmt.Fprint(w, reply)
}
I must also mention here that in the above code, my redis instance is actually a twemproxy client built by twitter which proxies three different redis client running behind on three different ports. Everything seemed to worked normal yesterday and I did a successful load test for 5k concurrent reuquests. However, when I checked the log today some queries were being missed by redis and were being redirected to my json-producing-api and I could see redigo:nil error. I'm totally confused as to what exactly is going wrong? Any help will be greatly appreciated.
EDIT: As per discussions below, I'm detailing the code that I use to set the data in Redis
func SetIntoRedis(key string, value string) bool {
// returns true if successfully set, returns false in case of an error
conn := pool.Get()
_, err := conn.Do("SET", key, value)
if err != nil {
log.Printf("Error Setting %v : %v", key, err)
return false
}
return true
}
Configuration of my twemproxy client
leaf:
listen: 0.0.0.0:22121
hash: fnv1a_64
distribution: ketama
redis: true
auto_eject_hosts: true
server_retry_timeout: 3000
server_failure_limit: 3
servers:
- 127.0.0.1:6379:1
- 127.0.0.1:6380:1
- 127.0.0.1:6381:1