In the "net/http" package I can cache the DNS lookups by:
client := &http.Client{
Transport: &http.Transport{
Dial: (&nett.Dialer{
Resolver: &nett.CacheResolver{TTL: 5 * time.Minute},
IPFilter: nett.DualStack,
}).Dial,
},
}
then use client to retrieve websites. How do I cache the DNS lookups for the net package? for instance, a reverse DNS request:
net.LookupAddr(ip)
Since this does not use a variable I am confused as to how to get it setup and how to even know if I am using a cached instance.
The nett package seems to just have a single "Resolve" method rather than LookupAddr, LookupIP etc etc that the official net package has. So reverse lookups don't seem to be available. Here's how to do a normal lookup of address from name
package main
import (
"github.com/abursavich/nett"
"time"
)
func main() {
r := nett.CacheResolver{TTL: 5 * time.Minute}
a, _ := r.Resolve("muppet.com")
for _, i := range a {
print(i.String())
}
}
It would seem the behavior of the net.LookupAddr is to use the host resolver. The way I do this for my services in prod is to run dnsmasq on the hosts so the DNS lookups are cached per host. The docs do mention you can customize the behavior by implementing a custom Resolver as you did: https://golang.org/pkg/net/#LookupAddr
But I think the piece you're looking for is (from the top of that doc page):
DefaultResolver is the resolver used by the package-level Lookup functions and by Dialers without a specified Resolver.
Related
I am trying to write a reverse proxy in Golang using net/httputil/ReverseProxy that is able to forward requests to different target URLs without appending the proxy URL subpaths.
For example:
If my proxy URL is PROXYURL, (not enough reputation to post more than 8 links, even fake example ones, so hence replacing link with PROXYURL),
I would like PROXYURL/app1 to forward requests to TARGET1/directory and I would like PROXYURL/app2 to forward to TARGET2/directory2
Currently, I create the following ReverseProxy http handlers using NewSingleHostReverseProxy() and bind them to the desired subpaths (/app1 and /app2).
import (
"fmt"
"net/http"
"net/http/httputil"
)
func main() {
port := "6666"
proxy1 = httputil.NewSingleHostReverseProxy("http://target1.com/directory")
proxy2 = httputil.NewSingleHostReverseProxy("http://target2.com/directory2")
http.Handle("/app1", proxy1)
http.Handle("/app2", proxy2)
http.ListenAndServe(fmt.Sprintf(":%s", port), nil)
}
However, whenever I run this and send a request to PROXYURL/app1, it proxies TARGET1/directory/app1. I understand that from the description of ReverseProxy, this is the intended behavior (appending the subpath of the proxy URL to the target). However, I was wondering if it is possible to map a proxy URL subpath (/app1) to another target URL without the subpath being appended to the target URL.
In summary, I want
http://proxy.com/app1 -> http://target1.com/directory
and
http://proxy.com/app2 -> http://target2.com/directory2
not (as is currently happening)
http://proxy.com/app1 -> http://target1.com/directory/app1
and
http://proxy.com/app2 -> http://target2.com/directory2/app2
I found that by providing a custom Director function instead of using the default one provided by NewSingleHostReverseProxy() (https://golang.org/src/net/http/httputil/reverseproxy.go) , I was able to implement the behavior I wanted. I did this by setting req.URL.Path to target.Path and not appending the original req.URL.Path. Other than that, the director function was very similar to that in reverseproxy.go.
I'm using gin gonic as an HTTP framework and I need to group some paths with shared variable by like this:
ur := r.Group("/")
ur.Use(package.Prepare)
{
ur.GET("/", package.Home)
}
Inside the Prepare handler, I declare package variable like
package.tplPath because I want all sub routes can access to this variable instead of rewrite the code in each http handler.
var tplPath = ""
func Prepare(c *gin.Context) {
_, filename, _, _ := runtime.Caller(0)
s := helper.RelativeFilePath(filename)
tplPath = s[1:len(s)] + "template/"
}
I don't know how Go works with each process, and variables for each http request. If a variable was declared in package level, will it be set after each http request?
Is this considered good practice? If not, why not?
This is not threadsafe, package variables are shared across all goroutines, and modification in one routine will change the value in all the others, causing a data race.
In general; try and avoid using package level variables where possible.
Edit:
In go, packages are a kind of module. There exists only one instance of a package for any given import path (basically package name). This means that there is only 1 instance of any variable at package level.
Package variables are shared, global state. All accessors of that variable will be accessing the exact same memory.
It does not matter what type the package variable is, struct / string / int etc. If it is defined at package level, all accessors of that variable will share the same instance of it.
If (as with http servers) you have concurrent code, there will be multiple accessors of that variable at the same time. In some situations this is fine, like when that variable is only ever read, but it appears that in your case it will be modified. Making this code racy!
The tplPath is a global variable, all routine will access the same memory address, and set after each http request.
If you want set just once, and the tplPath not depend on the http request. You can set it in the init function.
func init(){
_, filename, _, _ := runtime.Caller(0)
s := helper.RelativeFilePath(filename)
tplPath = s[1:len(s)] + "template/"
}
The init function will run before main, and just do once.
How do I change the IP address of the DNS server?
In situation, I set Google DNS server in Windows Network Settins.
And I use LookupTXT function in Golang for getting DNS txt request.
But LookupTXT parameter is just the query string.
Any help or pointers would be highly appreciated. Thanks!
This is not straigtforward to do using golang at this point. You can however use a third party DNS package that allows configuring the resolver. First install the package:
go get github.com/bogdanovich/dns_resolver
Here is an example using it and the google resolvers 8.8.8.8 and 8.8.4.4:
package main
import (
"log"
"github.com/bogdanovich/dns_resolver"
)
func main() {
resolver := dns_resolver.New([]string{"8.8.8.8", "8.8.4.4"})
// In case of i/o timeout
resolver.RetryTimes = 5
ip, err := resolver.LookupHost("google.com")
if err != nil {
log.Fatal(err.Error())
}
log.Println(ip)
// Output [216.58.192.46]
}
Source
There is an open issue in golang here, so hopefully it becomes easier to do it with the builtin net package: https://github.com/golang/go/issues/12503. It could just be a documentation problem, as it is possible now, I just can't find an example.
EDIT: actually that package only supports lookupHost: https://github.com/bogdanovich/dns_resolver/blob/master/dns_resolver.go#L51-L79
So a PR would be required to add a TXT resolver.
2nd Edit: I made a PR with txt lookup here. That project hasn't been touched in years though so it may never get accepted.
I have a website, which is composed by three smaller 'independent' subsites:
mysite
index.html
favicons
images
doc
index.html
css
img
js
...
editor
index.html
images
js
src
...
Where doc is a site created with Hugo :: A fast and modern static website engine, editor is the mxgraph Graphditor example; and the remaining files make a hand-made landing page.
Besides deploying to any web server, I'd like to distribute the site as an 'standalone application'. To allow so, I wrote this really simple server in go:
package main
import (
flag "github.com/ogier/pflag"
"fmt"
"net/http"
"net/http/httputil"
"os"
"path/filepath"
)
var port = flag.IntP("port", "p", 80, "port to serve at")
var dir = flag.StringP("dir", "d", "./", "dir to serve from")
var verb = flag.BoolP("verbose", "v", false, "")
func init() {
flag.Parse();
}
type justFilesFilesystem struct {
fs http.FileSystem;
}
type neuteredReaddirFile struct {
http.File
}
func (fs justFilesFilesystem) Open(name string) (http.File, error) {
f, err := fs.fs.Open(name)
if err != nil { return nil, err; }
return neuteredReaddirFile{f}, nil
}
func (f neuteredReaddirFile) Readdir(count int) ([]os.FileInfo, error) {
return nil, nil;
}
func loggingHandler(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requestDump, err := httputil.DumpRequest(r, true)
if err != nil { fmt.Println(err); }
fmt.Println(string(requestDump))
h.ServeHTTP(w, r)
})
}
func main() {
str, err := filepath.Abs(*dir)
if err != nil { os.Exit(1); }
fmt.Printf("Serving at port %d from dir %s\n\n",*port,str)
http.ListenAndServe(fmt.Sprintf(":%d",*port), loggingHandler(http.FileServer(justFilesFilesystem{http.Dir(*dir)})))
}
As a result, I can run simpleserver -d <path-to-mysite> and browse the sites through localhost, localhost/doc and localhost/editor.
Then, I'd like to use custom (sub)domain(s) such as mylocal.app, doc.mylocal.app and editor.mylocal.app. So, I added the following line to my the /etc/hosts file: 127.0.0.1 mylocal.app. Therefore, I can browse mylocal.app, mylocal.app/editor and mylocal.app/doc. Moreover, I was able to change it to mylocal.app, mylocal.app:<editor-port> and mylocal.app:<doc-port> with different packages.
However, when I try to use a subdomain, it is not properly resolved, so any reverse-proxy strategy won't work. Since wildcards are not supported, I can add additional entries in the /etc/hosts file, but I'd prefer to avoid it.
Although, an alternative solution is to run dnsmasq, I'd like to keep the application standalone. I found some equivalent golang packages. However, I feel that many features are supported which I don't really need.
Furthermore, since I don't really have to resolve any IP, but to provide an alias of localhost, I think that a proxy could suffice. This would also be easier to configure, since the user could configure the browser only, and no system-wide modification would be required.
Yet, all the traffic from the user would be 'filtered' by my app. Is this correct? If so, can you point me to any reference to implement it in the most clean way. I know this is quite subjective, but I mean a relatively short (say 10 lines of code) snippet so that users can easily check what is going on.
EDIT
I'd like to use something like:
func main() {
mymux := http.NewServeMux()
mymux.HandleFunc("*.mylocal.app", myHandler)
mymux.HandleFunc("*", <systemDefaultHandler>)
http.ListenAndServe(":8080", mymux)
}
or
func main() {
mymux := http.NewServeMux()
mymux.HandleFunc("editor.mylocal.app", editorHandler)
mymux.HandleFunc("doc.mylocal.app", docHandler)
mymux.HandleFunc("*.mylocal.app", rootHandler)
mymux.HandleFunc("*", <systemDefaultHandler>)
http.ListenAndServe(":8080", mymux)
}
These are only snippets. A complete example is this, which was referenced in the comments by #Steve101.
However, at now, I don't know what systemDefaultHandler is. And that is not solved there.
Apart from that, #faraz suggested using goproxy. I think that the HTTP/HTTPS transparent proxy is the default handler I am looking for. But, using a package only to do that seems excessive to me. Can I achieve the same functionality with built-in resources?
Unfortunately, there's no dead simple way to do this through Go. You'll need to intercept your system's DNS requests just like dnsmasq, and that's inevitably going to require some modification the system DNS config (/etc/resolv.conf in Linux, /etc/resolver on a Mac, or firewall rule) to route your DNS requests to your app. Going the DNS has the downside that you'd need to build a DNS server inside your app similar to pow.cx, which seems unnecessarily complicated.
Since mucking with system config is inevitable, I'd vote for making changes to the hosts file on boot or when a directory is added/removed (via fsnotify.) On shutdown, you can clear the added entries too.
If you're looking to isolate these changes to a specific browser instead of make a system-wide change, you could always run your application through a proxy server like goproxy and tell your browser to use that proxy for requests. For example, you can do this in Chrome through its preferences or by setting the --proxy-server flag:
--proxy-server=<scheme>=<uri>[:<port>][;...] | <uri>[:<port>] | "direct://"
See Chrome's network settings docs for more details.
Also, if you're willing to much with browser configs, you could just use an extension to handle the requests as needed.
there is only solution that would work without proxy is that you register an domain for this and make an offical dns entry with wildcard to 127.0.0.1 like *.myApp.suche.org. So any client that resolve the ip get 127.0.0.1 and work local. this way you do not need administrator rights to modify the /etc/hosts or equivalent.
If it should work behind proxy without modify the resolver (etc/hosts etc) you can provide an wpad.dat (Javascript for Proxy detection) that say for your domain all traffic goes you server and the rest to the real proxy. If this is served in you server the script can automaticaly contain the real proxy as default.
I am creating a restful api in GO and every method essentially interacts with the database. The specific statement that I use to open a database connection is
db,err := sql.Open("postgres", "user=postgres password=password dbname=dbname sslmode=disable")
if err != nil {
log.Fatal(err)
println(err)
}
It is very simple but the issue is that once I want to change something inside that statement then I have to change it for all other methods that have that statement . I am trying to do a dependency injection or something of that nature so that I can have that statement or value in 1 place and just reference it. I am getting an import cycle not allowed error though like Import cycle not allowed . This is my project structure
What I have done is that in the Config.go I have written this
package Config
const Connect = "user=postgres password=password dbname=dbname sslmode=disable"
Then in the Listings.go I put this
package Controllers
import (
"net/http"
"database/sql"
"../Config"
)
func Listing_Expiration(w http.ResponseWriter, r *http.Request) {
db,err := sql.Open("postgres",Config.Connect)
if err != nil {
log.Fatal(err)
println(err)
}
notice I have the import ../Config and the Config.Connect but when I compile that I get import cycle not allowed . I have been trying to solve this issue but haven't been able to.
Yes, Go doesn't allow to have cycled imports. In your example you have 2 packages Config and Controllers. When you build a code, Controllers package requires Config package, then Config requires Controllers and it's endless. You should refactor your code to make Config package separated from Controllers, and only used by it. Also, you can make some common package, imported to Controllers and Config.
I got the same error. But in my case, I imported the package itself inside the file. so you can check if you did the same mistake.
Go does NOT allow import cycles. So you need to fix it. In your case, Controller and Config are importing each other, you are not allowed to do that.
I wrote a detailed blog about how you can deal with it. Refer https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them
Use interfaces.
Making use of go:linkname.
Though, I can't see any import cycle from your code.
Go don't support the import cycle. Based on your question, you must import the controller from the Config Package. You need to remove one of them and refactor your code.
Fore more: https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them