I am trying to write a reverse proxy in Golang using net/httputil/ReverseProxy that is able to forward requests to different target URLs without appending the proxy URL subpaths.
For example:
If my proxy URL is PROXYURL, (not enough reputation to post more than 8 links, even fake example ones, so hence replacing link with PROXYURL),
I would like PROXYURL/app1 to forward requests to TARGET1/directory and I would like PROXYURL/app2 to forward to TARGET2/directory2
Currently, I create the following ReverseProxy http handlers using NewSingleHostReverseProxy() and bind them to the desired subpaths (/app1 and /app2).
import (
"fmt"
"net/http"
"net/http/httputil"
)
func main() {
port := "6666"
proxy1 = httputil.NewSingleHostReverseProxy("http://target1.com/directory")
proxy2 = httputil.NewSingleHostReverseProxy("http://target2.com/directory2")
http.Handle("/app1", proxy1)
http.Handle("/app2", proxy2)
http.ListenAndServe(fmt.Sprintf(":%s", port), nil)
}
However, whenever I run this and send a request to PROXYURL/app1, it proxies TARGET1/directory/app1. I understand that from the description of ReverseProxy, this is the intended behavior (appending the subpath of the proxy URL to the target). However, I was wondering if it is possible to map a proxy URL subpath (/app1) to another target URL without the subpath being appended to the target URL.
In summary, I want
http://proxy.com/app1 -> http://target1.com/directory
and
http://proxy.com/app2 -> http://target2.com/directory2
not (as is currently happening)
http://proxy.com/app1 -> http://target1.com/directory/app1
and
http://proxy.com/app2 -> http://target2.com/directory2/app2
I found that by providing a custom Director function instead of using the default one provided by NewSingleHostReverseProxy() (https://golang.org/src/net/http/httputil/reverseproxy.go) , I was able to implement the behavior I wanted. I did this by setting req.URL.Path to target.Path and not appending the original req.URL.Path. Other than that, the director function was very similar to that in reverseproxy.go.
Related
Using GOA, I defined a service to serve static files using a wildcard (as described in the documentation):
var _ = Service("static", func() {
Files("/static/*filepath", "./static/")
})
But when I run the service, the endpoint always retrieves all the content it finds in the ./static/ directory, it seems it doesn't take into account the wildcard section at all.
For example, if I have ./static/uploads/file1.jpg and I request localhost/static/uploads/file1.jpg or localhost/static/anything , then the service retrieves the following:
<pre>
uploads/
</pre>
Digging into the code, I believe the problem is in the generated /gen/http/static/server/server.go file:
// Mount configures the mux to serve the static endpoints.
func Mount(mux goahttp.Muxer, h *Server) {
MountCORSHandler(mux, h.CORS)
MountStatic(mux, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, "./static/")
}))
}
// MountStatic configures the mux to serve GET request made to
// "/static/*filepath".
func MountStatic(mux goahttp.Muxer, h http.Handler) {
mux.Handle("GET", "/static/*filepath", handleStaticOrigin(h).ServeHTTP)
}
For what I see, the generated code is serving what we passed as base path no matter what, it doesn't take into account if we configured a wildcard at all (it only uses it to match the request, but not to customize the file that we'll serve).
I believe this was working ok in v2, I found this issue in the process of migrating to v3.
As I said, this seems like a bug in GOA, but maybe I'm missing something here. I created an issue in the repo to obtain more information (#2321)
As per the answer in the Github issue (#2321), it seems there was an error in the docs, and we should use curly braces in the pattern:
Thank you for the report, there is a typo in the docs, the path in the design needs to be /static/{*filepath} instead (with curly braces surrounding the wildcard).
I wanted to add trace id to logging done for each request to the micro service.I want this in similar as for springboot application we can set trace id in MDC and fetch it and use it while logging.
I have done some research and I found that MDC equivalent in go lang is context. So, I have set the trace id in my context. Now the problem is where ever I have to log with trace id ,I need to pass context to that function which is very ugly way. I am looking for a better solution for this problem.
func HandlerFunction(f gin.HandlerFunc) gin.HandlerFunc{
return func(cxt *gin.Context) {
reqraceId := cxt.Request.Header.Get("trace-id")
requid , _ := uuid.NewRandom()
if reqTraceId == "" {
c.Request.Header.Set("trace-id", requid.String())
}
f(c)
}
}
It might be worth reading up on context.Context particularly this article which has a section that says:
At Google, we require that Go programmers pass a Context parameter as the first argument to every function on the call path between incoming and outgoing requests.
TL;DR - it's fine to pass the context, but what's the best way?
There's two main patterns
Ask the context to give you a logger
Give the logger the context
Context can be used to store values:
context.WithValue(ctx, someKey, someValue)
This means we can either do:
somepackage.Log(ctx).Info("hello world")
// or
sompackage.Info(ctx, "hello world")
The implementation of these two sample APIs could interact with the context to retrieve the values required with out needing to worry about the extra information that would have been in MDC at any of the logging call sites.
From my side I found that using the default log package we could set a prefix as log.SetPrefix(traceId), doing so, the log will print the trace id as the prefix in the actual and sub-functions/structs.
import (
"log"
"github.com/google/uuid"
)
func (hdl *HTTPHandler) example() {
var traceId string = uuid.NewString()
log.SetPrefix(traceId + " - ")
log.SetFlags(log.LstdFlags)
// ...
// ...
log.Println("......")
}
This issue can also be solved using a dependency injection container.
We can implement "request-scoped" injections, and as a result, for each request, we will recreate all dependency tree that uses request-scoped dependency(logger, error reporter, clients which send requests to another service with context propagation).
But as I understood using dependency injection containers is not a best practice in go and not an "idiomatic" way.
Also, this approach can have some performance and memory issues since we will recreate objects for each request.
How do I change the IP address of the DNS server?
In situation, I set Google DNS server in Windows Network Settins.
And I use LookupTXT function in Golang for getting DNS txt request.
But LookupTXT parameter is just the query string.
Any help or pointers would be highly appreciated. Thanks!
This is not straigtforward to do using golang at this point. You can however use a third party DNS package that allows configuring the resolver. First install the package:
go get github.com/bogdanovich/dns_resolver
Here is an example using it and the google resolvers 8.8.8.8 and 8.8.4.4:
package main
import (
"log"
"github.com/bogdanovich/dns_resolver"
)
func main() {
resolver := dns_resolver.New([]string{"8.8.8.8", "8.8.4.4"})
// In case of i/o timeout
resolver.RetryTimes = 5
ip, err := resolver.LookupHost("google.com")
if err != nil {
log.Fatal(err.Error())
}
log.Println(ip)
// Output [216.58.192.46]
}
Source
There is an open issue in golang here, so hopefully it becomes easier to do it with the builtin net package: https://github.com/golang/go/issues/12503. It could just be a documentation problem, as it is possible now, I just can't find an example.
EDIT: actually that package only supports lookupHost: https://github.com/bogdanovich/dns_resolver/blob/master/dns_resolver.go#L51-L79
So a PR would be required to add a TXT resolver.
2nd Edit: I made a PR with txt lookup here. That project hasn't been touched in years though so it may never get accepted.
I am running a PureScript app that is being served up by a backend Suave application in F#. In the front end, I need to open a WebSocket connection in PureScript to the backend, but part of the path needs to be dynamic based on how the backend app is running (for example on some boxes it is: ws://host1:9999/ws/blah, on others it might be ws://host2:7777/ws/blah).
So I need to get the current URL that my app is being served up on so that I can just put a ws:// on the front, and a ws/blah on the end (or somehow do a relative WebSocket path?).
I've tried doing something like:
wdw <- window
htmldoc <- document wdw
let doc = htmlDocumentToDocument htmldoc
docUrl <- url doc
connection <- WS.create (WS.URL $ "ws://" <> docUrl <> "ws/blah") []
But the document URL given has http:// on the front of it. I could hack up the string and rip that part out, but I'm hoping to find a more elegant way.
If it matters, I'm also using Halogen here so I have access to their API if there is something useful in there for this situation.
I was able to piece it together from stholzm's suggestion above.
In the documentation for location, there are functions for Hostname and Port that can be used to piece together the base url. The location can be obtained via the location function that takes in a window instance.
In the end, my code looks like
I have a website, which is composed by three smaller 'independent' subsites:
mysite
index.html
favicons
images
doc
index.html
css
img
js
...
editor
index.html
images
js
src
...
Where doc is a site created with Hugo :: A fast and modern static website engine, editor is the mxgraph Graphditor example; and the remaining files make a hand-made landing page.
Besides deploying to any web server, I'd like to distribute the site as an 'standalone application'. To allow so, I wrote this really simple server in go:
package main
import (
flag "github.com/ogier/pflag"
"fmt"
"net/http"
"net/http/httputil"
"os"
"path/filepath"
)
var port = flag.IntP("port", "p", 80, "port to serve at")
var dir = flag.StringP("dir", "d", "./", "dir to serve from")
var verb = flag.BoolP("verbose", "v", false, "")
func init() {
flag.Parse();
}
type justFilesFilesystem struct {
fs http.FileSystem;
}
type neuteredReaddirFile struct {
http.File
}
func (fs justFilesFilesystem) Open(name string) (http.File, error) {
f, err := fs.fs.Open(name)
if err != nil { return nil, err; }
return neuteredReaddirFile{f}, nil
}
func (f neuteredReaddirFile) Readdir(count int) ([]os.FileInfo, error) {
return nil, nil;
}
func loggingHandler(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requestDump, err := httputil.DumpRequest(r, true)
if err != nil { fmt.Println(err); }
fmt.Println(string(requestDump))
h.ServeHTTP(w, r)
})
}
func main() {
str, err := filepath.Abs(*dir)
if err != nil { os.Exit(1); }
fmt.Printf("Serving at port %d from dir %s\n\n",*port,str)
http.ListenAndServe(fmt.Sprintf(":%d",*port), loggingHandler(http.FileServer(justFilesFilesystem{http.Dir(*dir)})))
}
As a result, I can run simpleserver -d <path-to-mysite> and browse the sites through localhost, localhost/doc and localhost/editor.
Then, I'd like to use custom (sub)domain(s) such as mylocal.app, doc.mylocal.app and editor.mylocal.app. So, I added the following line to my the /etc/hosts file: 127.0.0.1 mylocal.app. Therefore, I can browse mylocal.app, mylocal.app/editor and mylocal.app/doc. Moreover, I was able to change it to mylocal.app, mylocal.app:<editor-port> and mylocal.app:<doc-port> with different packages.
However, when I try to use a subdomain, it is not properly resolved, so any reverse-proxy strategy won't work. Since wildcards are not supported, I can add additional entries in the /etc/hosts file, but I'd prefer to avoid it.
Although, an alternative solution is to run dnsmasq, I'd like to keep the application standalone. I found some equivalent golang packages. However, I feel that many features are supported which I don't really need.
Furthermore, since I don't really have to resolve any IP, but to provide an alias of localhost, I think that a proxy could suffice. This would also be easier to configure, since the user could configure the browser only, and no system-wide modification would be required.
Yet, all the traffic from the user would be 'filtered' by my app. Is this correct? If so, can you point me to any reference to implement it in the most clean way. I know this is quite subjective, but I mean a relatively short (say 10 lines of code) snippet so that users can easily check what is going on.
EDIT
I'd like to use something like:
func main() {
mymux := http.NewServeMux()
mymux.HandleFunc("*.mylocal.app", myHandler)
mymux.HandleFunc("*", <systemDefaultHandler>)
http.ListenAndServe(":8080", mymux)
}
or
func main() {
mymux := http.NewServeMux()
mymux.HandleFunc("editor.mylocal.app", editorHandler)
mymux.HandleFunc("doc.mylocal.app", docHandler)
mymux.HandleFunc("*.mylocal.app", rootHandler)
mymux.HandleFunc("*", <systemDefaultHandler>)
http.ListenAndServe(":8080", mymux)
}
These are only snippets. A complete example is this, which was referenced in the comments by #Steve101.
However, at now, I don't know what systemDefaultHandler is. And that is not solved there.
Apart from that, #faraz suggested using goproxy. I think that the HTTP/HTTPS transparent proxy is the default handler I am looking for. But, using a package only to do that seems excessive to me. Can I achieve the same functionality with built-in resources?
Unfortunately, there's no dead simple way to do this through Go. You'll need to intercept your system's DNS requests just like dnsmasq, and that's inevitably going to require some modification the system DNS config (/etc/resolv.conf in Linux, /etc/resolver on a Mac, or firewall rule) to route your DNS requests to your app. Going the DNS has the downside that you'd need to build a DNS server inside your app similar to pow.cx, which seems unnecessarily complicated.
Since mucking with system config is inevitable, I'd vote for making changes to the hosts file on boot or when a directory is added/removed (via fsnotify.) On shutdown, you can clear the added entries too.
If you're looking to isolate these changes to a specific browser instead of make a system-wide change, you could always run your application through a proxy server like goproxy and tell your browser to use that proxy for requests. For example, you can do this in Chrome through its preferences or by setting the --proxy-server flag:
--proxy-server=<scheme>=<uri>[:<port>][;...] | <uri>[:<port>] | "direct://"
See Chrome's network settings docs for more details.
Also, if you're willing to much with browser configs, you could just use an extension to handle the requests as needed.
there is only solution that would work without proxy is that you register an domain for this and make an offical dns entry with wildcard to 127.0.0.1 like *.myApp.suche.org. So any client that resolve the ip get 127.0.0.1 and work local. this way you do not need administrator rights to modify the /etc/hosts or equivalent.
If it should work behind proxy without modify the resolver (etc/hosts etc) you can provide an wpad.dat (Javascript for Proxy detection) that say for your domain all traffic goes you server and the rest to the real proxy. If this is served in you server the script can automaticaly contain the real proxy as default.