I am trying to build a proxy server referring medium post. I am not able to log the RequestURI
func handleTunneling(w http.ResponseWriter, r *http.Request) {
fmt.Println("SCHEME:", r.URL.Scheme, "HOST:", r.Host, "PATH", r.URL.Path, )
dest_conn, err := net.DialTimeout("tcp", r.Host, 10*time.Second)
}
Result is for https://example.com/custom_page
SCHEME: HOST: example.com:443 PATH
But from the DialTimeout getting the response original uri. Any suggestions
Thanks
From what I can understand from the medium article you've mention, the handleTunneling actually still the origin of the request. The main process of proxy-ing actually lying the the go transfer() section.
go transfer(dest_conn, client_conn)
go transfer(client_conn, dest_conn)
which actually in the last block of the handleTunneling function.
Thus, it is make sense when you do:
fmt.Println("SCHEME:", r.URL.Scheme, "HOST:", r.Host, "PATH", r.URL.Path)
at the first line of the function, it is still the original path.
Related
I'm trying to develop a simple job queue server with some worker that query it but I encountered a problem with my net/http server. I'm surely doing something bad but after ~3 minutes my server start displaying :
http: Accept error: accept tcp [::]:4200: accept4: too many open files; retrying in 1s
For information it receive 10 request per second in my test case.
Here's two files to reproduce this error :
// server.go
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/get", func(rw http.ResponseWriter, r *http.Request) {
http.Error(rw, "Try again", http.StatusInternalServerError)
})
http.ListenAndServe(":4200", nil)
}
// worker.go
package main
import (
"net/http"
"time"
)
func main() {
for {
res, _ := http.Get("http://localhost:4200/get")
defer res.Body.Close()
if res.StatusCode == http.StatusInternalServerError {
time.Sleep(100 * time.Millisecond)
continue
}
return
}
}
I already done some search about this error and I found some interesting response but none of these fixed my issue.
The first response I saw was to correctly close the Body in the http.Get response, as you can see I did it.
The second response was to change the file descriptor ulimit of my system but as I will not control where my app will run I prefer to not use this solution (But for information it's set at 1024 on my system)
Can someone explain me why this problem happen and how I can fix it in my code ?
Thanks a lot for your time
EDIT :
EDIT 2 : In comment Martin says that I'm not closing the Body, I tried to close it (without defer so) and it fixed the issue. Thanks Martin ! I was thinking that continue will execute my defer, I was wrong.
I found a post explaining the root problem in a lot more detail.
Nathan Smith even explains how to control timeouts on the TCP level, if needed.
Below is a summary of everything I could find on this particular problem, as well as the best practices to avoid this problem in future.
Problem
When a response is received regardless of whether response-body is required or not, the connection is kept alive until the response-body stream is closed. So, as mentioned in this thread, always close the response-body. Even if you do not need to use/read the body content:
func Ping(url string) (bool) {
// simple GET request on given URL
res, err := http.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Best Practice
As mentioned by Nathan Smith never use the http.DefaultClient in production systems, this includes calls like http.Get as it uses http.DefaultClient at its base.
Another reason to avoid http.DefaultClient is that it is a Singleton (package level variable), meaning that the garbage collector will not try to clean it up, which will leave idling subsequent streams/sockets alive.
Instead create your own instance of http.Client and remember to always specify a sane Timeout:
func Ping(url string) (bool) {
// create a new instance of http client struct, with a timeout of 2sec
client := http.Client{ Timeout: time.Second * 2 }
// simple GET request on given URL
res, err := client.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Safety Net
The safety net is for that newbie on the team, who does not know the shortfalls of http.DefaultClient usage. Or even that very useful, but not so active, open-source library that is still riddled with http.DefaultClient calls.
Since http.DefaultClient is a Singleton we can easily change the Timeout setting, just to ensure that legacy code does not cause idle connections to remain open.
I find it best to set this on the package main file in the init function:
package main
import (
"net/http"
"time"
)
func init() {
/*
Safety net for 'too many open files' issue on legacy code.
Set a sane timeout duration for the http.DefaultClient, to ensure idle connections are terminated.
Reference: https://stackoverflow.com/questions/37454236/net-http-server-too-many-open-files-error
*/
http.DefaultClient.Timeout = time.Minute * 10
}
As Martin say in comment I don't really closed the Body after the Get request. I used defer res.Body.Close() but it's not executed since I'm staying in the for loop. So continue dont't trigger defer
Please note that in some cases the setting in /etc/sysctl.conf
net.ipv4.tcp_tw_recycle = 1
Could cause this error because TCP connections remain open.
A temporary solution, just increase the number of open files:
ulimit -Sn 10000
Currently my web app is just a router and handlers.
What are some important things I am missing to make this production worthy?
I believe I have to set the # of procs to ensure this uses maximum goroutines?
Should I be using output buffering?
Anything else you see missing that is best-practise?
var (
templates = template.Must(template.ParseFiles("templates/home.html")
)
func main() {
r := mux.NewRouter()
r.HandleFunc("/", WelcomeHandler)
http.ListenAndServe(":9000", r)
}
func WelcomeHandler(w http.ResponseWriter, r *http.Request) {
homePage, err := api.LoadHomePage()
if err != nil {
}
tmpl := "home"
renderTemplate(w, tmpl, homePage)
}
func renderTemplate(w http.ResponseWriter, tmpl string, hp *HomePage) {
err := templates.ExecuteTemplate(w, tmpl+".html", hp)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
You don't need to set/change runtime.GOMAXPROCS() as since Go 1.5 it defaults to the number of available CPU cores.
Buffering output? From the performance point of view, you don't need to. But there may be other considerations for which you may.
For example, your renderTemplate() function may potentially panic. If executing the template starts writing to the output, it involves setting the HTTP response code and other headers prior to writing data. And if a template execution error occurs after that, it will return an error, and so your code attempts to send back an error response. At this point HTTP headers are already written, and this http.Error() function will try to set headers again => panic.
One way to avoid this is to first render the template into a buffer (e.g. bytes.Buffer), and if no error is returned by the template execution, then you can write the content of the buffer to the response writer. If error occurs, then of course you won't write the content of the buffer, but send back an error response just like you did.
To sum it up, your code is production ready performance-wise (excluding the way you handle template execution errors).
WelcomeHandler should return when err != nil is true.
Log the error when one is hit to help investigation.
Place templates = template.Must(template.ParseFiles("templates/home.html") in the init. Split it into separate lines. If template.ParseFiles returns an then error make a Fatal log. And if you have multiple templates to initialize then initialize them in goroutines with a common WaitGroup to speed up the startup.
Since you are using mux, HTTP Server is too clean with its URLs might also be good to know.
You might also want to reconsider the decision of letting the user's know why they got the http.StatusInternalServerError response.
Setting the GOMAXPROCS > 1 if you have more the one core would definitely be a good idea but I would keep it less than number of cores available.
Short: How can I read the CGI var REMOTE_USER on golang using fastcgi?
Long:
I'm trying to write a program in go to work behind a httpd using fcgi over a socket. The httpd does the ssl termination and provides basic auth. I need to read $REMOTE_USER, but I cannot in golang, while I can in perl.
My code is based on this fcgi example. I try
func homeView(w http.ResponseWriter, r *http.Request) {
user, pass, authok := r.BasicAuth()
But authok is always false, user and pass remain empty, although I know for sure that the authorization (done by httpd) was OK. To eliminate other errors, I have done it in perl:
my $socket = FCGI::OpenSocket("/run/fcgi-check.sock", 5);
my $q = FCGI::Request(\*STDIN, \*STDOUT, \*STDERR, \%ENV, $socket);
while ($q->Accept() >= 0) {
my $c = CGI::Simple->new;
my $user_id = $c->remote_user();
and it works fine in perl.
To debug, I printed the output of r.Header and I got:
map[Authorization:[]
Am I right that the header that go sees does no hold any information about any authorization? But it does in perl.
Here is a full but minimal golang code example that demonstrates the problem (on OpenBSD 5.8 with go version go1.4.2 openbsd/amd64 and OpenBSDs httpd with 'authenticate "/" with restricted_users' in httpd.conf.
package main
import (
"github.com/gorilla/mux"
"io"
"log"
"fmt"
"net"
"net/http"
"net/http/fcgi"
)
func homeView(w http.ResponseWriter, r *http.Request) {
headers := w.Header()
headers.Add("Content-Type", "text/html")
headers.Add("Cache-Control", "no-cache, no-store, must-revalidate")
headers.Add("Pragma", "no-cache")
headers.Add("Expires", "0")
r.ParseForm()
user, pass, authok := r.BasicAuth()
if authok {
io.WriteString(w, fmt.Sprintln("Auth OK"))
io.WriteString(w, fmt.Sprintln("user is: "+user+", pass is: "+pass))
} else {
io.WriteString(w, fmt.Sprintln("Auth NOT OK"))
}
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/check/", homeView)
var err error
listener, err := net.Listen("unix", "/run/fcgi-check.sock")
if err != nil {
log.Fatal(err)
}
defer listener.Close()
err = fcgi.Serve(listener, r)
if err != nil { log.Fatal(err)}
}
Help will be appreciated!
Thanks in advance!
T.
Go 1.9 will expose cgi environment variables. As seen in this closed ticket:
https://go-review.googlesource.com/c/40012
The simple answer (as of go version 1.4.2) is that go currently does not support the transfer of CGI variable REMOTE_USER.
While #JimB is correct on that you're wrong in your approach, I'll answer the question as stated.
The net/http/fcgi package uses the machinery of net/http/cgi to populate an instance of http.Request—which is passed to your handler—with "parameters" (key/value pairs) submitted by the webserver during the FastCGI session (call).
This is done here.
Now if you'll inspect the relevant bit of the net/http/cgi code, you'll see that the variables which are not mapped to specific dedicated fields of http.Request get converted to HTTP "headers".
This means, your code should be able to access the variable you need using something like
ruser := r.Header.Get("Remote-User")
Update 2015-12-02: the reseach performed by #JimB and the OP showed that there's apparently no way to read the REMOTE_USER variable under FastCGI. Sorry for the noise.
This core change to the fcgi package is in review and is close to being merged. If it's no longer relevant to you, hopefully it will be useful to others.
I searched around and as far as I can tell, POST form requests are already limited to 10MB (http://golang.org/src/net/http/request.go#L721).
If I were to go about reducing this in my ServeHTTP method, I'm not sure how to properly do it. I would try something like this:
r.Body = http.MaxBytesReader(w, r.Body, MaxFileSize)
err := r.ParseForm()
if err != nil {
//redirect to some error page
return
}
But would returning upon error close the connection as well? How would I prevent having to read everything? I found this: https://stackoverflow.com/a/26393261/2202497, but what if content length is not set and in the middle of reading I realize that the file is too big.
I'm using this as a security measure to prevent someone from hogging my server's resources.
The correct way to limit the size of the request body is to do as you suggested:
r.Body = http.MaxBytesReader(w, r.Body, MaxFileSize)
err := r.ParseForm()
if err != nil {
// redirect or set error status code.
return
}
MaxBytesReader sets a flag on the response when the limit is reached. When this flag is set, the server does not read the remainder of the request body and the server closes the connection on return from the handler.
If you are concerned about malicious clients, then you should also set Server.ReadTimeout, Server.WriteTimeout and possibly Server.MaxHeaderBytes.
If you want to set the request body limit for all of your handlers, then wrap root handler with a handler that sets the limit before delegating to the root handler:
type maxBytesHandler struct {
h http.Handler
n int64
}
func (h *maxBytesHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, h.n)
h.h.ServeHTTP(w, r)
}
Wrap the root handler when calling ListenAndServe:
log.Fatal(http.ListenAndServe(":8080", &maxBytesHandler{h:mux, n:4096))
or when configuring a server:
s := http.Server{
Addr: ":8080",
Handler: &maxBytesReader{h:mux, n:4096},
}
log.Fatal(s.ListenAndServe())
There's no need for a patch as suggested in another answer. MaxBytesReader is the official way to limit the size of the request body.
Edit: As others cited MaxByteReader is the supported way. It is interesting that the default reader is instead, limitreader after type asserting for max byte reader.
Submit a patch to the Go source code and make it configurable! You are working with an open source project after all. Adding a setter to http.Request and some unit tests for it is probably only 20 minutes worth of work. Having a hardcoded value here is a bit clunky, give back and fix it :).
You can of course implement your own ParseForm(r *http.Request) method if you really need to override this. Go is essentially BSD, so you can copy paste the library ParseForm and change the limit, but thats a bit ugly no?
I wrote a cookie getter and setter. Now I want to test it, and wrote following test function.
func TestAuthorizationReader(t *testing.T) {
tw := httptest.NewServer(testWriter())
tr := httptest.NewServer(Use(testReader()))
defer tw.Close()
defer tr.Close()
c := &http.Client{}
rs, err := c.Get(tw.URL)
assert.NoError(t, err, "Should not contain any error")
// Assign cookie to client
url, err := rs.Location()
fmt.Print(url)
assert.NoError(t, err, "Should not contain any error")
//c.Jar.SetCookies(url, rs.Cookies())
}
The test fail at the second part, as output message I've got
- FAIL: TestAuthorizationReader (0.05s)
Location: logged_test.go:64
Error: No error is expected but got http: no Location header in response
Messages: Should not contain any error
I can not get the URL location pointer, what do I wrong here?
The Response.Location method returns the value of the Location response header. You would usually only expect to see this header for redirect responses, so it isn't surprising you got this error.
If you want to know the URL used to retrieve a particular response, try rs.Request.URL.String(). Even if the HTTP library followed a redirect to retrieve the document, this will look at the request used for this particular response, which is what you'd be after when determining a cookie's origin.
If you just want the client to keep track of cookies set by requests it processes though, all you should need to do is set the Jar attribute on your client. Something like this:
import "net/http/cookiejar"
...
c := &http.Client{
Jar: cookiejar.New(nil),
}
Now cookies set in earlier responses should be set in future requests to the same origin.