Short: How can I read the CGI var REMOTE_USER on golang using fastcgi?
Long:
I'm trying to write a program in go to work behind a httpd using fcgi over a socket. The httpd does the ssl termination and provides basic auth. I need to read $REMOTE_USER, but I cannot in golang, while I can in perl.
My code is based on this fcgi example. I try
func homeView(w http.ResponseWriter, r *http.Request) {
user, pass, authok := r.BasicAuth()
But authok is always false, user and pass remain empty, although I know for sure that the authorization (done by httpd) was OK. To eliminate other errors, I have done it in perl:
my $socket = FCGI::OpenSocket("/run/fcgi-check.sock", 5);
my $q = FCGI::Request(\*STDIN, \*STDOUT, \*STDERR, \%ENV, $socket);
while ($q->Accept() >= 0) {
my $c = CGI::Simple->new;
my $user_id = $c->remote_user();
and it works fine in perl.
To debug, I printed the output of r.Header and I got:
map[Authorization:[]
Am I right that the header that go sees does no hold any information about any authorization? But it does in perl.
Here is a full but minimal golang code example that demonstrates the problem (on OpenBSD 5.8 with go version go1.4.2 openbsd/amd64 and OpenBSDs httpd with 'authenticate "/" with restricted_users' in httpd.conf.
package main
import (
"github.com/gorilla/mux"
"io"
"log"
"fmt"
"net"
"net/http"
"net/http/fcgi"
)
func homeView(w http.ResponseWriter, r *http.Request) {
headers := w.Header()
headers.Add("Content-Type", "text/html")
headers.Add("Cache-Control", "no-cache, no-store, must-revalidate")
headers.Add("Pragma", "no-cache")
headers.Add("Expires", "0")
r.ParseForm()
user, pass, authok := r.BasicAuth()
if authok {
io.WriteString(w, fmt.Sprintln("Auth OK"))
io.WriteString(w, fmt.Sprintln("user is: "+user+", pass is: "+pass))
} else {
io.WriteString(w, fmt.Sprintln("Auth NOT OK"))
}
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/check/", homeView)
var err error
listener, err := net.Listen("unix", "/run/fcgi-check.sock")
if err != nil {
log.Fatal(err)
}
defer listener.Close()
err = fcgi.Serve(listener, r)
if err != nil { log.Fatal(err)}
}
Help will be appreciated!
Thanks in advance!
T.
Go 1.9 will expose cgi environment variables. As seen in this closed ticket:
https://go-review.googlesource.com/c/40012
The simple answer (as of go version 1.4.2) is that go currently does not support the transfer of CGI variable REMOTE_USER.
While #JimB is correct on that you're wrong in your approach, I'll answer the question as stated.
The net/http/fcgi package uses the machinery of net/http/cgi to populate an instance of http.Request—which is passed to your handler—with "parameters" (key/value pairs) submitted by the webserver during the FastCGI session (call).
This is done here.
Now if you'll inspect the relevant bit of the net/http/cgi code, you'll see that the variables which are not mapped to specific dedicated fields of http.Request get converted to HTTP "headers".
This means, your code should be able to access the variable you need using something like
ruser := r.Header.Get("Remote-User")
Update 2015-12-02: the reseach performed by #JimB and the OP showed that there's apparently no way to read the REMOTE_USER variable under FastCGI. Sorry for the noise.
This core change to the fcgi package is in review and is close to being merged. If it's no longer relevant to you, hopefully it will be useful to others.
Related
I am trying to build a proxy server referring medium post. I am not able to log the RequestURI
func handleTunneling(w http.ResponseWriter, r *http.Request) {
fmt.Println("SCHEME:", r.URL.Scheme, "HOST:", r.Host, "PATH", r.URL.Path, )
dest_conn, err := net.DialTimeout("tcp", r.Host, 10*time.Second)
}
Result is for https://example.com/custom_page
SCHEME: HOST: example.com:443 PATH
But from the DialTimeout getting the response original uri. Any suggestions
Thanks
From what I can understand from the medium article you've mention, the handleTunneling actually still the origin of the request. The main process of proxy-ing actually lying the the go transfer() section.
go transfer(dest_conn, client_conn)
go transfer(client_conn, dest_conn)
which actually in the last block of the handleTunneling function.
Thus, it is make sense when you do:
fmt.Println("SCHEME:", r.URL.Scheme, "HOST:", r.Host, "PATH", r.URL.Path)
at the first line of the function, it is still the original path.
I've been working on a Go project where gorilla/mux is used as the router.
I need to be able to have query values associated with a route, but these values should be optional.
That means that I'd like to catch both /articles/123 and /articles/123?key=456 in the same handler.
To accomplish so I tried using the r.Queries method that accepts key/value pairs:
router.
Path("/articles/{id:[0-9]+}").Queries("key", "{[0-9]*?}")
but this makes only the value (456) optional, but not the key.
So both /articles/123?key=456 and /articles/123?key= are valid, but not /articles/123.
Edit: another requirement is that, after registering the route, I'd like to build them programatically, and I can't seem to work out how to use r.Queries even though the docs specifically state that it's possible (https://github.com/gorilla/mux#registered-urls).
#jmaloney answer works, but doesn't allow to build URLs from names.
I would just register your handler twice.
router.Path("/articles/{id:[0-9]+}").
Queries("key", "{[0-9]*?}").
HandlerFunc(YourHandler).
Name("YourHandler")
router.Path("/articles/{id:[0-9]+}").HandlerFunc(YourHandler)
Here is a working program to demonstrate. Notice that I am using r.FormValue to get the query parameter.
Note: make sure you have an up to date version go get -u github.com/gorilla/mux since a bug of query params not getting added the built URLs was fixed recently.
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
var router = mux.NewRouter()
func main() {
router.Path("/articles/{id:[0-9]+}").Queries("key", "{key}").HandlerFunc(YourHandler).Name("YourHandler")
router.Path("/articles/{id:[0-9]+}").HandlerFunc(YourHandler)
if err := http.ListenAndServe(":9000", router); err != nil {
log.Fatal(err)
}
}
func YourHandler(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
key := r.FormValue("key")
u, err := router.Get("YourHandler").URL("id", id, "key", key)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
// Output:
// /articles/10?key=[key]
w.Write([]byte(u.String()))
}
If you register query parameters they are required doc:
All variables defined in the route are required, and their values must conform to the corresponding patterns.
Because those parameters are optional you just need to check for them inside of a handler function: id, found := mux.Vars(r)["id"]. Where found will show if the parameter in the query or not.
Seems like the best way to handle optional URL parameters is to define your router as normal without them, then parse the optional params out like this:
urlParams := request.URL.Query()
This returns a map that contains the URL parameters as Key/Value pairs.
I have a handler which connects to a db and retrieves the records. I wrote a test case for that and it goes this way:
main_test.go
package main
import (
"os"
"fmt"
"testing"
"net/http"
"net/http/httptest"
)
var a App
func TestMain(m *testing.M) {
a = App{}
a.InitializeDB(fmt.Sprintf("postgres://****:****#localhost/db?sslmode=disable"))
code := m.Run()
os.Exit(code)
}
func TestRulesetGet(t *testing.T) {
req, err := http.NewRequest("GET", "/1/sig/", nil)
if err != nil {
t.Fatal(err)
}
// We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
rr := httptest.NewRecorder()
handler := http.HandlerFunc(a.Get)
handler.ServeHTTP(rr, req)
// Check the response body is what we expect.
if len(rr.Body.String()) != 0 {
fmt.Println("Status OK : ", http.StatusOK)
fmt.Println("handler returned body: got ",
rr.Body.String())
}
}
I feel like this is a very basic test case where I'm just checking the response length (This is because I expect a slice). I'm not really sure whether this is the right way to write a test case. Please point out some loop holes so that I could write a solid test cases for my remaining handlers. And also, I'm not using the actual Error, and Fatal methods for checking the errors.
If it works then it's not wrong, but it doesn't validate much, only that the response body is non-empty - not even that it's successful. I'm not sure why it prints out http.StatusOK, which is a constant, and doesn't tell you what the status code in the response was.
Personally when I do this level of test I check, at the very least, that the response code is as expected, the response body unmarshals correctly (if it's JSON or XML), the response data is basically sane, etc. For more complex payloads I might use a golden file test. For critical code I might use a fuzz (aka monte carlo) test. For performance-critical code I'll likely add benchmarks and load tests. There are practically infinite ways to test code. You'll have to figure out what your needs are and how to meet them.
I'm trying to develop a simple job queue server with some worker that query it but I encountered a problem with my net/http server. I'm surely doing something bad but after ~3 minutes my server start displaying :
http: Accept error: accept tcp [::]:4200: accept4: too many open files; retrying in 1s
For information it receive 10 request per second in my test case.
Here's two files to reproduce this error :
// server.go
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/get", func(rw http.ResponseWriter, r *http.Request) {
http.Error(rw, "Try again", http.StatusInternalServerError)
})
http.ListenAndServe(":4200", nil)
}
// worker.go
package main
import (
"net/http"
"time"
)
func main() {
for {
res, _ := http.Get("http://localhost:4200/get")
defer res.Body.Close()
if res.StatusCode == http.StatusInternalServerError {
time.Sleep(100 * time.Millisecond)
continue
}
return
}
}
I already done some search about this error and I found some interesting response but none of these fixed my issue.
The first response I saw was to correctly close the Body in the http.Get response, as you can see I did it.
The second response was to change the file descriptor ulimit of my system but as I will not control where my app will run I prefer to not use this solution (But for information it's set at 1024 on my system)
Can someone explain me why this problem happen and how I can fix it in my code ?
Thanks a lot for your time
EDIT :
EDIT 2 : In comment Martin says that I'm not closing the Body, I tried to close it (without defer so) and it fixed the issue. Thanks Martin ! I was thinking that continue will execute my defer, I was wrong.
I found a post explaining the root problem in a lot more detail.
Nathan Smith even explains how to control timeouts on the TCP level, if needed.
Below is a summary of everything I could find on this particular problem, as well as the best practices to avoid this problem in future.
Problem
When a response is received regardless of whether response-body is required or not, the connection is kept alive until the response-body stream is closed. So, as mentioned in this thread, always close the response-body. Even if you do not need to use/read the body content:
func Ping(url string) (bool) {
// simple GET request on given URL
res, err := http.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Best Practice
As mentioned by Nathan Smith never use the http.DefaultClient in production systems, this includes calls like http.Get as it uses http.DefaultClient at its base.
Another reason to avoid http.DefaultClient is that it is a Singleton (package level variable), meaning that the garbage collector will not try to clean it up, which will leave idling subsequent streams/sockets alive.
Instead create your own instance of http.Client and remember to always specify a sane Timeout:
func Ping(url string) (bool) {
// create a new instance of http client struct, with a timeout of 2sec
client := http.Client{ Timeout: time.Second * 2 }
// simple GET request on given URL
res, err := client.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Safety Net
The safety net is for that newbie on the team, who does not know the shortfalls of http.DefaultClient usage. Or even that very useful, but not so active, open-source library that is still riddled with http.DefaultClient calls.
Since http.DefaultClient is a Singleton we can easily change the Timeout setting, just to ensure that legacy code does not cause idle connections to remain open.
I find it best to set this on the package main file in the init function:
package main
import (
"net/http"
"time"
)
func init() {
/*
Safety net for 'too many open files' issue on legacy code.
Set a sane timeout duration for the http.DefaultClient, to ensure idle connections are terminated.
Reference: https://stackoverflow.com/questions/37454236/net-http-server-too-many-open-files-error
*/
http.DefaultClient.Timeout = time.Minute * 10
}
As Martin say in comment I don't really closed the Body after the Get request. I used defer res.Body.Close() but it's not executed since I'm staying in the for loop. So continue dont't trigger defer
Please note that in some cases the setting in /etc/sysctl.conf
net.ipv4.tcp_tw_recycle = 1
Could cause this error because TCP connections remain open.
A temporary solution, just increase the number of open files:
ulimit -Sn 10000
I have a zip file stored on Google Drive (it is shared publicly). I want to know how to download it in Golang. This current code just creates a blank file named "file.zip":
package main
import (
"fmt"
"io"
"net/http"
"os"
)
func main() {
url := "https://docs.google.com/uc?export=download&id=0B2Q7X-dUtUBebElySVh1ZS1iaTQ"
fileName := "file.zip"
fmt.Println("Downloading file...")
output, err := os.Create(fileName)
defer output.Close()
response, err := http.Get(url)
if err != nil {
fmt.Println("Error while downloading", url, "-", eerrror)
return
}
defer response.Body.Close()
n, err := io.Copy(output, response.Body)
fmt.Println(n, "bytes downloaded")
}
This appears to be a bug, either with Google drive or with golang, I'm not sure which!
The problem is that the first URL you gave redirects to a second URL which looks something like this
https://doc-00-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/8i67l6m6cdojptjuh883mu0qqmtptds1/1376330400000/06448503420061938118/*/0B2Q7X-dUtUBebElySVh1ZS1iaTQ?h=16653014193614665626&e=download
Note the * in the URL which is legal according to this stack overflow question. However it does have a special meaning as a delimeter.
Go fetches the URL with the * encoded as %2A like this
https://doc-00-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/8i67l6m6cdojptjuh883mu0qqmtptds1/1376330400000/06448503420061938118/%2A/0B2Q7X-dUtUBebElySVh1ZS1iaTQ?h=16653014193614665626&e=download
Which Google replies "403 Forbidden" to.
Google doesn't seem to be resolving the %2A into a *.
According to this article on wikipedia reserved characters (of which * is one) used in a URI scheme: if it is necessary to use that character for some other purpose, then the character must be percent-encoded.
I'm not enough of an expert on this to say who is right, but since Google wrote both parts of the problem it is definitely their fault somewhere!
Here is the program I was using for testing
I found the solution.
Use: https://googledrive.com/host/ID
Instead of: https://docs.google.com/uc?export=download&id=ID
I'm still investigating on why this is happening, in the meanwhile you can use this workaround:
http://play.golang.org/p/SzGBAiZdGJ
CheckRedirect is called when a redirect happens and you can add an Opaque path to avoid having the URL url-encoded.
Francesc