log http.ResponseWriter content - go

Premise: I've found a similar issue but not working in my case, so please do not mark this as a duplicate.
I've a HTTP server in Go and I've created a middleware to log the request, the response time and I would like to log the response too.
I've used httputil.DumpRequest in a function called HTTPRequest under the package log.
How can I correctly get the response body and status and headers from the w http.ResponseWriter and log them together with the other data?
My ISSUE is: I would like to intercept the Response Headers, Status and Body and to log the together with the Request and Response Time
Here's the code:
log "core/logger"
...
func RequestLoggerMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
defer func() {
log.Info(
fmt.Sprintf(
"[Request: %s] [Execution time: %v] [Response: %s]",
log.HTTPRequest(r),
time.Since(start),
// RESPONSE DATA HERE !!!!!!!
))
}()
next.ServeHTTP(w, r)
})
}

Thanks, #Sivachandran for the response. It was almost perfect, only it didn't implement the http.ResponseWriter because of the pointers.
For the sake of completeness, I post here the correct solution code, because it's not easy to find any documentation on it, even if this question has been given a negative score.
Stackoverflow is a good place to exchange questions and this, in my opinion, was a very good and difficult question, either for a middle lever Golang programmer, so it didn't deserve a negative score at all!
That's the solution, enjoy:
// RequestLoggerMiddleware is the middleware layer to log all the HTTP requests
func RequestLoggerMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
rww := NewResponseWriterWrapper(w)
w.Header()
defer func() {
log.Info(
fmt.Sprintf(
"[Request: %s] [Execution time: %v] [Response: %s]",
log.HTTPRequest(r),
time.Since(start),
rww.String(),
))
}()
next.ServeHTTP(rww, r)
})
}
// ResponseWriterWrapper struct is used to log the response
type ResponseWriterWrapper struct {
w *http.ResponseWriter
body *bytes.Buffer
statusCode *int
}
// NewResponseWriterWrapper static function creates a wrapper for the http.ResponseWriter
func NewResponseWriterWrapper(w http.ResponseWriter) ResponseWriterWrapper {
var buf bytes.Buffer
var statusCode int = 200
return ResponseWriterWrapper{
w: &w,
body: &buf,
statusCode: &statusCode,
}
}
func (rww ResponseWriterWrapper) Write(buf []byte) (int, error) {
rww.body.Write(buf)
return (*rww.w).Write(buf)
}
// Header function overwrites the http.ResponseWriter Header() function
func (rww ResponseWriterWrapper) Header() http.Header {
return (*rww.w).Header()
}
// WriteHeader function overwrites the http.ResponseWriter WriteHeader() function
func (rww ResponseWriterWrapper) WriteHeader(statusCode int) {
(*rww.statusCode) = statusCode
(*rww.w).WriteHeader(statusCode)
}
func (rww ResponseWriterWrapper) String() string {
var buf bytes.Buffer
buf.WriteString("Response:")
buf.WriteString("Headers:")
for k, v := range (*rww.w).Header() {
buf.WriteString(fmt.Sprintf("%s: %v", k, v))
}
buf.WriteString(fmt.Sprintf(" Status Code: %d", *(rww.statusCode)))
buf.WriteString("Body")
buf.WriteString(rww.body.String())
return buf.String()
}

You need to wrap the ResponseWriter to capture the response data.
type ResponseWriterWrapper struct {
w http.ResponseWriter
body bytes.Buffer
statusCode int
}
func (i *ResponseWriterWrapper) Write(buf []byte) (int, error) {
i.body.Write(buf)
return i.w.Write(buf)
}
func (i *ResponseWriterWrapper) WriteHeader(statusCode int) {
i.statusCode = statusCode
i.w.WriteHeader(statusCode)
}
func (i *ResponseWriterWrapper) String() {
var buf bytes.Buffer
buf.WriteString("Response:")
buf.WriteString("Headers:")
for k, v := range i.w.Header() {
buf.WriteString(fmt.Sprintf("%s: %v", k, v))
}
buf.WriteString(fmt.Sprintf("Status Code: %d", i.statusCode))
buf.WriteString("Body")
buf.WriteString(i.body.String())
}
Pass the wrapper to ServeHTTP and log captured response data.
func RequestLoggerMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
rww := ResponseWriterWrapper{ w: w }
defer func() {
log.Info(
fmt.Sprintf(
"[Request: %s] [Execution time: %v] [Response: %s]",
log.HTTPRequest(r),
time.Since(start),
log.Info(rww.String())
))
}()
next.ServeHTTP(rww, r)
})
}

Related

GzipResponseWriter field declaration

I am looking at this type
type GzipResponseWriter struct {
gw *gzip.Writer
http.ResponseWriter
}
And functions that will implement it
func (w GzipResponseWriter) Write(b []byte) (int, error) {
if _, ok := w.Header()["Content-Type"]; !ok {
// If content type is not set, infer it from the uncompressed body.
w.Header().Set("Content-Type", http.DetectContentType(b))
}
return w.gw.Write(b)
}
func (w GzipResponseWriter) Flush() {
w.gw.Flush()
if fw, ok := w.ResponseWriter.(http.Flusher); ok {
fw.Flush()
}
}
Does the http.ResponseWriter relate to the second field?
Why not
gw1 http.ResponseWriter?

rewrite content length within a middleware

below code rewrites the http body response of some queries.
However, it fails to update the "content length" header field, it always remains the same original value.
How can i update the content length header field of the http response ?
type writeReplacer struct {
http.ResponseWriter
search []byte
replace func(*http.Request) string
buf []byte
r *http.Request
dir string
}
func (w *writeReplacer) Write(in []byte) (int, error) {
if w.buf == nil {
w.buf = []byte{}
}
w.buf = append(w.buf, in...)
n := len(in)
if index := bytes.LastIndex(w.buf, w.search); index > -1 {
var r []byte
if w.dir == "before" {
g := []byte(w.replace(w.r))
n += len(g)
r = append(g, w.buf[index:]...)
w.buf = append(w.buf[:index], r...)
} else {
g := []byte(w.replace(w.r))
n += len(g)
r = append(r, w.buf[:index+len(w.search)]...)
r = append(r, g...)
r = append(r, w.buf[index:]...)
w.buf = r
}
}
return n, nil
}
func (w *writeReplacer) Flush() {
w.ResponseWriter.Header().Set("Content-Length", fmt.Sprint(len(w.buf)))
w.ResponseWriter.Write(w.buf[:])
w.buf = w.buf[:0]
}
func InsertAfter(h http.Handler, path string, search []byte, replace func(*http.Request) string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == path {
w.Header().Del("Content-length")
w = &writeReplacer{ResponseWriter: w, search: search, replace: replace, r: r, dir: "after"}
defer w.(http.Flusher).Flush()
}
h.ServeHTTP(w, r)
})
}
func InsertBefore(h http.Handler, path string, search []byte, replace func(*http.Request) string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == path {
w.Header().Del("Content-length")
w = &writeReplacer{ResponseWriter: w, search: search, replace: replace, r: r, dir: "before"}
defer w.(http.Flusher).Flush()
}
h.ServeHTTP(w, r)
})
}
I receive two errors message, the first one from nginx
2019/03/08 05:58:37 [error] 31194#0: *19
upstream prematurely closed connection while reading upstream,
client: 82.21.18.16, server: buycoffee.online, request:
"GET / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "buycoffee.online"
the second one from curl
curl: (18) transfer closed with 6237 bytes remaining to read
as mkopriva suggested the trick was to rewrite ResponseWriter.WriteHeader.
I believe it has to do with the fact that write might call writeheader at first call, and from there i was not putting the instructions at the right place.
in order to prevent further difficulties i prefer to use chunked transfer.
the code change is:
func (w *writeReplacer) WriteHeader(statusCode int) {
w.Header().Del("Content-length")
w.Header().Set("Transfer-Encoding", "chunked")
w.ResponseWriter.WriteHeader(statusCode)
}

How does creating handlers works?

I am looking at this example
var name string
type helloWorldResponse struct {
Message string `json:"message"`
}
type helloWorldRequest struct {
Name string `json:"name"`
}
func main() {
port := 8080
handler := newValidationHandler(newHelloWorldHandler())
http.Handle("/helloworld", handler)
log.Printf("Server starting on port %v\n", port)
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%v", port), nil))
}
type validationHandler struct {
next http.Handler
}
func newValidationHandler(next http.Handler) http.Handler {
return validationHandler{next: next}
}
func (h validationHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
var request helloWorldRequest
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&request)
if err != nil {
http.Error(rw, "Bad request", http.StatusBadRequest)
return
}
name = request.Name
h.next.ServeHTTP(rw, r)
}
type helloWorldHandler struct{}
func newHelloWorldHandler() http.Handler {
return helloWorldHandler{}
}
func (h helloWorldHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
response := helloWorldResponse{Message: "Hello " + name}
encoder := json.NewEncoder(rw)
encoder.Encode(response)
}
The author of the code explained that we are
going to be chaining handlers together, the first handler, which is our validation handler,
needs to have a reference to the next in the chain as it has the responsibility for calling
ServeHTTP or returning a response. I am newbe to Go and I do not understand this line
return validationHandler{next: next}
Which data structure next:next represents?
type validationHandler struct {
next http.Handler // 1
}
func newValidationHandler(next /* 2 */ http.Handler) http.Handler {
return validationHandler{next: next}
// 1 2
}
next number 1 is a field from validationHandler struct (a few lines above). And the other next is method's parameter (from the signature). All in all, this simply sets a field in a struct. No magic.
Which data structure next:next represents?
Not a data structure. It is struct initialization syntax. See more examples here: https://gobyexample.com/structs

http HandleFunc argument in golang

I want to use a rate limiting or throttler library to limit the number of client requests. I use a vendor library in my code base. I want to pass in a ResponseWriter, Request and a third variable retrieved from the URL. When I use the library for throttling, it gives me back a handler that only handles two arguments. How can I pass my third argument into the handler?
Here is my current code:
package main
import (
"fmt"
"github.com/didip/tollbooth"
"net/http"
)
func viewHandler(
w http.ResponseWriter,
r *http.Request,
uniqueId string,
) {
//data := getData(uniqueId)
fmt.Println("Id:", uniqueId)
p := &objects.ModelApp{LoggedUser: "Ryan Hardy", ViewData: "data"}
renderTemplate(w, "view", p)
}
//URL validation for basic web services
var validPath = regexp.MustCompile("^/$|/(home|about|view)/(|[a-zA-Z0-9]+)$")
func makeHandler(
fn func(
http.ResponseWriter,
*http.Request, string,
)) http.HandlerFunc {
return func(
w http.ResponseWriter,
r *http.Request,
) {
m := validPath.FindStringSubmatch(r.URL.Path)
if m == nil {
http.NotFound(w, r)
return
}
fn(w, r, m[2])
}
}
func main() {
http.Handle("/view/", makeHandler(tollbooth.LimitFuncHandler(tollbooth.NewLimiter(1, time.Second), viewHandler)))
http.ListenAndServe(":8080", nil)
}
Could anyone help me with this?
I'm on my phone so this may be difficult to type but you could use the http.Handle function which takes an interface of Handler something like
type makeHandler struct {
YourVariable string
}
func (m *makeHandler) ServeHTTP (w http.ResponseWriter, r *http.Request) {
yourVariableYouNeed := m.YourVariable
// do whatever
w.Write()
}
// do whatever you need to get your variable
blah := &makeHandler{ yourThing }
http.Handle("/views", blah)
On my phone so can't test but it should work, let me know if it doesn't.

Showing custom 404 error page with standard http package

Assuming that we have:
http.HandleFunc("/smth", smthPage)
http.HandleFunc("/", homePage)
User sees a plain "404 page not found" when they try a wrong URL. How can I return a custom page for that case?
Update concerning gorilla/mux
Accepted answer is ok for those using pure net/http package.
If you use gorilla/mux you should use something like this:
func main() {
r := mux.NewRouter()
r.NotFoundHandler = http.HandlerFunc(notFound)
}
And implement func notFound(w http.ResponseWriter, r *http.Request) as you want.
I usually do this:
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", homeHandler)
http.HandleFunc("/smth/", smthHandler)
http.ListenAndServe(":12345", nil)
}
func homeHandler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
errorHandler(w, r, http.StatusNotFound)
return
}
fmt.Fprint(w, "welcome home")
}
func smthHandler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/smth/" {
errorHandler(w, r, http.StatusNotFound)
return
}
fmt.Fprint(w, "welcome smth")
}
func errorHandler(w http.ResponseWriter, r *http.Request, status int) {
w.WriteHeader(status)
if status == http.StatusNotFound {
fmt.Fprint(w, "custom 404")
}
}
Here I've simplified the code to only show custom 404, but I actually do more with this setup: I handle all the HTTP errors with errorHandler, in which I log useful information and send email to myself.
Following is the approach I choose. It is based on a code snippet which I cannot acknowledge since I lost the browser bookmark.
Sample code : (I put it in my main package)
type hijack404 struct {
http.ResponseWriter
R *http.Request
Handle404 func (w http.ResponseWriter, r *http.Request) bool
}
func (h *hijack404) WriteHeader(code int) {
if 404 == code && h.Handle404(h.ResponseWriter, h.R) {
panic(h)
}
h.ResponseWriter.WriteHeader(code)
}
func Handle404(handler http.Handler, handle404 func (w http.ResponseWriter, r *http.Request) bool) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
hijack := &hijack404{ ResponseWriter:w, R: r, Handle404: handle404 }
defer func() {
if p:=recover(); p!=nil {
if p==hijack {
return
}
panic(p)
}
}()
handler.ServeHTTP(hijack, r)
})
}
func fire404(res http.ResponseWriter, req *http.Request) bool{
fmt.Fprintf(res, "File not found. Please check to see if your URL is correct.");
return true;
}
func main(){
handler_statics := http.StripPrefix("/static/", http.FileServer(http.Dir("/Path_To_My_Static_Files")));
var v_blessed_handler_statics http.Handler = Handle404(handler_statics, fire404);
http.Handle("/static/", v_blessed_handler_statics);
// add other handlers using http.Handle() as necessary
if err := http.ListenAndServe(":8080", nil); err != nil{
log.Fatal("ListenAndServe: ", err);
}
}
Please customize the func fire404 to output your own version of message for error 404.
If you happen to be using Gorilla Mux, you may wish to replace the main function with below :
func main(){
handler_statics := http.StripPrefix("/static/", http.FileServer(http.Dir("/Path_To_My_Static_Files")));
var v_blessed_handler_statics http.Handler = Handle404(handler_statics, fire404);
r := mux.NewRouter();
r.PathPrefix("/static/").Handler(v_blessed_handler_statics);
// add other handlers with r.HandleFunc() if necessary...
http.Handle("/", r);
log.Fatal(http.ListenAndServe(":8080", nil));
}
Please kindly correct the code if it is wrong, since I am only a newbie to Go. Thanks.
Ancient thread, but I just made something to intercept http.ResponseWriter, might be relevant here.
package main
//GAE POC originally inspired by https://thornelabs.net/2017/03/08/use-google-app-engine-and-golang-to-host-a-static-website-with-same-domain-redirects.html
import (
"net/http"
)
func init() {
http.HandleFunc("/", handler)
}
// HeaderWriter is a wrapper around http.ResponseWriter which manipulates headers/content based on upstream response
type HeaderWriter struct {
original http.ResponseWriter
done bool
}
func (hw *HeaderWriter) Header() http.Header {
return hw.original.Header()
}
func (hw *HeaderWriter) Write(b []byte) (int, error) {
if hw.done {
//Silently let caller think they are succeeding in sending their boring 404...
return len(b), nil
}
return hw.original.Write(b)
}
func (hw *HeaderWriter) WriteHeader(s int) {
if hw.done {
//Hmm... I don't think this is needed...
return
}
if s < 400 {
//Set CC header when status is < 400...
//TODO: Use diff header if static extensions
hw.original.Header().Set("Cache-Control", "max-age=60, s-maxage=2592000, public")
}
hw.original.WriteHeader(s)
if s == 404 {
hw.done = true
hw.original.Write([]byte("This be custom 404..."))
}
}
func handler(w http.ResponseWriter, r *http.Request) {
urls := map[string]string{
"/example-post-1.html": "https://example.com/post/example-post-1.html",
"/example-post-2.html": "https://example.com/post/example-post-2.html",
"/example-post-3.html": "https://example.com/post/example-post-3.html",
}
w.Header().Set("Strict-Transport-Security", "max-age=15768000")
//TODO: Put own logic
if value, ok := urls[r.URL.Path]; ok {
http.Redirect(&HeaderWriter{original: w}, r, value, 301)
} else {
http.ServeFile(&HeaderWriter{original: w}, r, "static/"+r.URL.Path)
}
}
i think the clean way is this:
func main() {
http.HandleFunc("/calculator", calculatorHandler)
http.HandleFunc("/history", historyHandler)
http.HandleFunc("/", notFoundHandler)
log.Fatal(http.ListenAndServe(":80", nil))
}
if the address is not /calulator or /history, then it handles notFoundHandler function.
Maybe I'm wrong, but I just checked the sources: http://golang.org/src/pkg/net/http/server.go
It seems like specifying custom NotFound() function is hardly possible: NotFoundHandler() returns a hardcoded function called NotFound().
Probably, you should submit an issue on this.
As a workaround, you can use your "/" handler, which is a fallback if no other handlers were found (as it is the shortest one). So, check is page exists in that handler and return a custom 404 error.
You just need to create your own notFound handler and register it with HandleFunc for the path that you don't handle.
If you want the most control over your routing logic you will need to use a custom server and custom handler type of your own.
http://golang.org/pkg/net/http/#Handler
http://golang.org/pkg/net/http/#Server
This allows you to implement more complex routing logic than the HandleFunc will allow you to do.
you can define
http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
if request.URL.Path != "/" {
writer.WriteHeader(404)
writer.Write([]byte(`not found, da xiong dei !!!`))
return
}
})
when access not found resource, it will execute to http.HandleFunc("/", xxx)
You can simply use something like:
func Handle404(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "404 error\n")
}
func main(){
http.HandleFunc("/", routes.Handle404)
}
If you need to get the standard one, just write:
func main(){
http.HandleFunc("/", http.NotFound)
}
And you'll get:
404 page not found

Resources