Using the Golang's http.Get() i am able to get effective or final url after multiple redirects,but in few cases where there are 303 redirects and special characters in URL golang is being weird, i am not able to get the actual final url. Following is the example i am dealing with - "http://swiggy.com//google.com/%2f.." , if we open this url in browser we get redirected to google, but i couldnt get the same using http.Get()
Other side behavior may depends on a lot of factors - for example of your User-Agent used, Cookies, IP and so on. Also sometimes it can change because of DDoS protection mechanism or things alike.
You may modify your app and see how it passes redirect stages:
Result:
getURL: http://swiggy.com//google.com/%2f..
Redirecting: 301 https://swiggy.com/google.com/%2f..
Redirecting: 301 https://www.swiggy.com/google.com/%2f..
Redirecting: 303 https://www.swiggy.com/google.com/%2f../
finalURL: https://www.swiggy.com/google.com/%2f../
Req Headers: map[Referer:[https://www.swiggy.com/google.com/%2f..]]
Resp Headers: map[Date:[Mon, 06 Nov 2017 12:51:20 GMT] Content-Type:[text/html; charset=utf-8] Content-Security-Policy-Report-Only:[default-src 'self';script-src https://chuknu.sokrati.com/15946/ https://www.google-analytics.com/ https://cdn.inspectlet.com/ https://tracking.sokrati.com/ https://connect.facebook.net/ https://bam.nr-data.net/ https://maps.googleapis.com/ https://js-agent.newrelic.com/ https://www.googletagmanager.com/ https://s3-ap-southeast-1.amazonaws.com/static.swiggy/ https://*.juspay.in https://connect.facebook.net/ https://www.googletagmanager.com/ *.swiggy.in *.swiggy.com https://chat2.hotline.io/ 'self' 'unsafe-inline' 'unsafe-eval' 'nonce-150997268072300';style-src https://fonts.googleapis.com/ https://www.swiggy.com/ https://s3-ap-southeast-1.amazonaws.com/static.swiggy/ https://chat2.hotline.io/ 'self' 'unsafe-inline' 'unsafe-eval';img-src https://res.cloudinary.com/swiggy/ https://www.google-analytics.com/ https://www.google.co.in/ https://www.facebook.com/ https://tracking.sokrati.com/ http://api.swiggy.in/ https://api.swiggy.com https://d3oxf4lkkqx2kx.cloudfront.net/ https://maps.googleapis.com/ https://maps.gstatic.com/ https://csi.gstatic.com/ https://fonts.gstatic.com/ https://stats.g.doubleclick.net/ https://googleads.g.doubleclick.net/ https://www.google.com/ data: 'self'; font-src https://www.swiggy.com/ https://fonts.gstatic.com/ data: 'self';connect-src https://hn.inspectlet.com/ https://www.swiggy.com/ https://www.facebook.com/tr/ https://*.juspay.in/txns https://sentry.swiggyapp.com/ 'self';frame-src https://www.facebook.com/tr/ https://chat2.hotline.io/ https://*.webpush.hotline.io 'self';report-uri /csp/log] Etag:[W/"6f97-"] Vary:[Accept-Encoding] X-Data-Origin:[dweb_cluster/port-dweb-06 naxsi/waf rate-limiter-plain/rate-limiter-plain] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Frame-Options:[Deny] Set-Cookie:[__SW=sjfsljfd; Path=/]]
Modified code:
package main
import (
"fmt"
"net/http"
)
func CheckRedirect(r *http.Request, via []*http.Request) error {
fmt.Println("Redirecting:", r.Response.StatusCode, r.URL)
return nil
}
func main() {
getURL := "http://swiggy.com//google.com/%2f.."
fmt.Println("getURL:", getURL)
client := &http.Client{
CheckRedirect: CheckRedirect,
}
resp, err := client.Get(getURL)
if err != nil {
fmt.Println(err)
return
}
finalURL := resp.Request.URL.String()
fmt.Println("finalURL:", finalURL)
fmt.Println("Req Headers:", resp.Request.Header)
fmt.Println("Resp Headers:", resp.Header)
}
Related
Hello. I'm using the go language in Google App Engine. I'm having trouble getting the logged-in user's information. Similarly, the login URL and logout URL cannot be obtained. All nil will be returned. user.IsAdmin (c) returns false. please help me.
admin.go
func Entry(w http.ResponseWriter, r *http.Request) {
...
c := appengine.NewContext(r)
inUrl, err := user.LoginURL(c, "/admin/top/")
...
}
func AdminTop(w http.ResponseWriter, r *http.Request) {
...
c := appengine.NewContext(r)
booisadmin := user.IsAdmin(c)
u := user.Current(c)
outUrl, err := user.LogoutURL(c, "/")
...
}
app.yaml
runtime: go116
app_engine_apis: true
handlers:
- url: /assets/css
mime_type: text/css
static_dir: assets/css
- url: /assets/html
mime_type: text/html
static_dir: assets/html
- url: /assets/img
static_dir: assets/img
- url: /admin/.*
login: require
script: _go_app
- url: /.*
script: _go_app
When you use login: required in app.yaml, you can get the logged in users information via the following headers -
X-Appengine-User-Id
X-Appengine-User-Nickname
X-Appengine-User-Email
I confirmed the above works in Go (ran it on my local machine)
I believe the same headers should work when you use the Users API but you can always dump all the headers to figure out the values that you need.
Regarding using the User API to get login/logout urls, I also got blank values when I tried it on my local machine but I'm a novice when it comes to Go. You might want to try and see if the calls work when you deploy to Production
Thank you. I was able to solve it. It was because I was using Gorillamux. I solved it with the following code.
import (
...
"github.com/gorilla/mux"
"net/http"
...
)
func main() {
r := mux.NewRouter()
r.HandleFunc("/", indexHandler)
r.HandleFunc("/admin/", admin.Entry)
http.Handle("/", r)
}
The last http.Handle ("/", r) was missing.
I wrote the details here.
https://uubaago.blogspot.com/
Thank you very much NoCommandLine!
I am just started learning Go, and this question made me stuck.
Trying to test request handling on localhost in testing func using github.com/valyala/fasthttp.
First running the server like in https://github.com/valyala/fasthttp/blob/master/server_example_test.go:
ln, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatalf("error in net.Listen: %s", err)
}
requestHandler := func(ctx *fasthttp.RequestCtx) {
fmt.Println(ctx, "Requested path is")
}
if err := fasthttp.Serve(ln, requestHandler); err != nil {
log.Fatalf("error in Serve: %s", err)
}
then if I run the request func (FastRequest(url string)) from the same testing function it works fine...
Fasthttp request func:
func FastRequest(url string) error {
Request := &fasthttp.Request{}
Response := &fasthttp.Response{}
FastHTTPClient := &fasthttp.Client{}
Request.SetRequestURI(url)
for {
err := FastHTTPClient.DoTimeout(Request, Response, time.Minute)
switch err {
case fasthttp.ErrTimeout, fasthttp.ErrDialTimeout:
<-time.After(time.Minute * 2)
continue
case fasthttp.ErrNoFreeConns:
<-time.After(time.Minute * 2)
continue
case nil:
return nil
default:
if strings.Contains(err.Error(), "connection reset by peer") {
<-time.After(time.Minute * 2)
continue
} else {
return err
}
}
}
}
But what I truly need to test is sending a request from my object, which implements the same FastRequest method in goroutine.
And here I've got this error message:
error when serving connection ":8080"<->":51325": error when reading request headers: invalid header key " http/1.1\r\nuser-Agent". Buffer size=206, contents: "GET here_is_request_url \n http/1.1\r\nuser-Agent: fasthttp\r\nHost: localhost:8080\r\n\r\n"
In FastRequest I haven't specified any user agent and the functions FastRequest() are the same. Only the place where the function is called is different. Whether it's called in goroutine or not does not matter.
So, fasthttp.RequestCtx cannot parse its own header? or what is going on?
==========================================================================
Also, I should have added that in first case I've used fasthttp v1.6.0, when I changed it to 1.8.0 the error was:
error when serving connection ":8080"<->":57093": error when reading request headers: invalid header name. Buffer size=215, contents: "GET here_is_request_url\n HTTP/1.1\r\nUser-Agent: fasthttp\r\nHost: localhost:8080\r\n\r\n"
And finally, the issue was in "/n" added at the end of the url, that works for real servers, by my small localhost server couldn't handle it.
... contents: "GET here_is_request_url \n http/1.1\r\n ...
^^^^^^^^^^^^^
WRONG!
The URL you use in your code likely has a newline character (\n) still at the end since this is included in your request before the HTTP version and thus messes up the HTTP request. A real URL should have no white space which includes spaces and also newline characters.
Additionally the HTTP version should be all-uppercase HTTP/1.1, i.e. your lower-case http/1.1 is wrong too. You don't show how you create the HTTP request but it is very likely messed up.
I'm basically trying to write a reverse proxy server so that when I curl localhost:8080/get it proxies the request to https://nghttp2.org/httpbin/get.
Note: the https://nghttp2.org/httpbin/get service listed above is http/2. But this behavior happens with http/1 as well, such as https://httpbin.org/get.
I'm using httputil.ReverseProxy for this and I'm rewriting the URL while customizing the Host header to not to leak the localhost:8080 to the actual backend.
However, the request still hits the backend with Host: localhost:8080 no matter how many times I set it on the header. Similarly, I used mitmproxy to snoop on the request and it looks like the net/http.Client sets the :authority pseudo-header to localhost:8080
Here's my source code:
package main
import (
"log"
"net/http"
"net/http/httputil"
)
func main() {
proxy := &httputil.ReverseProxy{
Transport: roundTripper(rt),
Director: func(req *http.Request) {
req.URL.Scheme = "https"
req.URL.Host = "nghttp2.org"
req.URL.Path = "/httpbin" + req.URL.Path
req.Header.Set("Host", "nghttp2.org") // <--- I set it here first
},
}
log.Fatal(http.ListenAndServe(":8080", proxy))
}
func rt(req *http.Request) (*http.Response, error) {
log.Printf("request received. url=%s", req.URL)
req.Header.Set("Host", "nghttp2.org") // <--- I set it here as well
defer log.Printf("request complete. url=%s", req.URL)
return http.DefaultTransport.RoundTrip(req)
}
// roundTripper makes func signature a http.RoundTripper
type roundTripper func(*http.Request) (*http.Response, error)
func (f roundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) }
When I query curl localhost:8080/get the request gets proxied to https://nghttp2.org/httpbin/get. The echoed response shows that clearly my directives setting the Host header didn't do anything:
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "localhost:8080",
"User-Agent": "curl/7.54.0"
},
"origin": "2601:602:9c02:16c2:fca3:aaab:3914:4a71",
"url": "https://localhost:8080/httpbin/get"
}
mitmproxy snooping also clearly shows that the request was made with :authority pseudo-header set to localhost:8080:
From http.Request docs:
// For server requests, Host specifies the host on which the URL
// is sought. Per RFC 7230, section 5.4, this is either the value
// of the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
// To prevent DNS rebinding attacks, server Handlers should
// validate that the Host header has a value for which the
// Handler considers itself authoritative. The included
// ServeMux supports patterns registered to particular host
// names and thus protects its registered Handlers.
//
// For client requests, Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
So the value of URL.Host is only used in case request.Host is empty which is not the case. Setting request.Host should resolve the issue:
req.Host = "nghttp2.org"
Related issue discussed here.
I'm running a webserver written in go on an embedded system. The timestamp of index.html may go backwards if someone has downgraded the firmware version. If index.html is older than the previous version, the server sends a http 304 response (not modified), and serves a cached version of the file.
The webserver code is using http.FileServer() and http.ListenAndServe().
The problem can easily reproduced by modifying the timestamp of index.html using the Posix command touch
touch -d"23:59" index.html
reloading the page, then
touch -d"23:58" index.html
reloading this time will give a 304 response on index.html.
Is there a way to prevent timestamp based caching?
Assuming your file server code is like the example in the docs:
http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("/static"))))
You can write a handler that sets the appropriate cache headers to prevent this behaviour by stripping ETag headers and setting Cache-Control: no-cache, private, max-age=0 to prevent caching (both locally and in upstream proxies):
var epoch = time.Unix(0, 0).Format(time.RFC1123)
var noCacheHeaders = map[string]string{
"Expires": epoch,
"Cache-Control": "no-cache, private, max-age=0",
"Pragma": "no-cache",
"X-Accel-Expires": "0",
}
var etagHeaders = []string{
"ETag",
"If-Modified-Since",
"If-Match",
"If-None-Match",
"If-Range",
"If-Unmodified-Since",
}
func NoCache(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
// Delete any ETag headers that may have been set
for _, v := range etagHeaders {
if r.Header.Get(v) != "" {
r.Header.Del(v)
}
}
// Set our NoCache headers
for k, v := range noCacheHeaders {
w.Header().Set(k, v)
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
Use it like so:
http.Handle("/static/", NoCache(http.StripPrefix("/static/", http.FileServer(http.Dir("/static")))))
Note: I originally wrote this at github.com/zenazn/goji/middleware, so you can also just import that, but it's a simple piece of code to write and I wanted to show a full example for posterity!
I'm trying to write a twitter reader that resolves the final URLs of link shorteners etc, but gives me a URL along the way for a list of manually defined host patterns. The reason to do this is that i don't want to end up with the paywall URL but the one before.
As far as i can tell the way to do this is write my own client based on the default RoundTripper because returning an error from a custom CheckRedirect function aborts the client without yielding a response.
Is there a way to use the default client and record a list of URLs/specific URL from a custom checkRedirect function?
The client request will actually still return the last valid Response in cases where your custom CheckResponse yields an error (As mentioned in the comments).
http://golang.org/pkg/net/http/#Client
If CheckRedirect returns an error, the Client's Get method returns both the previous Response and CheckRedirect's error (wrapped in a url.Error) instead of issuing the Request req.
If you maintain a list of "known" paywall-urls, you can abort the paywall-redirect in your CheckResponse with a custom error type (Paywalled in the example below).
Your error handling code later has to consider that error type as a special (non-erroneous) case.
Example:
package main
import (
"errors"
"fmt"
"net/http"
"net/url"
)
var Paywalled = errors.New("next redirect would hit a paywall")
var badHosts = map[string]error{
"registration.ft.com": Paywalled,
}
var client = &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
// N.B.: when used in production, also check for redirect loops
return badHosts[req.URL.Host]
},
}
func main() {
resp, err := client.Get("http://on.ft.com/14pQBYE")
// ignore non-nil err if it's a `Paywalled` wrapped in url.Error
if e, ok := err.(*url.Error); (ok && e.Err != Paywalled) || (!ok && err != nil) {
fmt.Println("error: ", err)
return
}
resp.Body.Close()
fmt.Println(resp.Request.URL)
}