Selectively Follow Redirects in Go - go

Question

I'm trying to write a twitter reader that resolves the final URLs of link shorteners etc, but gives me a URL along the way for a list of manually defined host patterns. The reason to do this is that i don't want to end up with the paywall URL but the one before.
As far as i can tell the way to do this is write my own client based on the default RoundTripper because returning an error from a custom CheckRedirect function aborts the client without yielding a response.
Is there a way to use the default client and record a list of URLs/specific URL from a custom checkRedirect function?

Answer 1

The client request will actually still return the last valid Response in cases where your custom CheckResponse yields an error (As mentioned in the comments).
http://golang.org/pkg/net/http/#Client
If CheckRedirect returns an error, the Client's Get method returns both the previous Response and CheckRedirect's error (wrapped in a url.Error) instead of issuing the Request req.
If you maintain a list of "known" paywall-urls, you can abort the paywall-redirect in your CheckResponse with a custom error type (Paywalled in the example below).
Your error handling code later has to consider that error type as a special (non-erroneous) case.
Example:
package main
import (
"errors"
"fmt"
"net/http"
"net/url"
)
var Paywalled = errors.New("next redirect would hit a paywall")
var badHosts = map[string]error{
"registration.ft.com": Paywalled,
}
var client = &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
// N.B.: when used in production, also check for redirect loops
return badHosts[req.URL.Host]
},
}
func main() {
resp, err := client.Get("http://on.ft.com/14pQBYE")
// ignore non-nil err if it's a `Paywalled` wrapped in url.Error
if e, ok := err.(*url.Error); (ok && e.Err != Paywalled) || (!ok && err != nil) {
fmt.Println("error: ", err)
return
}
resp.Body.Close()
fmt.Println(resp.Request.URL)
}