The Status Quo
Having picked a side project (building a wrapper around a third party API), I'm stuck. I am using sling to compose my HTTP requests.
So parts of the Client are composed as follows:
type Client struct {
// some services etc..
sling *sling.Sling <-- this is initialized with *http.Client
}
func NewClient(httpClient *http.Client) *Client {
sling := sling.New().Client(httpClient).Base(BaseURL)
}
//....
Things I can't wrap my head around
I am following the same principle as go-github and go-twitter that authentication should not be handled by my library, but rather by golangs oauth1/2 package.
As the the API provides application and user level authentication and some of the workflows require initial application level authentication and then user level authentication, my question is, if there is any way to change the *http.Transport in order to change the authentication header on a client basis.
So far, I haven't found a way to do so.
The http.Client has a Transport field that you could use to "change the authentication header on a client basis" if that's what you want. The Transport field has type http.RoundTripper which is a one method interface, so all you need to do is to define your transport with an implementation of the RoundTrip method.
type MyTransport struct {
apiKey string
// keep a reference to the client's original transport
rt http.RoundTripper
}
func (t *MyTransport) RoundTrip(r *http.Request) (*http.Response, error) {
// set your auth headers here
r.Header.Set("Auth", t.apiKey)
return t.rt.RoundTrip(r)
}
Now you can use an instance of this type to set the Transport field on an http.Client.
var client *http.Client = // get client from somewhere...
// set the transport to your type
client.Transport = &MyTransport{apiKey: "secret", tr: client.Transport}
Depending on how and where from you got the client, it's possible that its Transport field is not yet set, so it might be a good idea to ensure that your type uses the default transport in such a case.
func (t *MyTransport) transport() http.RoundTripper {
if t.rt != nil {
return t.rt
}
return http.DefaultTransport
}
// update your method accordingly
func (t *MyTransport) RoundTrip(r *http.Request) (*http.Response, error) {
// set your auth headers here
r.Header.Set("Auth", t.apiKey)
return t.transport().RoundTrip(r)
}
It might be worth noting that the Go documentation recommends not to modify the *http.Request inside the RoundTrip method, so what you can do, and what the go-github package you linked to is doing, is to create a copy of the request, set the auth headers on it, and pass that to the underlying Transport. See here: https://github.com/google/go-github/blob/d23570d44313ca73dbcaadec71fc43eca4d29f8b/github/github.go#L841-L875
Related
Wondering if there is a way to access the underlying net.Conn to retrieve user credentials using SO_PEERCRED and verify a request before it is handled by the server.
From https://blog.jbowen.dev/2019/09/using-so_peercred-in-go/, the net.UnixConn is needed to return the unix.Ucred used for verification. So if there is some way for the server request handler to get at the net.Conn, this should be easy
I looked at a UnaryServerInterceptor, but nothing provided in UnaryServerInterceptor seems to contain the net.Conn
func interceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
log.Printf("Intercepted: %+v %+v", info.Server, req) // anything here?
return handler(ctx, req)
}
The interface method TransportCredentials.ServerHandshake is the seam that you need. Your implementation can read from the input net.Conn and return the credential as an AuthInfo. Then in your handler code, you can get the credential out from the context via peer.FromContext. Alternatively, if you prefer to have authentication occur before the handler code is reached, you can do that directly in the TransportCredentials.ServerHandshake or via an interceptor.
See also: https://groups.google.com/g/grpc-io/c/FeQV7NXpeqA
I reuse the http client connection to make external calls to a single endpoint. An excerpt of the program is shown below:
var AppCon MyApp
func New(user, pass string, platformURL *url.URL, restContext string) (*MyApp, error) {
if AppCon == (MyApp{}) {
AppCon = MyApp{
user: user,
password: pass,
URL: platformURL,
Client: &http.Client{Timeout: 30 * time.Second},
RESTContext: restContext,
}
cj, err := cookiejar.New(nil)
if err != nil {
return &AppCon, err
}
AppCon.cookie = cj
}
return &AppCon, nil
}
// This is an example only. There are many more functions which accept *MyApp as a pointer.
func(ma *MyApp) GetUser(name string) (string, error){
// Return user
}
func main(){
for {
// Get messages from a queue
// The message returned from the queue provide info on which methods to call
// 'm' is a struct with message metadata
c, err := New(m.un, m.pass, m.url)
go func(){
// Do something i.e c.GetUser("123456")
}()
}
}
I now have the requirement to set up a client connections with different endpoints/credentials received via queue messages.
The problem I foresee is I can't just simply modify AppCon with the new endpoint details since a pointer to MyApp is returned, resulting in resetting c. This can impact a goroutine making a HTTP call to an unintended endpoint. To make matters non trivial, the program is not meant to have awareness of the endpoints (I was considering using a switch statement) but rather receive what it needs via queue messages.
Given the issues I've called out are correct, are there any recommendations on how to solve it?
EDIT 1
Based on the feedback provided, I am inclined to believe this will solve my problem:
Remove the use of a Singleton of MyApp
Decouple the http client from MyApp which will enable it for reuse
var httpClient *http.Client
func New(user, pass string, platformURL *url.URL, restContext string) (*MyApp, error) {
AppCon = MyApp{
user: user,
password: pass,
URL: platformURL,
Client: func() *http.Client {
if httpClient == nil {
httpClient = &http.Client{Timeout: 30 * time.Second}
}
return httpClient
}()
RESTContext: restContext,
}
return &AppCon, nil
}
// This is an example only. There are many more functions which accept *MyApp as a pointer.
func(ma *MyApp) GetUser(name string) (string, error){
// Return user
}
func main(){
for {
// Get messages from a queue
// The message returned from the queue provide info on which methods to call
// 'm' is a struct with message metadata
c, err := New(m.un, m.pass, m.url)
// Must pass a reference
go func(c *MyApp){
// Do something i.e c.GetUser("123456")
}(c)
}
}
Disclaimer: this is not a direct answer to your question but rather an attempt to direct you to a proper way of solving your problem.
Try to avoid a singleton pattern for you MyApp. In addition, New is misleading, it doesn't actually create a new object every time. Instead you could be creating a new instance every time, while preserving the http client connection.
Don't use constructions like this: AppCon == (MyApp{}), one day you will shoot in your leg doing this. Use instead a pointer and compare it to nil.
Avoid race conditions. In your code you start a goroutine and immediately proceed to the new iteration of the for loop. Considering you re-use the whole MyApp instance, you essentially introduce a race condition.
Using cookies, you make your connection kinda stateful, but your task seems to require stateless connections. There might be something wrong in such an approach.
I'm trying to compose a simple API client I'm stuck trying to figure out how to make it readable and testable. How can I compose a nested structure while keeping it testable?
Psuedo code:
type VehicleEndpoint struct {
Car CarEndpoint
VehicleGetter
}
type VehicleGetter interface {
Get(string) Vehicle
}
type Vehicle struct {
kind string
}
type VehicleClient struct {
http.Client
url string
}
func (v *VehicleClient) Get(kind string) Vehicle {
resp := v.Do(v.url, kind)
return Vehicle{
kind: resp.Kind
}
}
type CarEndpoint struct
...
type CarGetter interface
...
type Car struct
...
type CarClient struct
...
type API struct {
Vehicle VehicleEndpoint
}
api := API{
Vehicle: VehicleEndpoint{
VehicleGetter: VehicleClient{
http.Client{},
}
Car: CarEndpoint{
CarGetter: CarClient{
http.Client{},
}
}
}
}
Now I can call API like so:
api.Vehicle.Car.Get(kind)
This gives me a very readable (nested) implementation to work with however I'm having a hard time mocking these endpoints because the use of interface would effectively remove any recognition of the nested structure.
What would be the recommended way to construct an API keeping it very readable while also having each endpoint mocked?
You are fighting with language and bringing your OOP hobbies into language that is not designed for that.
I personally would change direction and use old good flat structures and functions.
Although, if you want to continue with your design, you can mock not interface but whole http stack. You can test your code with much better confidence as you testing real http payload vs making calls to your interfaces.
Inject HttpClient into Vehicle: func NewVehicle(httpClient *http.Client){}
In test code, use *http.ServeMux:
mux.Handle("/path1", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// assessments and mocked response
}))
mux.Handle("/path2", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// assessments and mocked response
}))
// fallback to show not implemented routes
result.mux.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
result.t.Errorf("Not Supported route %q", r.URL.Path)
}))
Build Http Server:
server := httptest.NewServer(mux)
Create Http Client from mux server:
client := server.Client()
Coming from a Java background, I have some questions on how things are typically done in Golang. I am specifically talking about services and dao's/repositories.
In java, I would use dependency injection (probably as singleton/application-scoped), and have a Service injected into my rest endpoint / resource.
To give a bit more context. Imagine the following Golang code:
func main() {
http.ListenAndServe("localhost:8080", nil)
}
func init() {
r := httptreemux.New()
api := r.NewGroup("/api/v1")
api.GET("/blogs", GetAllBlogs)
http.Handle("/", r)
}
Copied this directly from my code, main and init are split because google app engine.
So for now I have one handler. In that handler, I expect to interact with a BlogService.
The question is, where, and in what scope should I instantiate a BlogService struct and a dao like datastructure?
Should I do it everytime the handler is triggered, or make it constant/global?
For completeness, here is the handler and blogService:
// GetAllBlogs Retrieves all blogs from GCloud datastore
func GetAllBlogs(w http.ResponseWriter, req *http.Request, params map[string]string) {
c := appengine.NewContext(req)
// need a reference to Blog Service at this point, where to instantiate?
}
type blogService struct{}
// Blog contains the content and meta data for a blog post.
type Blog struct {...}
// newBlogService constructs a new service to operate on Blogs.
func newBlogService() *blogService {
return &blogService{}
}
func (s *blogService) ListBlogs(ctx context.Context) ([]*Blog, error) {
// Do some dao-ey / repository things, where to instantiate BlogDao?
}
You can use context.Context to pass request scoped values into your handlers (available in Go 1.7) , if you build all your required dependencies during the request/response cycle (which you should to avoid race conditions, except for dependencies that manage concurrency on their own like sql.DB). Put all your services into a single container for instance, then query the context for that value :
container := request.Context.Value("container").(*Container)
blogs,err := container.GetBlogService().ListBlogs()
read the following material :
https://golang.org/pkg/context/
https://golang.org/pkg/net/http/#Request.Context
I've written a REST web service that uses mongo as the backend data store. I was wondering at this stage (before deployment), what the best practices were, considering a service that essentially runs forever(ish).
Currently, I'm following this type of pattern:
// database.go
...
type DataStore struct {
mongoSession *mgo.Session
}
...
func (d *DataStore) OpenSession () {
... // read setup from environment
mongoSession, err = mgo.Dial(mongoURI)
if err != nil {}
...
}
func (d *DataStore) CloseSession() {...}
func (d *DataStore) Find (...) (results...) {
s := d.mongoSession.Copy()
defer s.Close()
// do stuff, return results
}
In main.go:
func main() {
ds := NewDataStore()
ds.OpenSession()
defer ds.CloseSession()
// Web Service Routes..
...
ws.Handle("/find/{abc}", doFindFunc)
...
}
My question is - what's the recommended practice for recovery from session that has timed out, lost connection (the mongo service provider I'm using is remote, so I assume that this will happen), so on any particular web service call, the database session may no longer work? How do people handle these cases to detect that the session is no longer valid and a "fresh" one should be established?
Thanks!
what you may want is to do the session .Copy() for each incoming HTTP request (with deffered .Close()), copy again from the new session in your handlers if ever needed..
connections and reconnections are managed by mgo, you can stop and restart MongoDB while making an HTTP request to your web service to see how its affected.
if there's a db connection problem while handling an HTTP request, a db operation will eventually timeout (timeout can be configured by using DialWithTimeout instead of the regular Dial, so you can respond with a 5xx HTTP error code in such case.