I have this unit test for a proxy i'm writing. I cannot for the life of me see why the environment get ignored and my test make a direct access to the target server, acap.
func TestHandleHTTPS(t *testing.T) {
successfulCalls := 0
proxyPassed := 0
acap := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
successfulCalls++
}))
defer acap.Close()
testproxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxyPassed++
}))
defer testproxy.Close()
os.Setenv("https_proxy", testproxy.URL)
defer os.Setenv("https_proxy", "")
client := acap.Client()
tmp := client.Transport.(*http.Transport)
tmp.Proxy = http.ProxyFromEnvironment // <--- This should make client use env vars!
req, err := http.NewRequest("GET", acap.URL, nil)
if err != nil {
t.Errorf("Unable to create request: %s", err.Error())
}
resp, err := client.Do(req)
if err != nil {
t.Errorf("Something is wrong with the test: %s", err.Error())
return
}
if resp.StatusCode != 200 {
t.Errorf("Unexpected status code: %d", resp.StatusCode)
body, _ := ioutil.ReadAll(resp.Body)
t.Errorf("Body: %s", string(body))
}
if successfulCalls == 0 {
t.Errorf("No successful call over HTTPS occurred")
}
if proxyPassed == 0 {
t.Errorf("Proxy got ignored")
}
}
The only failure i get is Proxy got ignored. I use Go v1.10, everything compiles.
Edit 1:
I do the tmp.Proxy dance because the client already have certificates and stuff configured in the Transport. I don't want to mess that up by replacing the entire Transport struct
If you take a look at doc for ProxyFromEnvironment you will find there a special case:
As a special case, if req.URL.Host is "localhost" (with or without a port number), then a nil URL and nil error will be returned.
That means that no proxy will be used. I would suggest you to use ProxyURL instead
proxyURL, _ := url.Parse(testproxy.URL)
tmp.Proxy = http.ProxyURL(proxyURL)
It will take into account your Proxy, but won't work, because you are trying to make an https call throw http proxy...
Related
I have a Go GRPC server-side streaming function:
func (server *Server) GetClients(req *iam.GetClientsRequest, client iam.IAM_GetClientsServer) error {
ctx := client.(interface{ Context() context.Context }).Context()
userID, err := getUserIDStream(client)
if err != nil {
return err
}
clients, err := server.db.QueryByUserID(ctx, userID)
if err != nil {
return grpc.Errorf(codes.Internal, apiutils.ServerError)
}
for _, value := range clients {
converted, err := server.fromInternalClient(value)
if err != nil {
return err
}
if err := client.Send(converted); err != nil {
return err
}
}
return nil
}
and I'm testing it like this:
It("GetClients - Send fails - Error", func() {
handler := createHandler(db)
lis := bufconn.Listen(bufSize)
server := grpc.NewServer()
iam.RegisterIAMServer(server, NewServer(handler))
go func() {
if err := server.Serve(lis); err != nil {
log.Fatalf("Server exited with error: %v", err)
}
}()
defer lis.Close()
defer server.GracefulStop()
conn, err := grpc.DialContext(context.Background(), "bufnet",
grpc.WithContextDialer(createBufDialier(lis)), grpc.WithInsecure())
Expect(err).ShouldNot(HaveOccurred())
defer conn.Close()
client := iam.NewIAMClient(conn)
cclient, _ := client.GetClients(addAccessToken(context.Background()), new(iam.GetClientsRequest))
resp, err := cclient.Recv()
Expect(resp).Should(BeNil())
Expect(err).Should(HaveOccurred())
Expect(err.Error()).Should(Equal(message))
})
My issue is that I'm not sure how to induce a failure on Send so I can test the response. Since I'm using an actual test server and client, I can't just mock out the object and I'd prefer not to go that route anyway. Is there a way I can do this?
Originally, I was trying to force Send to fail by setting bufSize to an artificially low value. However, this wasn't producing an error so I decided to try modifying the maxSendMessageSize on the server:
opts := []grpc.ServerOption{}
if sendFails {
opts = append(opts, grpc.MaxSendMsgSize(10))
}
lis := bufconn.Listen(bufSize)
server := grpc.NewServer(opts...)
And this worked in producing the error.
I have been implementing server using golang. I need to shutdown my server after receiving the expected parameter 'code'. Before shutting down the sever I need to redirect to a another web page. I have implemented as give below. This code is working. I need to know whether it is the best way of doing it ? Your suggestions are appreciated..
func main() {
var code string
const port int = 8888
httpPortString := ":" + strconv.Itoa(port)
mux := http.NewServeMux()
fmt.Printf("Http Server initialized on Port %s", httpPortString)
server := http.Server{Addr: httpPortString, Handler: mux}
var timer *time.Timer
mux.HandleFunc("/auth", func(w http.ResponseWriter, r *http.Request) {
err := r.ParseForm()
if err != nil {
fmt.Printf("Error parsing the code: %s", err)
}
code = r.Form.Get("code")
if err != nil {
log.Printf("Error occurred while establishing the server: %s", err)
}
http.Redirect(w, r, "https://cloud.google.com/sdk/auth_success", http.StatusMovedPermanently)
timer = time.NewTimer(2 * time.Second)
go func() {
<-timer.C
server.Shutdown(context.Background())
}()
})
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
fmt.Printf("Error while establishing the service: %s", err)
}
fmt.Println("Finished executing the the service")
}
Thank you ..!
Taking #Peter flushing suggestion and ideas from the example cited here:
f, ok := w.(http.Flusher)
if !ok {
http.Error(w, "no flush support", http.StatusInternalServerError)
return
}
http.Redirect(w, r, "https://cloud.google.com/sdk/auth_success", http.StatusSeeOther)
f.Flush() // <-- ensures client gets all writes
// this is done implicitly on http handler returns, but...
// we're shutting down the server now!
go func() {
server.Shutdown(context.Background())
close(idleConnsClosed)
}()
See full playground version for idleConnsClosed setup/cleanup: https://play.golang.org/p/UBmLfyhKT0B
P.S. Don't use http.StatusMovedPermanently unless you really want users never to use the source URL again. Users' browsers will cache this (301) code - and not hit your server - which may not be what you want. If you want temporary redirects use http.StatusSeeOther (code 303).
I want to use hijack in golang, while recieve invalid response on client
func hijack(w http.ResponseWriter, r *http.Request) {
fmt.Println("start")
hj, ok := w.(http.Hijacker)
fmt.Println(ok)
c, buf, err := hj.Hijack()
if err != nil {
panic(err)
}
n, err := buf.Write([]byte("hello"))
if err != nil {
panic(err)
}
fmt.Println("n == ",n)
err = buf.Flush()
if err != nil {
panic(err)
}
fmt.Println("end")
}
follow printed on server:
start
true
n == 5
end
but I got following error on the client
localhost sent an invalid response. ERR_INVALID_HTTP_RESPONSE
As the Hijacker's documentation says
Hijack lets the caller take over the connection. After a
call to Hijack the HTTP server library will not do
anything else with the connection.
It becomes the caller's responsibility to manage and close
the connection.
The returned net.Conn may have read or write deadlines already set,
depending on the configuration of the Server. It is the
caller's responsibility to set or clear those deadlines
as needed.
The returned bufio.Reader may contain unprocessed buffered data from
the client.
After a call to Hijack, the original Request.Body must not be used. The
original Request's Context remains valid and is not canceled until
the Request's ServeHTTP method returns.
You need to write to c rather than buf. And you need to write response status and Content-Length header.
http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
fmt.Println("start")
writer.Header().Add("Content-Length", "5")
writer.WriteHeader(200)
hj, ok := writer.(http.Hijacker)
fmt.Println(ok)
c, _, err := hj.Hijack()
if err != nil {
panic(err)
}
n, err := c.Write([]byte("hello"))
if err != nil {
panic(err)
}
fmt.Println("n == ",n)
err = c.Close()
if err != nil {
panic(err)
}
fmt.Println("end")
})
I am trying to get the stackdriver tracer to work with gRPC and I need some help. I have been looking at these two links for reference and I still can't get it to work:
https://medium.com/#harlow/tracing-grpc-calls-in-golang-with-google-stackdriver-b22495763a06#.81oa9q21v
https://rakyll.org/grpc-trace/
For simplicity, I am just working with the hello world gRPC example. Here's my client:
func main() {
// Set up a connection to the server.
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithUnaryInterceptor(grpc.UnaryClientInterceptor(clientInterceptor)))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
ctx := context.Background()
tc, err := trace.NewClient(ctx, "{PROJECT-ID}")
if err != nil {
log.Fatal(err)
}
span := tc.NewSpan("/greeter/SayHello")
defer span.Finish()
ctx = trace.NewContext(ctx, span)
r, err := c.SayHello(ctx, &pb.HelloRequest{Name: "world"})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
println("Response:", r.Message)
}
func clientInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
// trace current request w/ child span
span := trace.FromContext(ctx).NewChild(method)
defer span.Finish()
// new metadata, or copy of existing
md, ok := metadata.FromContext(ctx)
if !ok {
md = metadata.New(nil)
} else {
md = md.Copy()
}
// append trace header to context metadata
// header specification: https://cloud.google.com/trace/docs/faq
md["X-Cloud-Trace-Context"] = append(
md["X-Cloud-Trace-Context"], fmt.Sprintf("%s/%d;o=1", span.TraceID(), 0),
)
ctx = metadata.NewContext(ctx, md)
return invoker(ctx, method, req, reply, cc, opts...)
}
.. and my gRPC server:
// server is used to implement helloworld.GreeterServer.
type server struct{}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
println("HERE")
return &pb.HelloReply{Message: "Hello " + in.Name}, nil
}
func main() {
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
ctx := context.Background()
tc, err := trace.NewClient(ctx, "{PROJECT-ID}")
if err != nil {
log.Fatal(err)
}
s := grpc.NewServer(EnableGRPCTracingServerOption(tc))
pb.RegisterGreeterServer(s, &server{})
println("listening on :50051")
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
// EnableGRPCTracingServerOption enables parsing google trace header from metadata
// and adds a new child span to the incoming request context.
func EnableGRPCTracingServerOption(traceClient *trace.Client) grpc.ServerOption {
return grpc.UnaryInterceptor(serverInterceptor(traceClient))
}
func serverInterceptor(traceClient *trace.Client) grpc.UnaryServerInterceptor {
return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
// fetch metadata from request context
md, ok := metadata.FromContext(ctx)
if !ok {
md = metadata.New(nil)
}
header := strings.Join(md["X-Cloud-Trace-Context"], "")
// create new child span from google trace header, add to
// current request context
span := traceClient.SpanFromHeader(info.FullMethod, header)
defer span.Finish()
ctx = trace.NewContext(ctx, span)
return handler(ctx, req)
}
}
I when I run the client to initiate the trace, I get the error:
rpc error: code = 13 desc = stream terminated by RST_STREAM with error code: 1
I'm confused because I don't see anything else about authentication; only providing the project ID which can't be enough to initiate tracing for a specific project. What am I missing?
The issue was with:
defer span.Finish()
That call does not block so because I was just doing preliminary testing with one call my program was exiting before the traces could be uploaded. I contacted the author of https://rakyll.org/grpc-trace/ and she actually updated her post with the option of using:
defer span.FinishWait()
which blocks and that fixed it by allowing the traces to be successfully uploaded before the program exited.
Also, with a long running webserver this wouldn't have been an issue because the process wouldn't have been terminated.
I followed those same tutorials and ran into similar problems.
Header keys are converted to lowercase. If you retrieve it on the server side with header := strings.Join(md["x-cloud-trace-context"], "") you should be good.
You can also define your metadata headers with:
span := trace.FromContext(ctx).NewChild(method)
defer span.Finish()
md := metadata.Pairs(
"x-cloud-trace-context", fmt.Sprintf("%s/%d;o=1", span.TraceID(), 0),
)
ctx = metadata.NewContext(ctx, md)
Here my use case
We have one services "foobar" which has two version legacy and version_2_of_doom (both in go)
In order to make the transition from legacy to version_2_of_doom , we would like in a first time, to have the two version alongside, and have the POST request (as there's only one POST api call in this ) received on both.
The way I see how to do it. Would be
modifying the code of legacy at the beginning of the handler, in order to duplicate the request to version_2_of_doom
func(w http.ResponseWriter, req *http.Request) {
req.URL.Host = "v2ofdoom.local:8081"
req.Host = "v2ofdoom.local:8081"
client := &http.Client{}
client.Do(req)
// legacy code
but it seems to not be as straightforward as this
it fails with http: Request.RequestURI can't be set in client requests.
Is there a well-known method to do this kind of action (i.e transfering without touching) a http.Request to an other server ?
You need to copy the values you want into a new request. Since this is very similar to what a reverse proxy does, you may want to look at what "net/http/httputil" does for ReverseProxy.
Create a new request, and copy only the parts of the request you want to send to the next server. You will also need to read and buffer the request body if you intend to use it both places:
func handler(w http.ResponseWriter, req *http.Request) {
// we need to buffer the body if we want to read it here and send it
// in the request.
body, err := ioutil.ReadAll(req.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// you can reassign the body if you need to parse it as multipart
req.Body = ioutil.NopCloser(bytes.NewReader(body))
// create a new url from the raw RequestURI sent by the client
url := fmt.Sprintf("%s://%s%s", proxyScheme, proxyHost, req.RequestURI)
proxyReq, err := http.NewRequest(req.Method, url, bytes.NewReader(body))
// We may want to filter some headers, otherwise we could just use a shallow copy
// proxyReq.Header = req.Header
proxyReq.Header = make(http.Header)
for h, val := range req.Header {
proxyReq.Header[h] = val
}
resp, err := httpClient.Do(proxyReq)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
defer resp.Body.Close()
// legacy code
}
In my experience, the easiest way to achieve this was to simply create a new request and copy all request attributes that you need into the new request object:
func(rw http.ResponseWriter, req *http.Request) {
url := req.URL
url.Host = "v2ofdoom.local:8081"
proxyReq, err := http.NewRequest(req.Method, url.String(), req.Body)
if err != nil {
// handle error
}
proxyReq.Header.Set("Host", req.Host)
proxyReq.Header.Set("X-Forwarded-For", req.RemoteAddr)
for header, values := range req.Header {
for _, value := range values {
proxyReq.Header.Add(header, value)
}
}
client := &http.Client{}
proxyRes, err := client.Do(proxyReq)
// and so on...
This approach has the benefit of not modifying the original request object (maybe your handler function or any middleware functions that are living in your stack still need the original object?).
Using original request (copy or duplicate only if original request still need):
func handler(w http.ResponseWriter, r *http.Request) {
// Step 1: rewrite URL
URL, _ := url.Parse("https://full_generic_url:123/x/y")
r.URL.Scheme = URL.Scheme
r.URL.Host = URL.Host
r.URL.Path = singleJoiningSlash(URL.Path, r.URL.Path)
r.RequestURI = ""
// Step 2: adjust Header
r.Header.Set("X-Forwarded-For", r.RemoteAddr)
// note: client should be created outside the current handler()
client := &http.Client{}
// Step 3: execute request
resp, err := client.Do(r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Step 4: copy payload to response writer
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
resp.Body.Close()
}
// copyHeader and singleJoiningSlash are copy from "/net/http/httputil/reverseproxy.go"
func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}
func singleJoiningSlash(a, b string) string {
aslash := strings.HasSuffix(a, "/")
bslash := strings.HasPrefix(b, "/")
switch {
case aslash && bslash:
return a + b[1:]
case !aslash && !bslash:
return a + "/" + b
}
return a + b
}
I've seen the accepted anwser, but I would like to say that I dont like this. I've used this code for months with it working, but after some time you encounter requests that break (POST requests in my case). My preferred solution is the following:
r.URL.Host = "example.com"
r.RequestURI = ""
client := &http.Client{}
delete(r.Header, "Accept-Encoding")
delete(r.Headers, "Content-Length")
resp, err := client.Do(r.WithContext(context.Background())
if err != nil {
return nil, err
}
return resp, nil