this is my actual code :
func cURL(cURL string, follow bool) (*string, error) {
var err error
var resp *http.Response
var req *http.Request
var u *url.URL
if u, err = url.Parse(cURL); nil != err {
if logLevel >= 30 {
log.Print("ERROR: ", err)
}
return nil, err
}
if req, err = http.NewRequest("GET", u.Scheme+"://"+u.Host+u.Path, nil); err != nil {
if logLevel >= 40 {
log.Print("ERROR: ", err)
}
return nil, err
}
req.URL.RawQuery = u.RawQuery
client := &http.Client{
Timeout: 3 * time.Second, //timeout connexion
}
if resp, err = client.Do(req); nil != err {
if logLevel >= 50 {
log.Print("ERROR: ", err)
}
return nil, err
}
defer resp.Body.Close()
var body []byte
if body, err = ioutil.ReadAll(resp.Body); nil != err {
if logLevel >= 30 {
log.Print("ERROR: ", err)
}
return nil, err
}
var html string
html = string(body)
return &html, nil
}
same time print this error in terminal, i want to hide this :
2017/09/29 18:19:28 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nServer: AkamaiGHost\r\nMime-Version: 1.0\r\nDate: Fri, 29 Sep 2017 16:18:21 GMT\r\nContent-Type: text/html\r\nContent-Length: 218\r\nExpires: Fri, 29 Sep 2017 16:18:21 GMT\r\n\r\n<HTML><HEAD>\n<TITLE>Request Timeout</TITLE>\n</HEAD><BODY>\n<H1>Request Timeout</H1>\nThe server timed out while waiting for the browser's request.<P>\nReference 
i want to hide this error, not print in terminal, how i can do this ? Where im wrong ? i use this function for check large list urls if have specific keyword, the list is really big 50 milions urls.
That message comes from http.Transport within the http.Client, specifically here
log.Printf("Unsolicited response received on idle HTTP channel starting with %q; err=%v", buf, peekErr)
Unfortunately, unlike http.Server which has an ErrorLog field for overriding the default logger, http.Transport always writes to the standard logger.
To prevent the message from being printed you can use log.SetOutput(io.Discard) or log.SetOutput(ioutil.Discard) before 1.16. Keep in mind this will discard everything written to the standard logger. As JimB mentions, if you still want to log in other places, you can create a custom logger.
Related
In a simple gRPC example, I connect to the server, request to send a text message, and in return a success message is sent. Once the server is live, the first client request is successful, with no errors.
However, after the first try, each subsequent request (identical to the first) return this error, and does not return a response as the results (the text message is still sent, but the generated ID is not sent back):
rpc error: code = DeadlineExceeded desc = context deadline exceeded
After debugging a little bit, I found that the error
func (c *messageSchedulerClient) SendText(ctx context.Context, in *TextRequest, opts ...grpc.CallOption) (*MessageReply, error) {
...
err := c.cc.Invoke(ctx, "/communication.MessageScheduler/SendText", in, out, opts...)
...
return nil, err
}
returns
rpc error: code = DeadlineExceeded desc = context deadline exceeded
Here is my client code:
func main() {
// Set up a connection to the server.
conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
c := pb.NewMessageSchedulerClient(conn)
var r *pb.MessageReply
r, err = pbSendText(c, false)
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.GetId())
err = conn.Close()
if err != nil {
log.Printf("Connection Close Error: %v", err)
}
return
}
func pbSendText(c pb.MessageSchedulerClient, instant bool) (*pb.MessageReply, error) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second * 5)
minute := timestamppb.New(time.Now().Add(time.Second * 30))
r, err := c.SendText(ctx, &pb.TextRequest{})
if err != nil {
log.Printf("DEBUG MESSAGE: Error after c.SendText(ctx, in): %v", err)
}
cancel()
return r, err
}
and the server code is...
func (s *MessageServer) SendText(ctx context.Context, in *pb.TextRequest) (*pb.MessageReply, error) {
return &pb.MessageReply{Id: GeneratePublicId()}, nil
}
func GrpcServe() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterMessageSchedulerServer(s, &MessageServer{})
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
return
}
const Alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHJKMNOPQRSTUVWXYZ0123457"
// executes instantly, O(n)
func GeneratePublicId() string {
return id.NewWithAlphabet(Alphabet)
}
I've tried changing the context to .TODO from .Background. Doesn't work. I am SURE it's something so simple that I'm missing.
Again, the first time I execute the application from the client side, it works. But after the first client application execution, meaning application execution and beyond -- until I restart the gRPC Server, it gives me the error.
The GeneratePublicId function executes almost instantly, as it is O(n) and uses a random number function.
Most Recent Version of Go (1.153)
Below is the code for reproducibility. Please try to access https://easy-dp.ngrok.io to see the issue.
Here's what I did:
Create a Reverse Proxy accessing Gzipped/ Br encoded Content
Request a publicly available URL, I just grabbed Google Analytics
Attempt to encode and decode the response via an http2 connection with a proxy.modifyresponse function
Watch as content is dropped.
However, this only occurs under the following conditions:
Under SSL, like with https://easy-dp.ngrok.io
When running a proxy.ModifyResponse function
Decompressing and re-compressing the body (for example, just reading and rewriting the body to new bytes works)
package main
import (
"bytes"
"compress/gzip"
"fmt"
"golang.org/x/net/http2"
"golang.org/x/net/http2/h2c"
"io/ioutil"
"net/http"
"net/http/httputil"
"strconv"
"time"
)
func ForwardAnalytics(req *http.Request) {
req.URL.Scheme = "https"
req.URL.Host = "www.google-analytics.com"
req.Host = "www.google-analytics.com"
req.URL.Path = "/analytics.js"
req.Header.Set("Accept-Encoding", "gzip")
}
func ModifyAnalytics(r *http.Response) error {
bytesFromBody, err := ioutil.ReadAll(r.Body)
defer r.Body.Close()
if err != nil {
return nil
}
if r.Header.Get("Content-Encoding") == "gzip" {
gzipReader, err := gzip.NewReader(bytes.NewBuffer(bytesFromBody))
if err != nil {
return nil
}
defer gzipReader.Close()
readableBytes, err := ioutil.ReadAll(gzipReader)
var b bytes.Buffer
gzipWriter, err := gzip.NewWriterLevel(&b, gzip.DefaultCompression)
if err != nil {
return nil
}
defer gzipWriter.Close()
writtenLen, err := gzipWriter.Write(readableBytes)
fmt.Println("Wrote ", writtenLen)
if err != nil {
return nil
}
r.ContentLength = int64(len(readableBytes))
r.Header.Set("Content-Length", strconv.FormatInt(int64(len(readableBytes)), 10))
r.Body = ioutil.NopCloser(&b)
return nil
} else {
return nil
}
}
func handleProxy(w http.ResponseWriter, req *http.Request) {
proxy := httputil.ReverseProxy{
Director: ForwardAnalytics
}
proxy.ModifyResponse = ModifyAnalytics
proxy.ServeHTTP(w, req)
}
func main() {
h2s := &http2.Server{
IdleTimeout: 20 * time.Second,
}
mux := http.NewServeMux()
mux.HandleFunc( "/", handleProxy)
s := &http.Server{
ReadHeaderTimeout: 20 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 30 * time.Second,
Addr: "localhost:8456",
Handler: h2c.NewHandler(mux, h2s),
}
s.ListenAndServe()
}
What did you expect to see?
I expect to see the ability to open the bytes, modify them, and update the response body on an H2C connection
What did you see instead?
Two things of note happen:
Chrome gives a nice little error that expands upon what's going on
{"params":{"description":"Server reset stream.","net_error":"ERR_HTTP2_PROTOCOL_ERROR","stream_id":5},"phase":0,"source":{"id":1493828,"start_time":"732370299","type":1},"time":"732375561","type":224},
Under the normal http connection, there's no problem, but under the https connection the script may or may not print out to a certain length. Sometimes it doesn't print at all, sometimes it prints about 30%.
This is a cross browser issue.
The Content-Length header indicates the size of the entity body in the message, in bytes. The size includes any content encodings (the Content-Length of a gzip-compressed text file will be the compressed size, not the original size).
src
I thought I had tried this but kept running into ERR_CONTENT_LENGTH_MISMATCH because of how I was closing my gzip writer. Related Question
Final handler looked like this:
if r.Header.Get("Content-Encoding") == "gzip" {
gzipReader, err := gzip.NewReader(bytes.NewBuffer(bytesFromBody))
if err != nil {
return nil
}
defer gzipReader.Close()
readableBytes, err := ioutil.ReadAll(gzipReader)
var b bytes.Buffer
gzipWriter, err := gzip.NewWriterLevel(&b, gzip.DefaultCompression)
if err != nil {
return nil
}
writtenLen, err := gzipWriter.Write(readableBytes)
gzipWriter.Close() // This was the culprit. It needed to be closed here
fmt.Println("Wrote ", writtenLen)
if err != nil {
return nil
}
r.ContentLength = int64(b.Len())
r.Header.Set("Content-Length", strconv.FormatInt(int64(b.Len()), 10))
r.Body = ioutil.NopCloser(&b)
return nil
}
Can somebody help me understand what am I doing wrong here, all I'm trying to do is write a Ping message over a net.Conn instance (server), and reply back with a Pong which is expected on a net.Conn instance (client).
I have annotated the code with some errors that I receive.
reader.go
func read(conn net.Conn) {
for {
conn.SetReadDeadline(time.Now().Add(2 * time.Second))
_, op, err := wsutil.ReadClientData(conn)
if err != nil {
log.Printf("wsmanager read: %v", err) // read: write pipe: deadline exceeded
return
}
if op != ws.OpPing {
continue
}
c.conn.SetWriteDeadline(time.Now().Add(3 * time.Second))
if err = wsutil.WriteServerMessage(c.conn, ws.OpPong, []byte{}); err != nil {
log.Printf("wsmanager: send pong error: %v", err)
return
}
}
}
// reader_test.go
client, server := net.Pipe()
go read(server) // starts the loop above
err := wsutil.WriteClientMessage(server, ws.OpPing, []byte{})
if err != nil {
t.Fatalf("failed sending pings message %v", err)
}
_, op, err := wsutil.ReadServerData(client)
if err != nil {
t.Errorf("exp no err, got %v", err)
}
if op != ws.OpPong {
t.Errorf("exp ws.OpPong, got %v", op)
}
Thank you for using this library :)
As the doc states, the ReadData functions read data from the connection; that is, application specific data, not the control messages. Control frames are handled implicitly in these functions. If you want to read any kind of message, you could use wsutil.Reader or the plain ws.Read functions.
https://godoc.org/github.com/gobwas/ws/wsutil#ReadClientData
I have use case where a Go Client with a file-watcher listens to changes in a file and sends these changes to a Go server. If the Client can't reach the server during the POST requests with the payload of the file changes, the Client will keep trying every 3 seconds to send the request until err := http.NewRequest() dosen't return a non-nil err
But If the Client is currently trying every 3 seconds to send a POST request but at the same time a new change occurs to the file under file-watch, I want the current POST requests's payload to be overwritten by the new payload(new changes from the file)
How Do I archive this best?
Client code for sending an HTTP requests
func makeRequest(method string, body io.Reader) (*http.Response, error) {
client := &http.Client{}
request, err := http.NewRequest(.., .., ..)
if err != nil {
log.Println("Error: Couldn't make a new Request:", err)
return nil, err
}
response, err := client.Do(request)
if err != nil {
log.Printf("Error: Couldn't execute %s request:%s", method, err)
}
return response, err
}
The function that retries until err !=nil
func autoRetry(f func() error) {
if err := backoff.Retry(f, getBackOff()); err != nil {
log.Println("Error: Couldn't execute exponential backOff retries: ", err)
}
}
autoRetry() is just a function which takes a function and uses ExponentialBackOff to calculate the amount of tries until err !=nil
The call to the method doing the POST request with retries
func postTodo() {
autoRetry(func() error {
r, err := makeRequest("POST", getFileData())
if err != nil {
return err
}
if r.StatusCode != 200 {
return errors.New("Error:" + r.Status)
}
return nil
})
}
Is there any way how to serve files in Go with GRPC, like in gin-gonic's variant:
router.Static("/static", "/var/www")
You can't do it exactly like that.
But you can use the proto bytes type and put the file bytes in that field.
Also (as pointed out in the comments) with large files you should use streaming instead of a unary call. (most GRPC implementation have a limit of 4MB per message).
Proto example:
syntax = "proto3";
message Response {
bytes fileChunk = 1;
}
message Request {
string fileName = 1;
}
service TestService {
rpc Download(Request) returns (stream Response);
}
Server implementation example:
func (srv *Server) Download(req *pbgo.Request, responseStream pbgo.TestService_DownloadServer) error {
bufferSize := 64 *1024 //64KiB, tweak this as desired
file, err := os.Open(req.GetFileName())
if err != nil {
fmt.Println(err)
return err
}
defer file.Close()
buff := make([]byte, bufferSize)
for {
bytesRead, err := file.Read(buff)
if err != nil {
if err != io.EOF {
fmt.Println(err)
}
break
}
resp := &pbgo.Response{
FileChunk: buff[:bytesRead],
}
err = responseStream.Send(resp)
if err != nil {
log.Println("error while sending chunk:", err)
return err
}
}
return nil
}
Client would call it like this:
conn, err := grpc.Dial("localhost:9090", grpc.WithInsecure())
if err != nil {
log.Fatal("client could connect to grpc service:", err)
}
c := pbgo.NewTestServiceClient(conn)
fileStreamResponse, err := c.Download(context.TODO(), &pbgo.Request{
FileName: "test.txt",
})
if err != nil {
log.Println("error downloading:", err)
return
}
for {
chunkResponse, err := fileStreamResponse.Recv()
if err == io.EOF {
log.Println("received all chunks")
break
}
if err != nil {
log.Println("err receiving chunk:", err)
break
}
log.Printf("got new chunk with data: %s \n", chunkResponse.FileChunk)
}
If you need to be able to serve arbitrary files, you would need to handle which files you allow serving (say someone requests the file /etc/passwd or something).
Not sure what exactly is the use case here.