I'm going to send http request via ssl proxy in Go.
Unfortunately, it always fails. Here is my code I have tried.
proxyUrl, err := url.Parse("https://" + proxyURLStr)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
}
//adding the Transport object to the http Client
client := &http.Client{
Transport: transport,
Timeout: 60 * time.Second,
}
request, err := http.NewRequest("GET", "https://test.com", nil)
if err != nil {
log.Print("Create Request Error - test.com ")
return -2, invalid_data
}
random_agent := browser.Random()
request.Header.Set("User-Agent", random_agent)
// Make request
response, err := client.Do(request)
if err != nil {
log.Println(err)
current := time.Now()
elapsed := current.Sub(start)
log.Printf("test.com is not available. %02d, %02d ", int(elapsed.Minutes()), int(elapsed.Seconds()))
return -2, invalid_data
}
I get this error:
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I coded the same function in Python and tested it — works good.
But the Go code does not work.
Related
I have a program in Go which takes around 10k urls (same base url, simply different resource) and request response for them from 10k goroutines.
At some point, I start receiving :
Get "https://.....": dial tcp <some_ip>: connect: operation timed out
I can't understand whether that's due to some Go limitation, my local machine (macbook) limitation, or the limitations of the Server(s).
And how can it be solved.
My code is simple :
var transport = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
Dial: (&net.Dialer{
Timeout: 0,
KeepAlive: 0,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
}
var httpClient = &http.Client{Transport: transport}
func main() {
for id := 1; id <= total; id++ {
myWaitGroup.Add(1)
go checkUri(myBaseUrl, id, some_chan)
}
myWaitGroup.Wait()
}
func checkUri(baseUrl string, id int, myChan string) {
defer myWaitGroup.Done()
url := fmt.Sprintf(`%s/%d.json`, baseUrl, id)
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Connection", "close")
response, err := httpClient.Do(req)
if err != nil {
fmt.Println("ERROR: http remote call, err:", err)
return
} else {
defer response.Body.Close()
if response.StatusCode != 200 {
fmt.Printf("ERROR: remote call status code %d [%s]\n", response.StatusCode, url)
return
} else {
.. io.ReadAll(response.Body)
myChan <- fmt.Sprintf(string(b))
..
}
}
}
I'm building a small service that interacts with other server.
I must send a cookie with session token to that server. And use it later for my identification. But that cookie isn't set. And no cookies are sent back.
The request is made over https protocol.
Here's the code:
// ---> create Client
client := &http.Client{}
// ---> create Request
req, err := http.NewRequest("GET", indexURL, nil)
if err != nil {
fmt.Printf("NewRequest fail: %s\n", err)
}
// ---> define a cookie
cookie := http.Cookie{Name: "sess", Value: "value", HttpOnly: true, MaxAge: 0, Path: "/", Domain: "the.server.domain"}
req.AddCookie(&cookie)
// ---> fire in the hole!
resp, err := client.Do(req)
if err != nil {
fmt.Printf("client.Do fail: %s\n", err)
}
defer resp.Body.Close()
// ---> read the cookies
fmt.Printf("cookies: %d\n", len(resp.Cookies())) // prints zero (((
// ---> not even launched ((
for _, cookie := range resp.Cookies() {
fmt.Printf("GET cookie[%s] = %s\n", cookie.Name, cookie.Value)
}
What I am doing wrong?
I tried also
req.Header.Set(`Cookie`, `sess=value`)
but with no effect
Your code seems fine, it must be an issue with the server. Even without sending
a cookie, I can get a cookie back:
package main
import "net/http"
func main() {
r, e := http.Get("https://stackoverflow.com")
if e != nil {
panic(e)
}
defer r.Body.Close()
c := r.Cookies()
println(len(c) == 1)
}
When I try to resend requests from simple proxy
http.HandleFunc("/",func(w http.ResponseWriter, r *http.Request) {
log.Printf("proxy rq: %v", r)
client := &http.Client{}
proxyRes, err := client.Do(r) // 👈🏻 Get "http://localhost:8097/tmp": http: Request.RequestURI can't be set in client requests
if err != nil {
log.Fatalf("err proxy request: %v",err)
}
resBody := proxyRes.Body
defer resBody.Close()
if _, err := io.Copy(w, resBody); err != nil {
log.Printf("copy error:%v\n", err)
}
})
http.ListenAndServe(":8099", nil)
, with set http_proxy ENV (to send requests thought my proxy)
% export http_proxy=http://localhost:8099
% curl -v http://localhost:8097/tmp
I get an error like
Get "http://localhost:8097/tmp": http: Request.RequestURI can't be set in client requests
What did I miss?
You can't use the client request as a parameter of Do. Create a new request with the same parameter as r, then perform Do on this request
The error is defined in the ../src/net/http/client.go:217 (https://go.dev/src/net/http/client.go) as:
if req.RequestURI != "" {
req.closeBody()
return nil, alwaysFalse, errors.New("http: Request.RequestURI can't be set in client requests")
}
You should be able to reuse existing r *http.Request if you set r.RequestURI = "" before sending a HTTP request:
r.RequestURI = ""
proxyRes, err := client.Do(r)
I have proxyString 123.44.333.42:51244
and want to request url through the proxy
proxyString := "123.44.333.42:51244"
proxyURL, err := url.Parse("http://"+proxyStr)
if err != nil {
panic(err)
}
////adding the proxy settings to the Transport object
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
client := &http.Client{
Transport: transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
}}
resp, contents := getContent(client, person.personHref)
get errror Proxy Authentication Required
/* Keep Alive Client*/
HttpClient{
Client: &http.Client{
Transport: &http.Transport{
Dial: (&net.Dialer{
Timeout: dialTimeout,
KeepAlive: dialTimeout * 60,
}).Dial,
DisableKeepAlives: false,
MaxIdleConnsPerHost: idleConnectionsPerHost,
},
},
Timeout: 5 * time.Second,
}
/* Execute Request */
timeoutContext, cancelFunction := context.WithTimeout(context.Background(), self.Timeout)
defer cancelFunction()
if response, err = self.Client.Do(request.WithContext(timeoutContext)); err == nil {
defer response.Body.Close()
/* Check If Request was Successful */
statusCode = response.StatusCode
if response.StatusCode == http.StatusOK {
/* Read Body & Decode if Response came & unmarshal entity is supplied */
if responseBytes, err = ioutil.ReadAll(response.Body); err == nil && unmarshalledResponse != nil {
//Process Response
}
} else {
err = errors.New(fmt.Sprintf("Non 200 Response. Status Code: %v", response.StatusCode))
}
}
In Golang whenever a request times out in a Keep-Alive connection that connection is lost and is Reset. For the above code incase of timeout Seeing Packets in Wireshark reveals that RST is sent by the client hence connection is no longer reused.
Event tried using Timeout of Httpclient rather than ContextWithTimeout but having similar findings where connection gets reset.
Anyway we can retain established keep-alive connection even in case of a Timeout of request.
The net/http client closes the connection on timeout because the connection cannot be reused.
Consider what would happen if the connection is reused. If the client receives none of the response or a partial response from the server before timeout, then the next request will read some amount of the previous response.
To keep the connection alive on timeout, implement the timeout in application code. Continue to use a longer timeout in the net/http client to handle cases where the connection is truly stuck or dead.
result := make(chan []byte)
go func() {
defer close(result)
resp, err := client.Do(request)
if err != nil {
// return
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
p, err := ioutil.ReadAll(resp.Body)
if err != nil {
return
}
result <- resp
}
}()
select {
case p, ok <- result:
if ok {
// p is the response body
}
case time.After(timeout):
// do nothing
}