go version: go1.13.5 linux/amd64
I am using "x/net/proxy" to connect with the "http_proxy".
I have referred following proxy page:
https://godoc.org/golang.org/x/net/proxy
To get proxy information I have set environment variable "all_proxy" to the the desired proxy "http://192.130.0.10:3200", and performed the tcp connection, but following error is raised:
[Network Error : socks connect tcp 192.130.0.10:3200->mx.eu1.mico.io:8883: read tcp 172.17.0.2:48118->192.130.0.10:3200: read: connection reset by peer]
I have looked "x/net/proxy", It seems "http_proxy" support is not available instead of "SOCKS5" proxy is supported. I have similar implementation for "http_proxy", but unfortunately it does not worked.
I have created a sample code (with port 1883) which is working for the non proxy environment, Please suggest how I can enable "http_proxy" or "https_proxy" support?
package main
import (
"fmt"
"os"
"golang.org/x/net/proxy"
)
//The host address which we want to connect with the proxy
var host = "google.com:80"
func main() {
fmt.Println("Inside main...")
//Setting the proxy before starting the application
if os.Getenv("http_proxy") == "" {
os.Setenv("http_proxy", "http://192.130.0.10:3200")
}
os.Setenv("all_proxy", os.Getenv("http_proxy"))
if os.Getenv("all_proxy") != os.Getenv("http_proxy") {
fmt.Println("Environment variables are not matching...")
return
}
fmt.Println("System proxy is:", os.Getenv("all_proxy"))
proxyDialer := proxy.FromEnvironment()
fmt.Println("Connecting to...", host)
conn, err := proxyDialer.Dial("tcp", host)
if err != nil {
fmt.Println("Unable to dial...", err)
return
}
fmt.Println("Connected...", conn)
}
Output:
Inside main...
System proxy is: http://192.130.0.10:3200
Connecting to... google.com:80
Unable to dial... dial tcp 172.217.23.174:80: connect: connection timed out
What's your purpose?
If you need to use an http-proxy server for http requests, you can just configure your http-client without another package usage:
package main
import (
"fmt"
"net/http"
"net/url"
"time"
)
func main() {
proxyUrl, err := url.Parse("http://192.130.0.10:3200")
if err != nil {
// TODO handle me
panic(err)
}
cl := http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
},
Timeout: 3000 * time.Millisecond,
}
resp, err := cl.Get("http://google.com")
if err != nil {
// TODO handle me
panic(err)
}
// TODO work with the response
fmt.Println(resp)
}
Related
I'm trying to dial using credentials and maintain a connection with a socks5 proxy server in Go.
This works nicely if I have IP authorisation set up with the proxy provider, however there is no way pass any auth credentials using net.Dial function in Go:
package main
import (
"io"
"net"
)
func main() {
dst, err := net.Dial("tcp", "11.22.33.44:1111")
if err != nil {
panic("Dial Error:" + err.Error())
}
dst.Close()
}
Go has a useful proxy library and allows authenticated forward requests via proxy using this:
package main
import (
"io"
"net"
)
func main() {
var proxyAuth *proxy.Auth
if conf.Username != "" {
proxyAuth = new(proxy.Auth)
proxyAuth.User = conf.Username
proxyAuth.Password = conf.Password
}
proxyconn, _ := proxy.SOCKS5("tcp", "11.11.11.11:1111", proxyAuth, nil) //returns a Dialer with proxy that can be invoked to connect to another address
dst := proxyconn.Dial("tcp", "22.33.44.55:6666") //connects to an address via proxy
dst.Close()
}
However it returns a Dialer that then asks to connect a target/ultimate address through this authenticated proxy rather the proxy server itself:
My objective here is to return a net.conn connection with a credentials-authenticated proxy server - something like this:
package main
import (
"io"
"net"
)
func main() {
//net.Dial does not have a way to pass Auth creds
dst := net.Dial("tcp", "22.33.44.55:6666", proxyAuth)
dst.Close()
}
The net.Dial() method doesn't concerned with proxy authentication. However, If you want proxy authentication you can set it in header of the request before the call. Please refer this link
dst := net.Dial("tcp", "22.33.44.55:6666")
Running into some error, I must be overlooking something.
How can I debug this? Dropping connections?
I read the following:
golang - Why net.DialTimeout get timeout half of the time?
Go. Get error i/o timeout in server program
golang get massive read tcp ip:port i/o timeout in ubuntu 14.04 LTS
Locating the "read tcp" error in the Go source code
Getting sporadic "http: proxy error: read tcp: i/o timeout" on Heroku
Error created here:
https://github.com/golang/go/blob/b115207baf6c2decc3820ada4574ef4e5ad940ec/src/net/net.go#L179
Goal:
Send a Get request to a url.
Expected result:
return body in JSON.
Encountered problem:
I/O timeout
It works in Postman
Edit:
I added a modified timeout...
Edit2: traced error
Postman request:
GET /v2/XRP-EUR/candles?interval=1h HTTP/1.1
Host: api.bitvavo.com
Postman Result (1440 rows):
[
[
1609632000000,
"0.17795",
"0.17795",
"0.17541",
"0.17592",
"199399.874013"
],
[
1609628400000,
"0.17937",
"0.18006",
"0.17622",
"0.17852",
"599402.631894"
],
[
1609624800000,
"0.18167",
"0.18167",
"0.17724",
"0.17984",
"579217.962574"
],.....
Code:
package main
import (
"fmt"
"net/http"
"io/ioutil"
"time"
)
func main() {
url := "https://api.bitvavo.com/v2/XRP-EUR/candles?interval=1h"
method := "GET"
client := &http.Client {
}
client.Timeout = time.Second * 60
req, err := http.NewRequest(method, url, nil)
if err != nil {
fmt.Println(err)
return
}
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(body))
}
result:
Get "https://api.bitvavo.com/v2/XRP-EUR/candles?interval=1h": dial tcp 65.9.73.10:443: i/o timeout
I was with this issue when building inside docker containers.
Not sure why, but after a docker swarm leave --force and a systemctl restart docker the build worked.
Local environment, firewall not allowing golang to dial tcp..
It still allowed the url to be resolved to an ip though (DNS)
Solution:
Change firewall settings locally,
Check Docker/kubernetes/reverse proxy settings
I use memcache for caching and the client I use is https://github.com/bradfitz/gomemcache. When I tried initiate new client with dummy/invalid server address and then pinging to it, it return no error.
package main
import (
"fmt"
m "github.com/bradfitz/gomemcache"
)
func main() {
o := m.New("dummy_adress")
fmt.Println(o.Ping()) // return no error
}
I think it suppose to return error as the server is invalid. What do I miss?
It looks like the New() call ignores the return value for SetServers:
func New(server ...string) *Client {
ss := new(ServerList)
ss.SetServers(server...)
return NewFromSelector(ss)
}
The SetServers() function will only set the server list to valid servers (in
your case: no servers) and the Ping() funtion will only ping servers that are
set, and since there are no servers set it doesn't really do anything.
This is arguably a feature; if you have 4 servers and one is down then that's
not really an issue. Even with just 1 server memcache is generally optional.
You can duplicate the New() logic with an error check:
ss := new(memcache.ServerList)
err := ss.SetServers("example.localhost:11211")
if err != nil {
panic(err)
}
c := memcache.NewFromSelector(ss)
err = c.Ping()
if err != nil {
panic(err)
}
Which gives:
panic: dial tcp 127.0.0.1:11211: connect: connection refused
I am using gorilla mux to create a golang server to support a simple health GET endpoint.
The endpoint responds with a status of ok whenver the server is up.
I see a lot of connections (over 400) in CLOSE_WAIT state on one system.
This does not happen on other systems with the same code.
Output of netstat (9003 is my server port):
tcp 164 0 ::1:9003 ::1:60702 CLOSE_WAIT -
tcp 164 0 ::1:9003 ::1:44472 CLOSE_WAIT -
tcp 164 0 ::1:9003 ::1:31504 CLOSE_WAIT -
This seems to imply that I have a connection I need to close.
Most of the questions I read online seem to suggest that open connections pertain to the client not issuing a response.body.close() after a GET.
As per https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/, I could add read/write timeouts on server side but I would like to understand the root cause of CLOSE_WAITS before adding the improvements.
Am I missing any close on the server side?
My code is below:
import "github.com/gorilla/mux"
...
func (server *Srvr) healthHandler(w http.ResponseWriter, r *http.Request) {
resp := map[string]string{"status": "ok"}
respJSON, err := json.Marshal(resp)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
fmt.Fprintf(w, "Error creating JSON response %s", err)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write(respJSON)
}
// Load initializes the servers
func Load(port string) *Srvr {
srvrPort := ":" + port
log.Infof("Will listen on port %s", srvrPort)
serverMux := mux.NewRouter()
srvr := &Srvr{Port: port, Srv: &http.Server{Addr: srvrPort, Handler: serverMux}}
serverMux.HandleFunc("/api/v1.0/health", srvr.healthHandler).Methods("GET")
return srvr
}
// Run starts the server
func (server *Srvr) Run() {
log.Info("Starting the server")
// Starting a server this way to allow for shutdown.
// https://stackoverflow.com/questions/39320025/how-to-stop-http-listenandserve
err := server.Srv.ListenAndServe()
if err != http.ErrServerClosed {
log.Fatalf("ListenAndServe(): %s", err)
}
}
// Main resides outside the server package
func main() {
srvr := server.Load("9003")
// Now that all setup is done successfully, lets start the server
go srvr.Run()
// An unrelated forever loop executes below for different business logic
for {
glog.Info("Evaluation iteration begins now")
...
time.Sleep(time.Duration(evalFreq) * time.Minute)
}
}
I've been working on finding a way to try and override the default DNS server for a Go program for a while but still with no success sadly.
The current solution which I had thought would work was:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
DNS := "1.1.1.1"
net.DefaultResolver = &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
d := net.Dialer{
Timeout: time.Millisecond * time.Duration(3000),
}
return d.DialContext(ctx, "udp", fmt.Sprintf("%s:53", DNS))
},
}
resp, err := net.LookupHost("tsdfsdf.com")
if err != nil {
fmt.Printf(err.Error())
}
fmt.Printf("%+v", resp)
}
But the response from this is:
lookup tsdfsdf.com on 192.168.0.1:53: no such host[]
Which is sadly my default DNS server set in my reslov.conf
I have tried forcing to use the Go Resolver by setting export GODEBUG=netdns=go
The long term solution is to be able to over the default resolver for the HTTP Client which would, in turn be consumed by some AWS SDK stuff.
Has any faced this or knows how I can get around this?
FYI, I know "tsdfsdf.com" is not a real domain I'm just using it to spit a No such host error to see what DNS it asked.