I'm trying to get Go to use an internal enterprise Go-Proxy for module download - which requires an http_proxy to be accessible (enterprise firewall). However go get -u golang.org/x/lint/golint fails:
package golang.org/x/lint/golint: unrecognized import path "golang.org/x/lint/golint": https fetch: Get "https://golang.org/x/lint/golint?go-get=1": Forbidden
My setup:
http_proxy and https_proxy environment variables are set
no_proxy does not contain the IP or hostname of my Go-Proxy
GOPROXY is set (go env -w GOPROXY=https://artifactory.mycompany.com/api/go/myrepo-go-virtual)
I checked:
Using curl and directly querying the GOPROXY server works fine and I can download the file (so the https_proxy setting works)
Counter-check with curl and explicitly unsetting http/https_proxy: No connection, as expected
Using tcpdump, I discovered that running go get seems to ignore my GOPROXY and ask my http_proxy to connect directly to the original url on golang.org (Options/sequence and ack numbers omitted for brevity), which the proxy/firewall blocks.
06:52:53.926397 IP <my_ip.port> > <proxy.port>: Flags [S], length 0
06:52:53.927206 IP <proxy.port> > <my_ip.port>: Flags [S.], length 0
06:52:53.927232 IP <my_ip.port> > <proxy.port>: Flags [.], length 0
06:52:53.932003 IP <my_ip.port> > <proxy.port>: Flags [P.], length 89: HTTP: CONNECT golang.org:443 HTTP/1.1
06:52:53.932638 IP <proxy.port> > <my_ip.port>: Flags [.], length 0
06:52:53.933100 IP <proxy.port> > <my_ip.port>: Flags [P.], length 3939: HTTP: HTTP/1.1 403 Forbidden
Question: Why does Go ignore the GOPROXY? Did I not set something up correctly?
I'm using Go 1.15.3 in the golang:1.15.3 Docker container (with some added tools to check connectivity)
Try this: set GO111MODULE=on to use GOPROXY
Or run go mod init before you run go get
Related
I tried to download files from google drive using wget (on windows) using the script shown here:
[JULY 2020 - Windows users batch file solution]wget/curl large file from google drive.
It works well, but when computer is behind proxy, it will work ONLY if I will set environment variables PROXY_HTTP and PROXY_HTTP (It may be that it can also set by flag in the command, but I didn't try it)
The fact that I can download files from google drive using Chrome without these environment variables and without setting chrome for the proxy gives me the understanding that there is a way to download behind proxy without setting an application for the proxy.
How can I make wget works without need to set it manually (by flag or by environment variables) for the proxy?
In all likelyhood your Chrome also has a proxy set up in some way. In any case, the simplest way to define the proxy for wget is to create a .wgetrc file in your local home folder and set the following:
> vi ~/.wgetrc
use_proxy=on
http_proxy=http://[proxy_ip]:[proxy_port]
https_proxy=https://[proxy_ip]:[proxy_port]
ftp_proxy=http://[proxy_ip]:[proxy_port]
That should be all you need to do.
I found the solution after #Wilmar comment which he pointed out here (thanks!).
An application can automatically finds if it is behind a proxy by sending "http://wpad/wpad.dat".
If a proxy server is behind, it will answer with a message that contains PAC file with proxy details. The application then can extract the proxy details for any needed settings. Thats how Chrome can set itself for automatically for proxy.
Example using wget in windows to find proxy details
In Windows, you can use wget as follows to get the proxy server details. The details must be extracted from the text messages and you can use tool like jrepl for such task. Here I only show where the details are.
call wget "http://wpad/wpad.dat" -o "ProcessLog.txt" -O "PAC.txt"
There are three possible scenarios here:
In case there is no proxy behind, then PAC.txt is empty and ProcessLog.txt contains text message similar to this one.
ProcessLog.txt
--2020-09-01 08:38:29-- http://wpad/wpad.dat
Resolving wpad (wpad)... failed: The requested name is valid, but no data of the requested type was found. .
wget: unable to resolve host address 'wpad'
In case there is a proxy server behind, and windows environment variables for proxy are set:
http_proxy=http://proxy.mc.company.com:777
https_proxy=https://proxy.mc.company.com:777
then wget already knows the proxy address so PAC.txt is empty and ProcessLog.txt contains text message similar to the follow one that contains the proxy details. In this example, the proxy details are [proxy_ip]:[proxy_port] = proxy.mc.company.com:777
ProcessLog.txt
--2020-09-01 08:29:59-- http://wpad/wpad.dat
Resolving proxy.mc.company.com (proxy.mc.company.com)... 10.100.200.150
Connecting to proxy.mc.company.com (proxy.mc.company.com)|10.100.200.150|:777... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://www.wpad.com/wpad.dat [following]
--2020-09-01 08:30:00-- http://www.wpad.com/wpad.dat
Connecting to proxy.mc.company.com (proxy.mc.company.com)10.100.200.150|:777... connected.
Proxy request sent, awaiting response... 403 Forbidden
2020-09-01 08:30:00 ERROR 403: Forbidden.
In case there is a proxy server behind, but no windows environment variables for proxy are set, then wget gets the proxy details from proxy server. In this case PAC.txt contains long text message similar to the follow one that contains the proxy details. In this example, the proxy details are [proxy_ip]:[proxy_port] = proxy.mc.company.com:777
PAC.txt
function FindProxyForURL(url,host) {
var me=myIpAddress();
var resolved_ip = dnsResolve(host);
if (host == "127.0.0.1") {return "DIRECT";}
if (host == "localhost") {return "DIRECT";}
if (isPlainHostName(host)) {return "DIRECT";}
if (url.substring(0,37) == "http://lyncdiscoverinternal.company.com") {return "DIRECT";}
if (!resolved_ip) { if (url.substring(0,6) == "https:") {return "PROXY proxy-mc.company.com:778";} else {return "PROXY proxy-mc.company.com:777";}}
if (host == "moran-for-localhost-only.com") {return "DIRECT";}
...
...
Simplifying using wget in windows to find proxy details
When using wget to find proxy details, we can command it to ignore proxy environment variables (if are set) using the flag --no-proxy. This leaves us with only two possible scenarios (1) and (3) described above. So we just need the ProxyInfo file. If it is empty (scenario 1) then no proxy is behind, if it contains text (scenario 3), it is behind a proxy and you can extract the proxy details from it.
call wget --no-proxy "http://wpad/wpad.dat" -O "PAC.txt"
I´m newbie in this matter, and I don´t know why my application just work and runs in a open network, when is behind a proxy I have a return error.
I´m using a raspberry zero, with raspbian Stretch, using azure-iot-sdk-python and proxy squid
I already try this things:
The proxy allow HTTPS connection, and it has all PORT are available and without any restriction and the address *****. azure-devices.net is put inside a whitelist in
$ nano / etc / squid / whitelist
Beyond that I set the proxy in the operate system, raspbian Stretch, in the
$ nano / etc / environment
the follow configurations:
export http_proxy = "http://192.168.2.254:3128/"
export https_proxy = "https://192.168.2.254:3128/"
export no_proxy = "localhost, 127.0.0.1"
And also in
$ nano ~ / .bashrc
export http_proxy = http: //192.168.2.254:3128
export https_proxy = https: //192.168.2.254:3128
export no_proxy = localhost, 127.0.0.1
And,
$ nano /etc/apt/apt.conf.d/90proxy
Acquire :: http :: Proxy "http://192.168.2.254:3128/";
Acquire :: https :: Proxy "https://192.168.2.254:3128/";
from iothub_client import IoTHubClient, IoTHubTransportProvider, IoTHubMessage
import time
CONNECTION_STRING = "HostName=******.azure-devices.net;DeviceId=***;SharedAccessKey=*********"
PROTOCOL = IoTHubTransportProvider.MQTT
def send_confirmation_callback(message, result, user_context):
print("Confirmation received for message with result = %s" % (result))
if __name__ == '__main__':
client = IoTHubClient(CONNECTION_STRING, PROTOCOL)
message = IoTHubMessage("test message")
client.send_event_async(message, send_confirmation_callback, None)
print("Message transmitted to IoT Hub")
while True:
time.sleep(1)
Error: File: /usr/sdk/src/c/c-utility/adapters/socketio_berkeley.c Func: lookup_address_and_initiate_socket_connection Line: 282 Failure: getaddrinfo failure -3.
Error: File: /usr/sdk/src/c/c-utility/adapters/socketio_berkeley.c Func: socketio_open Line: 765 lookup_address_and_connect_socket failed
Error: File: /usr/sdk/src/c/c-utility/adapters/tlsio_openssl.c Func: on_underlying_io_open_complete Line: 760 Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.
Error: File: /usr/sdk/src/c/c-utility/adapters/tlsio_openssl.c Func: tlsio_openssl_open Line: 1258 Failed opening the underlying I / O.
Error: File: /usr/sdk/src/c/umqtt/src/mqtt_client.c Func: mqtt_client_connect Line: 1000 Error: io_open failed
Error: File: /usr/sdk/src/c/iothub_client/src/iothubtransport_mqtt_common.c Func: SendMqttConnectMsg Line: 2122 failure connecting
You can not use a HTTP proxy with (native) MQTT, they are 2 totally separate protocols.
If you can use MQTT over WebSockets then you should be able to use a HTTP proxy as WebSockets are initially established by upgrading a HTTP connection.
If you have a SOCKS proxy available on your network, then you may be able to use that with native MQTT. The following question has hints on how to use a SOCKS proxy with Python. How can I use a SOCKS 4/5 proxy with urllib2?
So I am trying to install my Go app's dependencies on my Raspberry 3/Raspbian system with "go get" and running into the following when trying to install gRPC for Go:
[pi#raspberrypi-1 camera-service] 17:32:28 % go get google.golang.org/grpc
package google.golang.org/grpc: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp: lookup google.golang.org on 192.168.1.1:53: read udp 192.168.1.64:33524->192.168.1.1:53: i/o timeout)
Meanwhile I am able to install other (non-google.golang.org) dependencies (for example go get github.com/asaskevich/EventBus) just fine.
To me this looks a DNS problem -- 192.168.1.1 is my router, 192.168.1.64 is my RPi. However I can resolve the address just fine:
[pi#raspberrypi-1 camera-service] 17:32:52 % host google.golang.org
google.golang.org is an alias for golang.org.
golang.org has address 216.58.209.145
golang.org has IPv6 address 2a00:1450:400f:804::2011
golang.org mail is handled by 1 aspmx.l.google.com.
golang.org mail is handled by 2 alt2.aspmx.l.google.com.
golang.org mail is handled by 2 alt1.aspmx.l.google.com.
golang.org mail is handled by 2 alt3.aspmx.l.google.com.
This is weird. Also I have installed the same packages on another system np.
My environment is
- Go 1.8.3, installed from prebuilt ARM6 binary
- export GOROOT="/usr/local/go"
- export GOPATH="/home/pi/go"
This may be because you are using the built in DNS resolver in Go, rather than delegating to the system name resolver. When you built gRPC, did you set CGO_ENABLED=1 ? You can check by setting GODEBUG=netdns=cgo when getting gRPC:
$ GODEBUG=netdns=cgo go get google.golang.org/grpc
I am working on an application that connects to a legacy database, Eloquence, through ODBC and SQL/R. I set up my server with UnixODBC and setup the drivers and datasources as follows:
File /etc/odbcinst.ini
[SQLR]
Description=SQLR for Elqouence
Driver=/opt/sqlr/lib/libsqlrodbc.so
Driver64=/opt/sqlr/lib64/libsqlrodbc64.so
FileUsage = 1
File /etc/odbc.ini
[reservations]
Description = SQLR datasource for RES database
Driver = SQLR
Database = res
Servername = eloq-dev
Port = 8003
UserName = sqlrodbc
I confirmed that I can connect to the datasource by running isql reservations and I ran a couple of queries to make sure. No issues. Then I connected my Ruby code up to the database using the ODBC gem and the following code:
require 'rdbi-driver-odbc'
RDBI.connect :ODBC, db: "reservations"
Which outputs the following error:
Unable to connect to host.
Host 127.0.0.1, Service sqlrodbc
errno 111: Connection refused
ODBC::Error: 08001 (3047) [unixODBC][Marxmeier][SQL/R ODBC Client]connection failure
I'm concerned that it's using 127.0.0.1 as the host even though the eloq-dev hostname is set in file /etc/hosts to a different address. I'm also concerned that isql works, but the ODBC gem doesn't.
Additionally, when I use the tcpdump command, the only output related to my connection is this:
tcpdump -i lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
18:38:39.688264 IP localhost.50447 > localhost.mcreport: Flags [S], seq 3355035364, win 43690, options [mss 65495,sackOK,TS val 1655798115 ecr 0,nop,wscale 7], length 0
18:38:39.688280 IP localhost.mcreport > localhost.50447: Flags [R.], seq 0, ack 3355035365, win 0, length 0
No packets are going out over the network at all.
I've also changed my code to use RDBI instead of Ruby-ODBC, but I have the same issue.
My issue was ultimately twofold. I was connecting to Eloquence and SQL/R over a VPN connection which wasn't as stable as I thought and so connections were dropping as a result.
The other issue was that SQL/R uses Server instead of ServerName and Service instead of Port in the odbc.ini file.
Once I stabilized my VPN and fixed the odbc.ini file I was able to connect without issue.
I try to use go get gopkg.in/fatih/pool.v2 to install pool according to Readme.md, but can't success:
C:\Users\xiaona\Documents\GitHub> go get -v gopkg.in/fatih/pool.v2
Fetching https://gopkg.in/fatih/pool.v2?go-get=1
https fetch failed.
Fetching http://gopkg.in/fatih/pool.v2?go-get=1
import "gopkg.in/fatih/pool.v2": http/https fetch: Get http://gopkg.in/fatih/poo
l.v2?go-get=1: dial tcp 107.178.216.236:80: ConnectEx tcp: A connection attempt
failed because the connected party did not properly respond after a period of ti
me, or established connection failed because connected host has failed to respon
d.
package gopkg.in/fatih/pool.v2: unrecognized import path "gopkg.in/fatih/pool.v2
"
While access 107.178.216.236:80 is OK in web browser.
I try to use go get github.com/fatih/pool, and it also works OK.
Could anyone give some clues on this issue?
BTW: I use windows powershell and git version is 1.9.5.
The root cause has been found: Because my computer use a web proxy, so I need to set proxy in environment variable:
C:\Users\xiaona>set https_proxy=https://web-proxy.corp.hp.com:8080/
C:\Users\xiaona>set http_proxy=https://web-proxy.corp.hp.com:8080/
C:\Users\xiaona>go get -v gopkg.in/fatih/pool.v2
Fetching https://gopkg.in/fatih/pool.v2?go-get=1
Parsing meta tags from https://gopkg.in/fatih/pool.v2?go-get=1 (status code 200)
get "gopkg.in/fatih/pool.v2": found meta tag main.metaImport{Prefix:"gopkg.in/fa
tih/pool.v2", VCS:"git", RepoRoot:"https://gopkg.in/fatih/pool.v2"} at https://g
opkg.in/fatih/pool.v2?go-get=1
gopkg.in/fatih/pool.v2 (download)
gopkg.in/fatih/pool.v2
Then, all is OK!