set up conda for caching proxy (MacOs + Squidman) - proxy

How can I configure conda to use a caching proxy?
So far I have:
installed SquidMan and set its host and port to 127.0.0.1:8080.
set the network settings to proxy both http and https at this address
edited .condarc to use a proxy
I think that SquidMan is set up correctly. If I switch it off and try to browse the internet, I get an error message "The proxy server is refusing connections". This happens for both http and https websites and also if I enter an IP directly (no DNS in between).
The edited .condarc is this:
proxy_servers:
http: http://127.0.0.1:8080
https: http://127.0.0.1:8080
Those are the same addresses as in the system proxy settings - which seem to work fine for browsing.
As a test I'm cycling through
conda install python=3.6
conda install python=3.7
conda clean --all
and hoping to see very fast download speeds for those python packages.
But they are always painfully slow.
I checked the SquidMan settings. There is a "maximum object size", maybe that prevents the conda downloads from being cached. Are they too big ?
So I dialled those settings up to the max (well bigger than the conda download) and tried again. Same results.
How do I configure squidman to work with conda ?

It sounds like you likely have configured everything properly for proxying traffic through squid. However, conda uses https to download its packages. In a basic configuration, squid can only pass SSL connections through from the client to the server. This traffic is already encrypted, so it can not be cached. Options that you have available to you are:
Use squid's ssl bump feature to have squid decrypt and re-encrypt the data passing through it. Getting this set up is somewhat tricky because you have to generate a self-signed certificate and get it trusted by conda (using conda install --insecure might be all that is needed for conda).
Use a conda-specific proxy server. Anaconda, Inc. offers such a server as a product, so you are unlikely to find much built-in support for this in the open source conda tools. Sonatype's Nexus repository manager also claims to proxy conda repositories in its documentation.
Use conda's built-in support for local caching. Since you referenced conda clean in your question, you are aware of this cache and must have some reason for not using it. For a single machine, the conda pkgs_dir should work pretty well. For multiple machines, maybe you could get by with a network share pkgs_dir or copying all the .tar.bz2 and .conda files from a local machine into the pkgs_dir for each machine.
Add a second layer of proxying. conda allows you to specify channels with an http protocol. You could set up a proxy server that accepts http requests and passes them on as https requests. You could put your squid caching proxy in front of this http-https proxy. This way, squid will see plain http traffic that it can inspect and cache, and you can still access https only conda repositories. As an example, with nginx, you could do this with a simple conf file like:
server {
listen 80;
server_name localhost 127.0.0.1;
location / {
proxy_pass https://repo.anaconda.com/;
}
}

Related

Issues connecting Vagrant Xdebug with PhpStorm 2020.3

Have spent over a day trying to get PhpStorm to debug a Drupal site inside a Vagrant virtual machine running Xdebug and I feel I'm close - but just not quite there yet.
Currently, in PhpStorm when debugging I have an error:
Waiting for incoming connection with ide key 'PHPSTORM'
In the Xdebug log in the VM (at /tmp/xdebug.log)
[2094] I: Checking remote connect back address.
[2094] I: Checking header 'HTTP_X_FORWARDED_FOR'.
[2094] I: Checking header 'REMOTE_ADDR'.
[2094] I: Remote address found, connecting to 192.168.88.1:9000.
[2161] E: Time-out connecting to client (Waited: 200 ms). :-(
[2161] Log closed at 2020-12-15 04:25:32
I tried setting up the Zero-configuration debugging without any luck.
For the debugging configuration, I have a PHP Remote Debug that validates correctly although it always defaults to the 'Local Web Server or Shared Folder' rather than the 'Remote Web Server' option.
When I started the project I set it up as a local web server option and I'm worried that I haven't changed the correct settings to now make this a remote web server. The connection type for the deployment is now 'Local or mounted folder' but originally this was 'In-place'.
Under Languages & Frameworks --> PHP --> Servers, I set this up on port 80, using Xdebug and without path mappings. I tried changing the port etc but then it doesn't validate so I'm confident that the server settings are correct and that PhpStorm is talking to the virtual machine correctly.
I changed the /etc/php/7.4/cli/php.ini file but phpinfo() says the configuration is coming from /etc/php/7.4/fpm/php.ini. The changes I made in the php.ini file are active though. phpinfo() is showing:
xdebug.idekey: PHPSTORM
xdebug.remote_host: 10.0.2.2
xdebug.remote_port: 9000
xdebug.remote_autostart: Off
xdebug.remote_connect_back: On
Really at a loss now as to what to try next. It is incredibly frustrating so hope someone can shed some light.
EDIT --- As per comment here is the screenshots of the setup:
Deployment: Local or mounted folder
PHP settings
The server setup
Debug settings
Validate the debugger from the remote server
Run Chrome with the debug addon running
The error message
The Run debug configuration settings
Start of the phpinfo with the config file showing where the xdebug settings need to be edited.
The Xdebug settings of phpinfo()
Some of the tests as per the comments:
After logging in to 'vagrant SSH' it shows the IP address to be used (10.0.2.2). The local computer is also IP 10.1.1.150 a telnet test of both of these works.
A 'sudo nano' of the Xdebug ini file
NOTE: Changing the remote connect back to 0 fixed the connection:
xdebug.remote_connect_back=0
And then the path mapping needed to be turned on in the Server settings and then everything was working correctly.
Big thanks to LazyOne for his helpful and thorough comments. :)
1) Check if there are separate php.ini files used by web server, -- you need to edit the right php.ini file.
Run phpinfo() and check to see if there is an ini file for Xdebug. In my case this was at /etc/php/7.4/fpm/conf.d/20-xdebug.ini
2) What's your Xdebug version?
Version 2 and 3 of Xdebug have slightly different parameters. In my case I was running Xdebug 2.9.5
3) What is the IP address of your Host OS as seen from the inside the VM? That's where Xdebug should connect (as it's Xdebug that connects to IDE and NOT other way around).
When you first log in to the SSH there will be an IP address shown. This is what the IP should be for Xdebug. In my example, this was 10.0.2.2
4) xdebug.remote_connect_back: On -- try set it off and ensure that you have correct xdebug.remote_host (as Xdebug v2 may not fallback to remote_host value when autodetected IP fails.
This fixed my connection!
sudo nano /etc/php/7.4/fpm/conf.d/20-xdebug.ini
Then change:
xdebug.remote_connect_back=1
To:
xdebug.remote_connect_back=0
Save, and then restart the server.
Test the connection again. After this step I had a connection between the host and remote server but I also had to turn on path mapping in PhpStorm to get the debugger working 100%.
5) Ensure that PhpStorm is the one that listens on Xdebug port when "phone handle" icon is enabled (use netstat or alike to confirm) and is allowed in firewall.
6) If you know the correct IP and sure that PhpStorm is listening -- you can just use telnet from inside the VM and try to connect to IDE on Xdebug port -- if connected then IP, port and firewall is most likely set up correctly.
Even with the error messages, the telnet check was working. So it pointed to the issue being with the Xdebug setup rather than the handshaking between the host and remote server.
Thanks to LazyOne's comments for finding the answer and for presenting a great workflow to help identify the problem.
I was experiencing something very similar to what you described.
I was able to start a debug session from the xdebug cli tool (dbgpClient) which proved to me that it was an issue with phpStorm.
My project is using a legacy version of xdebug. (2.2.7)
Downgrading to phpStorm 2020.2.4 resolved my issue.
(It's one-click in the jetbrains toolbox to downgrade)
Thanks for the answers in this issue. It took me half a day to find out that xdebug 2.2.7 with php5.3.10 doesn't work on Phpstorm 2020.3. So I downgraded to 2020.2.4 and it works again.

Mitmproxy Client connection killed by block_global

Installed the latest version of mitmproxy, added cert, all like in instructions. Proxy not working, can't reach any site...what I'm doing wrong? Thanks for your time.
There is an option in mitmproxy called block_global: Block connections from globally reachable networks, as defined in the IANA special purpose registries. Default: True
https://docs.mitmproxy.org/stable/concepts-options/
You could try to set this option as false: mitmdump --set block_global=false

How to use Maven with HTTPS repos via proxy?

How do I configure Maven to use a proxy with HTTPS resources? It seems not to work with the settings in xxx\.m2\settings.xml? (HTTP yes, but not HTTPS)
I am behind my company's squid proxy.
This took way too long to figure out, but the solution proposed here (german) works:
In your Maven installation directory, under the bin subfolder, there is an m2.conf file. Adding your proxy settings as follows works:
set https.proxyHost default myHost
set https.proxyPort default myPort
Omit the mentioned xxx\.m2\settings.xml file.
It is implied that you would also need to add your HTTP Proxy to these settings:
set http.proxyHost default myHost
set http.proxyPort default myPort

Configuring mercurial to use a proxy gives error "abort: error: ''"

Adding a proxy section to my ~/.hgrc file doesn't seem to work, nor does
setting the proxy on the command line; e.g.
$ hg --config=http_proxy.host=http://127.0.0.1:8181 incoming
The proxy section in my ~/.hgrc file looks like:
[http_proxy]
host=http://localhost:8181
Instead, I get a not so informative response printed to stderr:
abort: error: ''
After some investigation, (adding --debug --traceback) I found this error
happens in the keepalive.py file of one of the underlying python library files.
I have Firefox/foxyproxy set up to use this same proxy, and everything works
fine - the port number [and host] is correct.
After a few minutes I realised my proxy is set up as a socks proxy, and that
mercurial doesn't work with socks proxies. The fact that the section heading is
http_proxy was the give-away hint.
I installed polipo, set it to use the socks proxy as it's parent, and
configured mercurial in the ~/.hgrc file to use this proxy instead.
Everything is working nicely now. (And I've also set firefox to use it instead
as it caches files as required too).

PHP Proxy php.ini setting

I'm trying to set get PHP running properly in a chroot jail on OSX. Because DNS lookups don't work in a chroot on OSX I would like to set all TCP connections to go through a proxy on 127.0.0.1
There is a bug report where this option was allegedly added to PHP 5
Summary: Add php.ini option to set default proxy for wrappers
However I don't seem able to find out what this option is. How do you set the default proxy for all stream wrappers?

Resources