set up conda for caching proxy (MacOs + Squidman) - proxy

Question

How can I configure conda to use a caching proxy?
So far I have:
installed SquidMan and set its host and port to 127.0.0.1:8080.
set the network settings to proxy both http and https at this address
edited .condarc to use a proxy
I think that SquidMan is set up correctly. If I switch it off and try to browse the internet, I get an error message "The proxy server is refusing connections". This happens for both http and https websites and also if I enter an IP directly (no DNS in between).
The edited .condarc is this:
proxy_servers:
http: http://127.0.0.1:8080
https: http://127.0.0.1:8080
Those are the same addresses as in the system proxy settings - which seem to work fine for browsing.
As a test I'm cycling through
conda install python=3.6
conda install python=3.7
conda clean --all
and hoping to see very fast download speeds for those python packages.
But they are always painfully slow.
I checked the SquidMan settings. There is a "maximum object size", maybe that prevents the conda downloads from being cached. Are they too big ?
So I dialled those settings up to the max (well bigger than the conda download) and tried again. Same results.
How do I configure squidman to work with conda ?

Answer 1

It sounds like you likely have configured everything properly for proxying traffic through squid. However, conda uses https to download its packages. In a basic configuration, squid can only pass SSL connections through from the client to the server. This traffic is already encrypted, so it can not be cached. Options that you have available to you are:
Use squid's ssl bump feature to have squid decrypt and re-encrypt the data passing through it. Getting this set up is somewhat tricky because you have to generate a self-signed certificate and get it trusted by conda (using conda install --insecure might be all that is needed for conda).
Use a conda-specific proxy server. Anaconda, Inc. offers such a server as a product, so you are unlikely to find much built-in support for this in the open source conda tools. Sonatype's Nexus repository manager also claims to proxy conda repositories in its documentation.
Use conda's built-in support for local caching. Since you referenced conda clean in your question, you are aware of this cache and must have some reason for not using it. For a single machine, the conda pkgs_dir should work pretty well. For multiple machines, maybe you could get by with a network share pkgs_dir or copying all the .tar.bz2 and .conda files from a local machine into the pkgs_dir for each machine.
Add a second layer of proxying. conda allows you to specify channels with an http protocol. You could set up a proxy server that accepts http requests and passes them on as https requests. You could put your squid caching proxy in front of this http-https proxy. This way, squid will see plain http traffic that it can inspect and cache, and you can still access https only conda repositories. As an example, with nginx, you could do this with a simple conf file like:
server {
listen 80;
server_name localhost 127.0.0.1;
location / {
proxy_pass https://repo.anaconda.com/;
}
}