How do you use an HTTP/HTTPS proxy with boto3? - proxy

Question

On the old boto library is was simple enough to use the proxy, proxy_port, proxy_user and proxy_pass parameters when you open a connection. However, I could not find any equivalent way of programmatically define the proxy parameters on boto3. :(

Answer 1

As of at least version 1.5.79, botocore accepts a proxies argument in the botocore config.
e.g.
import boto3
from botocore.config import Config
boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))
boto3 resource
https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource
botocore config
https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config

Answer 2

If you user proxy server does not have a password
try the following:
import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"
if you user proxy server has a password
try the following:
import os
os.environ["HTTP_PROXY"] = "http://user:password#proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://user:password#proxy.com:port"

Answer 3

Apart from altering the environment variable, I'll present what I found in the code.
Since boto3 uses botocore, I had a look through the source code:
https://github.com/boto/botocore/blob/66008c874ebfa9ee7530d944d274480347ac3432/botocore/endpoint.py#L265
From this link, we end up at:
def _get_proxies(self, url):
# We could also support getting proxies from a config file,
# but for now proxy support is taken from the environment.
return get_environ_proxies(url)
...which is called by proxies = self._get_proxies(final_endpoint_url) in the EndpointCreator class.
Long story short, if you're using python2 it will use the getproxies method from urllib2 and if you're using python3, it will use urllib3.
get_environ_proxies is expecting a dict containing {'http:' 'url'} (and I'm guessing https too).
You could always patch the code, but that is poor practice.

Answer 4

This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:
import botocore.endpoint
def _get_proxies(self, url):
return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3