Transparent proxying, not affecting other traffic - proxy

I want configure the system as following:
There are several web app servers, each of which have global IP addresses.
The web app servers make HTTP and HTTPS requests to several (unidentified) external services. The source IP address of packets (thus destination port 80/tcp or 443/tcp) need to be fixed to a global IP address, so actually I need transparent proxy for HTTP/HTTPS, not affecting other traffic.
Other packets should not use the global IP address above, but should use global IP addresses that the server is each assigned.
The web application is very old and it's impossible to modify it to use proxy (CONNECT) protocol.
So I tried to use iptables to DNAT from 80/tcp and 443/tcp to the squid proxy on all web servers, and configured squid as an interception proxy.
However I failed it by redirection loop error.
I investigated what squid is doing by strace and found that it tries to connect to 10.0.0.252:80 after receiving a request, and therefore forwarding loop is detected.
I believe it is because of misconfiguration but I have no idea which should be fixed, or maybe totally I misunderstand what I should do.
(I googled but couldn't find the examples that NAT on each server.)
Hope could someone help solve the problem, or suggest another better way (not limited to using squid proxy)
All servers are on Amazon EC2, so it is a choice to use a vyos for a router...
ip a result on squid proxy
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
link/ether 0a:1c:ba:c3:9c:1d brd ff:ff:ff:ff:ff:ff
inet 10.0.0.211/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::81c:baff:fec3:9c1d/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
link/ether 0a:a9:2c:5e:eb:d7 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.252/24 brd 10.0.0.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::8a9:2cff:fe5e:ebd7/64 scope link
valid_lft forever preferred_lft forever
iptables on squid proxy
iptables -t nat -A PREROUTING -s 10.0.0.252 -p tcp --dport 80 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.0.0.252:3129
iptables on web servers
iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination 10.0.0.252:80
cache.log
2016/06/22 06:15:22 kid1| WARNING: Forwarding loop detected for:
GET / HTTP/1.1
User-Agent: squidclient/3.5.19
Accept: */*
Via: 1.0 unknown (squid/3.5.19)
X-Forwarded-For: 10.0.0.211
Cache-Control: max-age=259200
Connection: keep-alive
Host: ifconfig.moe
Full squid.conf
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl Safe_ports port 1025-65535 # unregistered ports
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
#http_access deny to_localhost
http_access allow localnet
http_access allow localhost
cache_dir ufs /var/spool/squid 100 16 256
coredump_dir /var/spool/squid
visible_hostname unknown
# Squid normally listens to port 3128
http_port 3128
http_port 3129 intercept
http_port 3130 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myCA.pem
always_direct allow all
ssl_bump none localhost
ssl_bump server-first all
# temporary: just test
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER
http_access deny all

If you DNAT to the proxy, you're changing the destination to the IP of the proxy. In this case, the proxy will lose the information about the original destination.
For http this is ok, since the host header can be used to resolve the target, but for https the proxy would need to rely on SNI in the TLS ClientHello packet to learn the target and connect to it, bootstrap the TLS layer and go from there.

Related

Kolla Ansilble: Openstack Instances Unable to access Internet or eachother

I am a newbie to openstack (deployed using kolla-ansible) and have created two instances both are ubuntu 20.04 VMs. I am able to ping and ssh them from the host machine (192.168.211.133) and vice versa. However instances are unable to access internet. The virtual router is also unable to access internet:
Configuration of one of the machine is below;
root#kypo-virtual-machine:/etc/apt/sources.list.d# ip netns ls
qrouter-caca1d42-86b4-42a2-b591-ec7a90437029 (id: 1)
qdhcp-0ec41857-9420-4322-9fef-e332c034e98e (id: 0)
root#kypo-virtual-machine:/etc/apt/sources.list.d# ip netns e qrouter-caca1d42-86b4-42a2-b591-ec7a90437029 route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.211.1 0.0.0.0 UG 0 0 0 qg-f31a26b7-25
192.168.64.0 0.0.0.0 255.255.192.0 U 0 0 0 qr-e5c8842c-c2
192.168.211.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-f31a26b7-25
Netplan of instance shows:
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
version: 2
ethernets:
ens3:
dhcp4: true
match:
macaddress: fa:16:3e:a7:9d:70
mtu: 1450
set-name: ens3
And IP sheme is:
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc
fq_codel state UP group default qlen 1000
link/ether fa:16:3e:a7:9d:70 brd ff:ff:ff:ff:ff:ff
inet 192.168.65.39/18 brd 192.168.127.255 scope global dynamic ens3
valid_lft 85719sec preferred_lft 85719sec
inet6 fe80::f816:3eff:fea7:9d70/64 scope link
valid_lft forever preferred_lft forever
From Horizon
IP Addresses
kypo-base-net
192.168.65.39, 192.168.211.250
Security Groups
kypo-base-proxy-sg
ALLOW IPv6 to ::/0
ALLOW IPv4 icmp from 0.0.0.0/0
ALLOW IPv4 22/tcp from 0.0.0.0/0
ALLOW IPv4 udp from b9904736-6d8a
ALLOW IPv4 tcp from b9904736-6d8a
ALLOW IPv4 tcp from 73ca626b-7cfb
ALLOW IPv4 udp from 73ca626b-7cfb
ALLOW IPv4 to 0.0.0.0/0
I was able to resolve the issue by pinpointing that the gateway used by the virtual router (192.168.211.1) was different form the one used by my host VM (192.168.211.2).
kypo#kypo-virtual-machine:/etc/kolla$ ip route show
default via 192.168.211.2 dev ens33 proto dhcp
src 192.168.211.133 metric 100
I modify the gateway;
openstack subnet set --gateway 192.168.211.2 public-subnet
And now my instances are able to access internet.
The main reason for this configuration issue was while creating the subnet I used auto for --gateway option and obviously it didn't pick the correct gateway.

How can I make gradle deal with multiple proxies?

I work on a project where there are multiple nexus registries behind different proxies :
How can I make sure that Gradle (or any repository related tool, such as NPM, maven, etc) can handle 3+ differents proxies at the same time to reach multiple Nexus instances ?
Until now, we were using a workaround : 1 nexus was accessed through HTTP proxy and 1 through HTTPS proxy. But now, we have 3 proxies to handle !
I think that it must be possible to add a machine (a squid instance ?) which would redirect proxy requests to the correct proxy, based on the domain name :
I'm not used to Squid and I still not managed to achieve this. Can anyone confirm if this is possible (or not) using Squid ? Does anyone would have another solution to suggest ?
Just for the background story, this network setting is due to multiple partner companies being involved in the project. We have access to each company Nexus through dedicated VPN and proxies.
OK, so I managed to run a Squid in a docker with the following config :
acl host src 10.0.0.0/8
acl host src 172.0.0.0/8
http_access allow host
maximum_object_size 256 MB
maximum_object_size_in_memory 256 MB
dns_nameservers 8.8.8.8 8.8.4.4
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access deny all
http_port 3128
cache_peer 10.1.1.1 parent 8080 0 proxy-only no-query name=proxy1
cache_peer 10.2.1.1 parent 8080 0 proxy-only no-query name=proxy2
cache_peer 10.3.1.1 parent 8080 0 proxy-only no-query name=proxy3
acl sites_proxy1 dstdomain .domain1.com
acl sites_proxy2 dstdomain .domain2.com
acl sites_proxy3 dstdomain .domain3.com
cache_peer_access proxy1 deny !sites_proxy1
cache_peer_access proxy2 deny !sites_proxy2
cache_peer_access proxy3 deny !sites_proxy3
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880
refresh_pattern . 0 20% 4320
never_direct allow all
Then, I run the docker using this command :
docker run --rm --volume ~/DevTools/squid/squid.conf:/etc/squid/squid.conf -v ~/DevTools/squid/logs:/var/log/squid -p 3128:3128 datadog/squid

Running Squid on localhost

I have a product form Symantec and their help is...less than helpful, including a nice message that says "Contact your reseller" in the "Contact Us" link. My reseller says to contact them. How? Anyways, it's a repackaged version if Squid for Windows. When I point IE to the proxy running locally I get "Access Denied. Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect." However, when I point IE on another machine to the server running Squid everything works fine.
I have zero experience with Squid or proxies. I tried some different configs based on searches here but nothing worked. I'm sure it's something simple. Here is the config:
digest_generation off
hierarchy_stoplist cgi-bin ?
acl all src 0.0.0.0/0.0.0.0
cache deny all
maximum_object_size 0 KB
emulate_httpd_log on
debug_options ALL,1
cache_store_log none
access_log none
useragent_log none
auth_param ntlm program c:/clientsiteproxy/libexec/mswin_ntlm_auth.exe
auth_param ntlm children 80
auth_param ntlm keep_alive on
auth_param negotiate children 80
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off
authenticate_ip_shortcircuit_ttl 30 seconds
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
read_timeout 15 minutes
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Smartconnect dstdomain ned.webscanningservice.com
acl CONNECT method CONNECT
acl authproxy proxy_auth REQUIRED
acl our_networks src 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8 169.254.0.0/16
acl HEAD method HEAD
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow HEAD
http_access deny !our_networks
http_access allow Smartconnect
http_access allow authproxy
http_access deny all
icp_access allow all
httpd_suppress_version_string on
visible_hostname ClientSiteProxy
forwarded_for off
header_access Via deny all
never_direct allow all
cache_dir null c:/ClientSiteProxy
coredump_dir c:/clientsiteproxy/var/cache
http_port 3128
This is most likely the culprit: http_access deny !our_networks. This statement denies outbound access for all source IPs apart from 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8 169.254.0.0/16.
When you browse from the same machine, browser would bind on localhost, so you can try expanding the our_networks definition with 127.0.0.1.

Squid Connection Refused HTTPS

I have managed to get my HTTP "proxy" connection to work but on most https connections I get the error:
Connection to failed.
The system returned: (111) Connection refused
Here is my config file I am currently using in squid:
http_port 3128
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
acl localnet src 10.0.0.0/8 # RFC 1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC 1918 possible internal network
acl myhost src <myip>
http_access allow myhost
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
via off
forwarded_for off
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow all
dns_nameservers 8.8.8.8 8.8.4.4
Running on a Ubuntu 12.04 VPS - connecting remotely via the browser proxy settings page...
All I want to do is to be able to connect to my server and browse http and https sites. Http works with the config above... https does not...
Remove the following line:
http_access deny CONNECT !SSL_ports
i would recommend keeping
http_access deny CONNECT !SSL_ports
but allowing those listed by adding:
http_access allow SSL_ports

pf NAT for local originated traffic

I am trying to proxy 22 traffic, originated from my machine, so it will be proxied / forwarded to a remote server. This was very easy with ipfw but Yosemite has only pf ....
Did anyone found out how to rdr on local (originated from the machine) traffic on OSX Yosemite ?
this is not working !
# Custom pf Rules
#######################################
# target_ip is the ip of which I would like traffic to be proxied
target_ip = "10.0.0.2”
# porxy_ip is the proxy address
porxy_ip = "192.168.1.100”
rdr pass log on lo0 proto tcp from en0 to $target_ip port 80 -> $porxy_ip port 8080
pass out on en0 route-to lo0 inet proto tcp from en0 to $target_ip port 80 keep state
# End
any advise ?

Resources