I made a simple proxy server using nc, here's the one-liner:
mkfifo queueueue
nc -l 8080 <queueueue | nc http://$JENKINS_HOSTNAME 80 >queueueue
It listens on port 8080 and then forwards the data to a connection to our Jenkins server. Jenkins is behind a VPN, and the machine I am running this proxy on has VPN access.
On my other machine (no VPN access), I would like to curl the Jenkins server, here's the command to initiate the request through the proxy:
http_proxy=10.1.10.10:8080 curl --user $JENKINS_USERNAME:$JENKINS_PASSWORD http://$JENKINS_HOSTNAME/api/json
Both the client and the proxy machine are on the same network, I can ping and ssh between them, also, I know that the client is connecting to the proxy server, I think the failure is arising when the client is trying to authenticate, here's the output when I try to curl:
$ http_proxy=10.1.10.10:8080 curl --user $JENKINS_USERNAME:$JENKINS_PASSWORD http://$JENKINS_HOSTNAME/api/json
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved here.</p>
<hr>
<address>Apache Server at $JENKINS_HOSTNAME Port 80</address>
</body></html>
How can I curl through a proxy like this with HTTP Basic Authentication?
I would use ssh for this instead of netcat.
Just to get some confusion out of the way, I will be referring to the node with VPN access as the "server", and the node without VPN access as the "client".
On the server side you should only need to install and have an ssh server running (in my test I have OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011).
On the client side you will need to do the following:
1) in your /etc/hosts file add in the address that your target URL resolves as on the server. I wasn't able to get curl to run DNS lookups through the proxy, which is why this is necessary.
2) setup ssh keys between the server and the client. while this is not necessary, it makes life easier.
3) run the following ssh command to have ssh act as a SOCKS proxy:
user#host$ ssh -vND 9999 <server>
-v is there so you can see what is going on with ssh,
-N tells ssh to not execute a remote command - this is useful for just simple port forwarding
-D this option is what actually forwards your local requests to the server
4) now you should be able to run the curl command you have above, but add in
---socks5 localhost:9999
Your full command will look like this:
curl --user $USER:$PASSWORD --socks5 localhost:9999 http://$JENKINS/api/json
If I can figure out how to forward the DNS requests from curl through ssh I'll update the ticket.
edit: formatting, awful grammar.
Related
I'm new to Amazon Web Service (AWS).
I already created a PostgreSQL from AWS RDS:
Endpoint: database-1.XXX.rds.amazonaws.com
Port: 5432
Public accessibility: Yes
Availablity zone: ap-northeast-1c
After that, I will push my application that using the database to AWS (maybe deploy to EKS).
However, I want to try testing the database server from my local computer first.
I haven't tried testing from my laptop PC at home yet, but I think it will connect OK because my laptop PC is not using the HTTP proxy to connect to the network.
The problem is that I want to try testing from my company PC, which needs setup the HTTP proxy to connect to the internet. The PC spec:
Windows 10
Installed PostgreSQL 10
Firstly, I tried using psql command-line:
psql -h database-1.XXXX.rds.amazonaws.com -U postgre
> Unknown host
set http_proxy=http://user:password#my_company_proxy:3128
set https_proxy=http://user:password#my_company_proxy:3128
psql -h database-1.XXXX.rds.amazonaws.com -U postgre
> Unknown host
set http_proxy=http://my_second_company_proxy:3128
set https_proxy=http://my_second_company_proxy:3128
psql -h database-1.XXXX.rds.amazonaws.com -U postgre
> Unknown host
Then, I tried using the pgAdmin tool.
As from the internet post, it said that we can use "SSH Tunnel" for inputing proxy:
However, the error message will be shown:
So, anyone can help suggest if we can connect to the public PostgreSQL server through HTTP proxy?
I think problem is Postgres uses plain TCP/IP protocol and you are trying to use HTTP proxy. Also you're trying to create SSH tunnel against your HTTP proxy server which won't work.
So I'd suggest following solutions:
Use TCP proxy instead of HTTP proxy
Create an EC2 or any instance that has SSH access from your company network and has access to public internet. So that you can create SSH tunnel through that instance to achieve your goal.
NOTE: Make sure you PostgreSQL is accessible from public internet (although this is usually bad idea, but it's out of scope this question) sometimes security group configs prevent it to connect from public internet.
Just add all ports(5432,3128...) in the Security Group from your RDS and specify your IP. Don't forget "/32"
Let me add that "unknown host" is usually an indication that you're not resolving the DNS hostname. Also, your HTTP proxy should not interfere with connections to databases since they aren't on port 80 or 443. A couple of things you can try (assuming you're on windows) sub in your actual url:
nslookup database-1.XXXX.rds.amazonaws.com
telnet database-1.XXXX.rds.amazonaws.com 5432
You should also check the security group that is attached to your RDS and make sure you've opened up the ip address that you're originating from on port TCP/5432.
Lastly check that your VPC has DNS and Hostnames enabled. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating
We have a Jenkins server which is accessible only from within the VPC on the cloud. On Mac and Linux I use sshuttle to make a ssh connection to the bastion instance (to act a proxy) and open the Jenkins console in the browser. Everything works fine.
Now I'm on Windows and trying to do the same on WSL2. If I'm not mistaken previously, sshuttle didn't work on WSL1 (failed with some error message), but I managed to run it on WSL2 without any issue. The ssh connection is established and I can access my Jenkins (using curl).
Then I tried to access my Jenkins on Windows via WSL2:
1. I found the IP address of WSL2 and the port the ssh tunnle:
# lsof -i -n | grep ssh
sshuttle 1234 rad 5u IPv4 39270 0t0 TCP *:socks (LISTEN)
ssh 5678 rad 3u IPv4 40252 0t0 TCP 172.25.236.84:57578->bastion:ssh (ESTABLISHED)
2. I configured network proxy setting of Firefox (v77) to use my ssh tunnle:
Manual proxy configuration
SOCK host: 172.25.236.84
Port: 1080
SOCKS V5 (tested with V4 as well)
But loading the page fails with "The connection was reset" error on Firefox. I tested via Powershell that the SOCKS port is open and responding (using Test-NetConnection).
1. Any idea what the problem is? How to make it work?
2. If it's not gonna work, is there any other solution (e.g. Docker, etc)?
Thanks.
I'm not sure, but my guess is that sshuttle doesn't actually act as a SOCKS proxy and that's why the connection gets reset.
I managed to access my Jenkins on Windows machine using ssh SOCKS proxy: ssh -D 0.0.0.0:1080 rad#bastion and configured Firefox to use the SOCKS proxy.
Interestingly, for this you don't even need WSL. It seems Windows 10 has OpenSSH and you can use it. Just open CMD and type ssh -D 1080 rad#bastion and setup Firefox to use localhost as the proxy.
If there's any better solution or any comment/concern (apart from DNS over SOCKS) with this approach, please share.
Thanks.
As alternative on WSL(2) you can run a regular SSH tunnel.
Eg:
ssh -N -L 127.0.0.1:5432:some_domain_to_forward:5432 user#jumpbox_ip
and then just connect to 127.0.0.1:5432
I have configured SOCKS5 proxy server in AWS with dante and it runs fine.
when I try the following command in CMD
curl -L -x socks5://user:password#23.29.xx.xx:1313 http://www.google.com/
it works,
But when I configure my LAN proxy configs, it can't access to the internet via any browser.
for CURL proxy server works but for browsers it doesn't.
please help.
The reason for this is, I have enabled the user authentication in the server. Unlikely in HTTP proxy servers, there is no signin popup in socks5. This was fixed after I added the username and password.
I’m using bash shell on Mac El Capitan. How do I pass a blank username/password for a proxy server using curl? I tried this
localhost:tmp davea$ curl http://www.google.com --proxy localhost:9050 --proxy-user "":""
514 Authentication required.
I’m running a tor daemon on my machine using this command
tor --CookieAuthentication 0 --HashedControlPassword "" --ControlPort 9050 --SocksPort 50001
and I’m able to connect through Telnet without entering a password like so
localhost:tmp davea$ telnet localhost 9050
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
AUTHENTICATE
250 OK
so I know that my password, at least, is correct.
Note that when using curl from the command line, the --proxy option is specifically for use with an HTTP proxy which Tor is not, it's a SOCKS proxy.
To get around this, use what Nehal suggested in the comments to use a SOCKS proxy (you'll probably want to use --socks5-hostname instead so the DNS resolution is also performed over Tor as well, otherwise you leak DNS requests locally).
So your call would look like:
curl http://www.google.com -L --socks5-hostname localhost:50001
Side note: The control port (9050) is only used for communicating commands to the controller. This port is not used to proxy requests at all.
This is a very basic Amazon EC2 question, but I'm stumped so here goes.
I want to launch an Amazon EC2 instance and allow access to HTTP on ports 80 and 8888
from anywhere. So far I can't even allow the instance to connect to on those ports using
its own IP address (but it will connect to localhost).
I configured the "default" security group for HTTP using the standard HTTP option on the management console (and also SSH).
I launched my instance in the default security group.
I connected to the instance on SSH port 22 twice and in one window launch an HTTP server
on port 80. In the other window I verify that I can connect to HTTP using the "localhost".
However when I try to access HTTP from the instance (or anywhere else) using either the public DNS or the Private IP address I het "connection refused".
What am I doing wrong, please?
Below is a console fragment showing the wget that succeeds and the two that fail run from the instance itself.
--2012-03-07 15:43:31-- http://localhost/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /__whiff_directory_listing__ [following]
--2012-03-07 15:43:31-- http://localhost/__whiff_directory_listing__
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “__whiff_directory_listing__”
[ <=>
] 7,512 --.-K/s in 0.03s
2012-03-07 15:43:31 (263 KB/s) - “__whiff_directory_listing__” saved [7512]
[ec2-user#ip-10-195-205-30 tmp]$ wget http://ec2-50-17-2-174.compute-1.amazonaws.com/
--2012-03-07 15:44:17-- http://ec2-50-17-2-174.compute-1.amazonaws.com/
Resolving ec2-50-17-2-174.compute-1.amazonaws.com... 10.195.205.30
Connecting to ec2-50-17-2-174.compute-1.amazonaws.com|10.195.205.30|:80... failed:
Connection refused.
[ec2-user#ip-10-195-205-30 tmp]$ wget http://10.195.205.30/
--2012-03-07 15:46:08-- http://10.195.205.30/
Connecting to 10.195.205.30:80... failed: Connection refused.
[ec2-user#ip-10-195-205-30 tmp]$
The standard tcp sockets interface requires that you bind to a particular IP address when you send or listen. There are a couple of somewhat special addresses: localhost (which you're probably familiar with) which is 127.0.0.1. There's also a special address, 0.0.0.0 or INADDR_ANY (internet protocol, special shorthand for ANY ADDRESS). It's a way to listen on ANY or more commonly, ALL addresses on the host. This is a way to tell the kernel/stack that you're not interested in a particular IP address.
So, when you're setting up a server that listens to "localhost" you're telling the service that you want to use the special reserved address that can only be reached by users of this host, and while it exists on every host, making a connection to localhost will only ever reach the host you're making the request from.
When you want a service to be reachable everywhere (on a local host, on all interfaces, etc.) you can specify 0.0.0.0.
(0) It's silly but the first thing you need to do is to make sure that your web server is running.
(1) You need to edit your Security Group to let incoming HTTP packets access your website. If your website is listening on port 80, you need to edit the Security Group to open access to port 80 as mentioned above. If your website is listening on some other port, then you need to edit the Security Group to access that other port.
(2) If you are running a Linux instance, the iptables firewall may be running by default. You can check that this firewall is active by running
sudo service iptables status
on the command line. If you get output, then the iptables firewall is running. If you get a message "Firewall not running", that's pretty self-explanatory. In general, the iptables firewall is running by default.
You have two options: knock out the firewall or edit the firewall's configuration to let HTTP traffic through. I opted to knock out the firewall as the simpler option (for me).
sudo service iptables stop
There is no real security risk in shutting down iptables because iptables, if active, merely duplicates the functionality of Amazon's firewall, which is using the Security Group to generate its configuration file. We are assuming here that Amazon AWS doesn't misconfigure its firewalls - a very safe assumption.
(3) Now, you can access the URL from your browser.
(4) The Microsoft Windows Servers also run their personal firewalls by default and you'll need to fix the Windows Server's personal firewall, too.
Correction: by AWS default, AWS does not fire up server firewalls such iptables (Centos) or UAF (Ubuntu) when you are ordering the creation of new EC2 instances - That's why EC2 instances that are in the same VPC can ssh into each other and you can "see" the web server that you fired up from another EC2 instance in the same VPC.
Just make sure that your RESTful API is listening on all interfaces i.e. 0.0.0.0:portID
As you are getting connection refused (packets are being rejected) I bet it is iptables causing the problem. Try to run
iptables -I INPUT -p tcp --dport 80 -j ACCEPT
iptables -I INPUT -p tcp --dport 8888 -j ACCEPT
and test the connection.
You will also need to add those rules permanently which you can do by adding the above lines into ie. /etc/sysconfig/iptables if you are running Red Hat.
Apparently I was "binding to localhost" whereas I needed to bind to 0.0.0.0 to respond to port 80 for the all incoming TCP interfaces (?). This is a subtlety of TCP/IP that I don't fully understand yet, but it fixed the problem.
Had to do the following:
1) Enable HTTP access on the instance config, it wasn't on by default only SSH
2) Tried to do nodejs server, so port was bound to 80 -> 3000 did the following commands to fix that
iptables -F
iptables -I INPUT -p tcp --dport 80 -j ACCEPT
sudo service iptables-persistent flush
Amazon support answered it and it worked instantly:
I replicated the issue on my end on a test Ubuntu instance and was able to solve it. The issue was that in order to run Tomcat on a port below 1024 in Ubuntu/Unix, the service needs root privileges which is generally not recommended as running a process on port 80 with root privileges is an unnecessary security risk.
What we recommend is to use a port redirection via iptables :-
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
I hope the above information helps.