I have some Raspberry Pi servers behind NATs (non configurable, ISP provided), on dynamic IPs, and a "master" server with static IP and port forwarding configured on the router. I want to be able to access the page served by any of those RPi servers from any browser. I've read about TCP hole punching, but I can't figure out how to make it work from a browser (I guess using AJAX). I could use the "master" server as a relay server, but don't know how to. BTW, all traffic to/from the servers use HTTPS, not HTTP.
EDIT:
The Raspberries and the server are NOT on the same network.
You might investigate the use of a reverse proxy (I've used NGINX). A reverse proxy allows traffic to hit your server with the static IP, and forward HTTP traffic to other servers behind the firewall.
It gets a little more tricky with HTTPS, but it can be worked out.
Related
I made a proxy server in python 3. It listens on the port 4444. It basically receives the request from clients and sends it to the server. I want to use it as a firewall to my Dvwa server. So added another functionality to the proxy. What it does is, before sending the request to the DVWA server, it validates the input.
But the problem is, the clients have to configure their proxy settings in the browser to use my proxy server. Is there any way to access the proxy without configuring the browser settings. Basically I want to host the proxy server instead of the original web server. So that all the traffic goes through the proxy before going to the webserver.
Thanks in advance...
You don't say whether your Python3 proxy is hosted on the same machine as the DVWA.
Assuming it is, the solution is simple: a reverse-proxy configuration. Your proxy transparently accepts and forwards requests to your server who then processes them and sends them back via the proxy to the client.
Have your proxy listen on port 80
Have the DVWA listen on a port other than 80 so it's not clashing (e.g. 8080)
Your proxy, which is now receiving requests for the IP/hostname which would otherwise go to the DVWA, then forwards them as usual.
The client/web browser is none the wiser that anything has changed. No settings need changing.
That's the best case scenario, given the information provided in your question. Unfortunately, I can't give any alternative solutions without knowing the network layout, where the machines reside, and the intent of the project. Some things to consider:
do you have a proper separation of concerns for this middleware you're building?
what is the purpose of the proxy?
is it for debugging/observing traffic?
are you actually trying to build a Web Application Firewall?
HTTP proxy with SSL and DNS support.
I must be lacking some key concepts about proxy-ing because I cannot grasp this. I am looking to run a simply http or https proxy without interfering with SSL. Simply, a fully transparent proxy that can passthrough all the traffic to the browser connected via HTTP or HTTPS proxy without modifying or intercepting any packets. Not able to find any code online or I'm not using the right keywords.
EX. On the browser adding server.someVPN.com:80 on the HTTP proxy field and as soon as you try to visit a website, it prompts for authentication. Then it works perfectly with any domain, any security, any ssl, no further steps needed. Most VPN providers have this.
How's this possible? it even resolves DNS itself. I thought on transparent proxy the dns relies on the client. Preferably looking for a nodeJS solution but any lang works.
Please don't propose any solutions such as SOCKS5 or sock forwarding or DNS overriding or CA based MITM. According to HTTP 1.1 which supports 'CONNECT' this should be easy.
Not looking to proxy specific domains, looking for an all inclusive solution just like most VPN Providers providers.
----Found the answer too quickly, feel free to delete this post/question admins.
The way it works is that the browser knows it is talking to a proxy server, so for example if the browser want to connect to htttp://www.example.com it sends a CONNECT www.example.com:443 HTTP/1.1 to the proxy server, the proxy server resolves wwww.example.com via DNS and then opens a TCP connection to wwww.example.com port 443 and proxies the TCP stream transparently to the client.
I don't know any solution for nodejs. Common proxy servers include Squid, Privoxy and Apache Traffic Server
See also: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT
Found the solution right after I asked...
This module works perfectly https://github.com/mpangrazzi/harrier
Does exactly what I was asking for.
I'm running a kubernetes application on GKE, which serves HTTP requests on port 80 and websocket on port 8080.
Now, HTTP part needs to know client's IP address, so I have to use HTTP load balancer as ingress service. Websocket part then has to use TCP load balancer, as it's clearly stated in docs that HTTP LB doesn't support it.
I got them both working, but on different IPs, and I need to have them on one.
I would expect that there is something like iptables on GCE, so I could forward traffic from port 80 to HTTP LB, and from 8080 to TCP LB, but I can't find anything like that. Anything including forwarding allows only one them.
I guess I could have one instance with nginx/HAproxy doing only this, but that seems like an overkill
Appreciate any help!
There's not a great answer to this right now. Ingress objects are really HTTP only right now, and we don't really support multiple grades of ingress in a single cluster (though we want to).
GCE's HTTP LB doesn't do websockets yet.
Services have a flaw in that they lose the client IP (we are working on that). Even once we solve this, you won't be able to use GCE's L7 balancer because of the extra port you need.
The best workaround I can think of, and has been used by a number of users until we preserve source IP, is this:
Run your own haproxy or nginx or even your own app as a Daemonset on some or all nodes (label controlled) with HostPorts.
Run a GCE Network LB (outside of Kubernetes) pointing at the nodes with HostPorts.
Once we can properly preserve external IPs, you can turn this back into a plain Service.
We need to have a persistent connection between servers on WAN and LAN. Basically multiple LAN servers have to register with a server on Internet and always keep an open connection, because we can not ask clients who own serves on LAN to set up port forwarding, and because there might be firewall issues.
What are my options here? I am thinking, either Websockets, or servers on LAN long polling our server on the Internet. Is there a better solution?
Your options are - considering you want to avoid problems with proxies and firewalls - probably:
Use websockets, either port 80 or 443
Use long polling (that's not really persistent, if you need bidirectional communication!) port 80 or 443
Encapsulate whatever data into a http request, using port 80
Use port 443 with ssl encryption, and do whatever you want on the protocol level. Normally, you won't have any problems using even some strange custom protocol as long as you use this port with ssl encryption, as neither firewalls nor proxies normally (try to) inspect what you're doing when they sense what they think is https.
Everything else is probably out of scope for you as far as I get the scenario, but you should be able to do what you need using port 443 with some protocol made for persistent connections (it's possible with websockets, but you'll run into other problems really fast (like missing ping/pong implementations) then if you don't custom build your code, which would probably make it pointless to use websockets in the first place)
I have Charles Proxy set up to look at outgoing https requests, and I need to re-route traffic from one server to a local http server.
I have a MacOSX machine that is set up this way: I have an ethernet connection that I hardwire, and share the internet connection via the airport interface. On a second machine, I've installed the Charles cert, and when I connect via the shared interface. I can see the traffic (unencrypted) in Charles, so I know the communications and certs are all working properly.
I need to intercept all the https traffic going to one server (https://www.foo.com) to a local http server (localhost:8001). I've tried using Map Remote, but it doesn't seem to unencrypt the traffic before forwarding it (or possibly it re-encrypts it).
How can I configure Charles to do this? (or, please point me to any other software package, if Charles isn't capable of this)
I figured out what was happening, there were two issues.
I had misconfigured the Map Remote entry, and my two different clients (MyApp and curl) were hitting two different servers - the app was hitting the correct server (locally) but the request was malformed.
Curl from the macOSX box where the proxy was running was NOT looping through the proxy, since I hadn't included the -x localhost:8888 flag.