I have been googling all night and reading the Squid wiki but for the life of me
I can not figure out if squid actually can cache content transmitted via HTTPs specially images, if so HOW? and where?! The manual and Wiki seems to be written for people who
are Squid Jedi masters or 50 years old netowork admins who were there when internet was invented or may be I am stupid... but can someone please help me understand and set up this feature?
Related
Heyo everyone !
I'm trying to reverse ingeneer an API using Burp. The problem is that I can't see any of the requests made by the software using the said API while I can see the requests made from Spotify, Discord, etc... I set the proxy parametter of windows to localhost:8080 and burp to listen to *:8080
Do you have any idea on what could make that happen ? I've been exploring the web I have yet to see an answer to my problem...
If I can improve by any means this question, to hesitate to let me know !
It is called SSL Pining and it's a method to protect against Man-in-middle type of attacks. It's very effective !
You can learn more about it through these articles:
https://sslmagic.com/blog/introduction-to-ssl-pinning/
https://medium.com/#anuj.rai2489/ssl-pinning-254fa8ca2109
https://www.indusface.com/learning/what-is-ssl-pinning-a-quick-walk-through/
I'm not related to any of these articles by any means. I just find them useful to understand SSL Pining
Some time ago, I needed HTTPS support for my express webserver. I found a tutorial that teached me a cool trick to achieve this. They basically explained me that an AWS load balancer can redirect HTTPS to HTTP.
So, I first created a load balancer.
And then redirected HTTPS to HTTP. The traditional HTTP, I just redirected 80 to 80. And I have a websocket (socket io) thing going on port 1337 (which I plan to change to port 1338 in the near future).
Just for clarity. I didn't really need a load balancer, since I actually only have 1 AWS instance. But using this setup, I did not have to go through the trouble of messing around with HTTPS certificate files, neither did I have to upgrade my webserver. It saved me a lot of trouble at first.
Then this morning, I received the bill, and discovered that this load balancing trick has a price tag of roughly 22usd/mo. (an expensive port forwarding trick)
I probably have to get rid of this load balancer. But I am wondering, perhaps I did something wrong in the configuration.
It's strange that charges are so high for a web app that is still in development. So, I am wondering if perhaps there is something wrong with my setup. And that leads me to the following question.
I noticed that I am actually using an old ELB setup: "Classic load balancer". And it actually states that this setup does not support websockets, which is a bit strange.
My web app hosts some static webpages (angular), but once it is downloaded, all traffic uses socket.io websockets. Even though the AWS documentation says that websockets are not supported, it seems to work fine. Unless ...
Now, socket io is a pretty smart thing. When it can't use modern websockets (e.g. because the webbrowser does not support it), it falls back to a kind of HTTP polling. I guess that means that from a load-balancer point of view, it creates 100s of visits per minute. And right now, I am wondering if that has an influence on the charges.
My really long question comes down to a simple one. Do you think upgrading my load balancer would decrease the number of counted "loadbalancer hours" ?
EDIT
Here are some ELB metrics. They are too complicated for me to draw conclusions. But perhaps some of you experts can. :)
Does linode load balancers support HTTP long polling and/or websockets? I couldn't find any info about this on their site.
Thanks.
they do, but unfortunately there's not going to be any documentation to help you. I setup my site noobniche.com with them and it was a complete pain. Even a shared host like Webfaction will allow this, but you'll consume your shared hosting plan pretty quickly. Linode is a pretty good bet, you have complete access to everything.
I'm searching for some examples on how to write a proxy in Ruby that supports HTTPS. I have a simple proxy implemented with Webricks HTTPProxyServer, but I noticed, that HTTPS traffic is just tunneling (as it should ;) ). But I want to record the content with VCR (regarding my question here VCRProxy: Record PhantomJS ajax calls with VCR inside Capybara) and as long the content is only tunnled through, VCR can't record it.
So I was thinking of writing the proxy as a man-in-the-middle, generate SSL certificates on the fly (I don't care about certificate errors, its just for testing), and then I would be able to record the content / playback it later.
So if somebody has a good ressource from how to start, or a tutorial or a gist, please let me know.
PS: I have already seen this questions, but they don't provide any further stuff (and it need to be in ruby):
Man in the Middle (MITM) proxy with HTTPS support
How do I write a simple HTTPS proxy server in Ruby?
Help with HTTP Intercepting Proxy in Ruby?
An old question, but for the sake of completeness here goes another answer.
I've implemented a HTTP/HTTPS interception proxy in Ruby, the project is hosted in github.
The project is new, so it's not (yet) as mature as Python's mitmproxy, but it supports HTTPS with certificates generation on-the-fly.
There's an excellent MITM proxy in Python aptly named mitmproxy. The netlib library by the author does the tricks and mitmproxy uses it.
The codebase isn't large and it shouldn't be hard to go through it given that you know Ruby.
How does Netnanny or k9 Web Protection setup web proxy without configuring the browsers?
How can it be done?
Using WinSock directly, or at the NDIS or hardware driver level, and
then filter at those levels, just like any firewalls soft does. NDIS being the easy way.
Download this ISO image: http://www.microsoft.com/downloads/en/confirmation.aspx?displaylang=en&FamilyID=36a2630f-5d56-43b5-b996-7633f2ec14ff
it has bunch of samples and tools to help you build what you want.
After you mount or burn it on CD and install it go to this folder:
c:\WinDDK\7600.16385.1\src\network\ndis\
I think what you need is a transparent proxy that support WCCP.
Take a look at squid-cache FAQ page
And the Wikipedia entry for WCCP
With that setup you just need to do some firewall configuration and all your web traffic will be handled by the transparent proxy. And no setup will be needed on your browser.
netnanny is not a proxy. It is tied to the host machine and browser (and possibly other applications as well. It then filters all incoming and outgoing "content" from the machine/application.
Essentially Netnanny is a content-control system as against destination-control system (proxy).
Easiest way to divert all traffic to a certain site to some other address is by changing hosts file on local host
You might want to have a look at the explanation here: http://www.fiddlertool.com/fiddler/help/hookup.asp
This is how Fiddler2 achieves inserting a proxy in between most apps and the internet without modifying the apps (although lots of explanation of how-to failing the default setup). This does not answer how NetNanny/K9 etc work though, as noted above they do a little more and may be a little more intrusive.
I believe you search for BrowserHelperObjects. These little gizmos capture ALL browser communication, and as such can either remote ads from the HTML (good gizmo), or redirect every second click to a spam site (bad gizmo), or just capture every URL you type and send it home like all the WebToolBars do.
What you want to do is route all outgoing http(s) requests from your lan through a reverse proxy (like squid). This is the setup for a transparent web proxy.
There are different ways to do this, although I've only ever set it up OpenBSD and Linux; and using Squid as the reverse proxy.
At a high level you have a firewall with rules to send all externally bound http traffic to a local squid server. The Squid server is configured to:
accept all http requests
forward the requests on to the real external hosts
cache the reply
forward the reply back to the requestor on the local lan
You can then add more granular rules in Squid to control access to websites, filter content, etc.
I pretty sure you can also get this functionality in different networking gear. I bet F5 has some products that do some or all of what I described, and probably Cisco as well. There is probably other proxies out there besides Squid that you can use too.
PS. I have no idea if this is how K9 Web Protection or NetNanny works.
Squid could provide an intercept proxy for HTTP and HTTPs ports, without configuring the browsers and it also supports WCCP.