I want to log all HTTP requests made by the browser to a file, so I thought I can run a HTTP/S proxy locally and do this. However, the proxies at proxies.xhaus.com/ don't meet my needs - either no HTTPS support or no logging. Do anyone of you know of a proxy that can do both HTTPS and HTTP and allow me to log the browser traffic to a file?
Squid can do that.
http://wiki.squid-cache.org/Features/SslBump
Squid was also my first thought given your description, but for development use you might prefer a more powerful intercepting proxy like:
Fiddler2
Paros Proxy
Burp Suite (despite the name, my personal favorite)
Related
I need to develop an app that intercepts HTTPS traffic on the machine and change it.
One approach is to use a local proxy, like Fiddler works.
I also saw HTTP Debugger (https://www.httpdebugger.com/) which works in a similar way to Fiddler but say they don't use a proxy.
Do you know what technology HTTP Debugger are using in order to intercept HTTPS traffic without proxy?
One way is using a pre-master secret key.
Here is an example with Wireshark: https://www.comparitech.com/net-admin/decrypt-ssl-with-wireshark/
Wireshark using WinPcap https://www.winpcap.org/ network capture library to intercept traffic, so there is no proxy involved.
EDIT: I don't know how exactly works HTTP Debugger, but if there is no proxy the only way is to decrypt traffic on network level.
I have a use case where I have to put a middle server or relay or tunnel to do network communication with the following points:
I have a web server running, let say when I hit an API /request hosted my web server, it creates a post request to https://www.google.com and gives me a response through the endpoint.
I want a middle server (proxy etc.) which I will call while creating this post request instead of communicating through my webserver,
the call goes to the middle server and gives me the same response as I was getting directly.
For this, the SQUID proxy worked for me.
I came across NGINX, but we can not use NGINX as a forward proxy, also there are some observations that might be useful with this regard.
SQUID proxy also uses the conf file as similar to NGINX,
HTTPS traffic is encrypted, the proxy server need to do some more work to get something with Https requests,
For intercepting, and creating ACL rules, someone will need to have a dummy certificate to be used by the server to act as the owner of the requested content through the proxy,
a list of rules can be incorporated within SQUID.conf to achieve the filtering.
I hope this could be useful to achieve something like this.
HTTP proxy with SSL and DNS support.
I must be lacking some key concepts about proxy-ing because I cannot grasp this. I am looking to run a simply http or https proxy without interfering with SSL. Simply, a fully transparent proxy that can passthrough all the traffic to the browser connected via HTTP or HTTPS proxy without modifying or intercepting any packets. Not able to find any code online or I'm not using the right keywords.
EX. On the browser adding server.someVPN.com:80 on the HTTP proxy field and as soon as you try to visit a website, it prompts for authentication. Then it works perfectly with any domain, any security, any ssl, no further steps needed. Most VPN providers have this.
How's this possible? it even resolves DNS itself. I thought on transparent proxy the dns relies on the client. Preferably looking for a nodeJS solution but any lang works.
Please don't propose any solutions such as SOCKS5 or sock forwarding or DNS overriding or CA based MITM. According to HTTP 1.1 which supports 'CONNECT' this should be easy.
Not looking to proxy specific domains, looking for an all inclusive solution just like most VPN Providers providers.
----Found the answer too quickly, feel free to delete this post/question admins.
The way it works is that the browser knows it is talking to a proxy server, so for example if the browser want to connect to htttp://www.example.com it sends a CONNECT www.example.com:443 HTTP/1.1 to the proxy server, the proxy server resolves wwww.example.com via DNS and then opens a TCP connection to wwww.example.com port 443 and proxies the TCP stream transparently to the client.
I don't know any solution for nodejs. Common proxy servers include Squid, Privoxy and Apache Traffic Server
See also: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT
Found the solution right after I asked...
This module works perfectly https://github.com/mpangrazzi/harrier
Does exactly what I was asking for.
I'm using Charles as a proxy to debug some HTTP requests. However, one of the endpoints is www.myendpoint.com:9064 and I need a proxy (myproxy.com:80) to access it. When I try to debug it with charles it fails to connect to that endpoint. Do I need to do any extra configuration for this case?
Charles Proxy automatically tries to configure your system's proxy (in Windows and macOS) so by this many applications, such as browsers, use it automatically.
By what you say in your comment, you seem to be developing an application, which I guess you configured to use the myproxy.com:80 proxy.
Being this the case, what you need to do is to configure your application to use the Charles Proxy, which usually opens up at your localhost:8888, instead of the myproxy:80.
This will make your application access the internet through the Charles proxy and thus your HTTP request will start to appear to the Charles overview window, but the requests will still not be able to reach www.myendpoint.com:9064.
To do so, you need to configure your proxy values (myproxy.com:80) on the External Proxy Settings (Proxy -> External Proxy Settings...) on Charles.
This will make Charles be able to reach your endpoint.
To access some websites that I need to test I need to plugin some specific proxies on firefox. However when I set my proxy through firefox or any browser, fiddler doesnt capture anymore request, due that it also uses its own proxy to get the requests. Is there any way I could capture traffic using fiddler while at the same time Im using an external proxy using any browser?
Fiddler allows you to chain to an upstream proxy, so traffic first goes to Fiddler, then goes from Fiddler to another proxy.
Fiddler will automatically chain to any proxy configured in Internet Explorer.
To manually chain to a different proxy, in Fiddler, click Tools > Fiddler Options and go to the Gateway tab. Specify the desired upstream proxy there.