Is there a way for Squid Proxy to intercept and filter HTTPS payload from request made before it is being sent? Just like the way mitmproxy does.
I found out it's possible to log the payload with ssl-bumping, I had some success also altering response data with sample ecap. Only if I knew some C programming.
Also, iptables has a function to filter payload by string but only with http traffic. I don't know if it's possible to strip the SSL traffic first before feeding it to iptables and send the request outside before sending back (mitm proxy basically).
The only drawback with mitmproxy right now is unable for daemonization (you need to hack it), and I have my doubt using it in a large load.
Any one way or another?
Related
I am trying to find (or write) a caching proxy tool that accepts all traffic from a specific container in my localhost (using Iptables). What I want to do with this traffic is to save it and cache the response, and later, if I see that a request was already sent to a server, return the cached response to the requesting party (and not sending the request to the server again, because a previous similar request was already sent).
Here's a diagram to demonstrate what I'm trying to do:
I'm not sure exactly how big is the problem I'm trying to deal with here. I want to do it for all traffic, including HTTP, TLS and other TCP based traffic (database connections and such). I tried to check mitmproxy, and it seems to deal pretty good with HTTP and the TLS part, but intercepting raw TCP traffic (for databases etc.) is not possible.
Any advices or resources I can use to accomplish that? (Not necessarily in Python). How complex do you think this problem is? Do you think I can find a generic solution?
Thanks in advance!
I am trying to capture an old application that didn't honour the system's proxy setting. The only config I can change is the server IP address.
Capturing the packets with Wireshark. Without the Charles reverse proxy, I can see requests after the first three handshake requests.
With the reverse proxy, the connection stuck after the handshake requests.
I notice that when Charles received a request and connecting to somewhere but it will just stuck there:
Following is the config of the reverse proxy (Remote host removed):
Any help, solution and workarounds would be appreciated!
First of all, your app uses neither HTTP nor HTTPS. Studying screen shot of successful connection gives some details on protocol used:
the first message after handhsake is originated by server contrary to common client-server approach, where client is responsible for sending query. This fact is enough to cross out HTTP and HTTPS.
payload data isn't human-readable, so it's a binary protocol.
based on PUSH flags, protocol is much more likely to be message-based rather than stream-based
So client establishes connection, immediately gets some command from server and replies it. Then communication continues. I can't guess exact protocol. Port number might be irrelevant, but even if it's not, there are only few protocols using 4321 port by default. Anyway, it can always be custom private protocol.
I'm not familiar with Charles, but forwarding arbitrary TCP stream is probably covered by its port forwarding feature rather than reverse proxy. However, I don't really see any benefits in sending traffic through Charles in this case, capturing data on your PC should be enough to study details.
If you are looking for traffic manipulation, for arbitrary TCP stream it's not an easy task, but it must be possible. I'm not aware of suitable tools, quick googling shows lots of utils, but some of them looks applicable to text based stream only, so deeper study is required.
Reason for Failure
It may be because you are requesting a local IP address from a remote scope, which Charles proxy doesn't applies. For POS(Proof Of Statement), please refer to the below link
https://www.charlesproxy.com/documentation/faqs/localhost-traffic-doesnt-appear-in-charles/
Solution
So In order to solve the problem for the current scenario, use
http://192.168.86.22.charlesproxy.com/
Note: The url that you request will only be proxied properly by Charles not any other proxy services.
I want to take a look at the direct encrypted https traffic of my requests. I got a server and 2 clients. With client 1 there are no problems. client 2 gets problems if the request exceeds a certain text-size. I was able to decrypt the traffic and found that client 1 extends my SOAP request every 3996 signs by an extra "f9c"-pattern. Client 2 is not doing this which is probably causing the problem on the serverside. But that is not all. If I use fiddler to decrypt the https request the server also throws an exception. So my guess is that the client is probably adding something on the https encryption too. That is why I want to take a look at this but I cannot figure out how to force fiddler to show this to me. I only get the http traffic if I disable https-decryption that shows my the handshake between client and server. So what can I do here?
problem solved.
the "f9c"-pattern is just the hex value for 3996-bytes. The problem was that client 1 was correctly chunking the http-request. Client 2 on the contrary was sending a single block and just setting the content-length of the http-request. Therefore I do not need to take a look anymore on the encrypted request.
Our web application has a button that is supposed to send data to a server on the local network that in turn prints something on a printer.
So far it was easy: The button triggered an AJAX POST request to http://printerserver/print.php with a token, that page connected to the web application to verify the token and get the data to print and then printed.
However, we are now delivering our web application via HTTPs (and I would rather not go back to HTTP for this) and newer versions of Chrome and Firefox don't make the request to the HTTP address anymore, they don't even send the request to check CORS headers.
Now, what is a modern alternative to the cross-protocol XHR? Do Websockets suffer from the same problem? (A Google search did not make clear what is the current state here.) Can I use TCP Sockets already? I would rather not switch to GET requests either, because the action is not idempotent and it might have practical implications with preloading and caching.
I can change the application on the printerserver in any way (so I could replace it with NodeJS or something) but I cannot change the users' browsers (to trust a self-signed certificate for printerserver for example).
You could store the print requests on the webserver in a queue and make the printserver periodically poll for requests to print.
If that isn't possible I would setup a tunnel or VPN between the webserver and printserver networks. That way you can make the print request from the webserver on the server-side instead of the client. If you use curl, there are flags to ignore invalid SSL certificates etc. (I still suspect it's nicer to introduce a queue anyway, so the print requests aren't blocking).
If the webserver can make an ssh connection to something on the network where the printserver is on, you could do something like: ssh params user#host some curl command here.
Third option I can think of, if printserver can bind to for example a subdomain of the webserver domain, like: print.somedomain.com, you may be able to make it trusted by the somedomain.com certificate, IIRC you have to create a CSR (Certificate Signing Request) from the printserver certificate, and sign it with the somedomain.com certificate. Perhaps it doesn't even need to be a subdomain for this per se, but maybe that's a requirement for the browser to do it client-side.
The easiest way is to add a route to the webapp that does nothing more than relay the request to the print server. So make your AJAX POST request to https://myapp.com/print, and the server-side code powering that makes a request to http://printerserver/print.php, with the exact same POST content it received itself. As #dnozay said, this is commonly called a reverse proxy. Yes, to do that you'll have to reconfigure your printserver to accept (authenticated) requests from the webserver.
Alternatively, you could switch the printserver to https and directly call it from the client.
Note that an insecure (http) web-socket connection on a secure (https) page probably won't work either. And for good reason: generally it's a bad idea to mislead people by making insecure connections from what appears to them to be a secure page.
The server hosting the https webapp can reverse proxy the print server,
but since the printer is local to the user, this may not work.
The print server should have the correct CORS headers
Access-Control-Allow-Origin: *
or:
Access-Control-Allow-Origin: https://www.example.com
However there are pitfalls with using the wildcard.
From what I understand from the question, printserver is not accessible from the web application so the reverse proxy solution won't work here.
You are restricted from making requests from the browser to the printserver by cross-origin-policy.
If wish to communicate with the printserver from an HTTPS page you will need the printserver to expose print.php as HTTPS too.
You could create a DNS A record as a subdomain of your web application that resolves to the internal address of your printserver.
With those steps in place you should be able to update your printserver page to respond with permissive CORS headers which the browser should then respect. I don't think the browser will even issue CORS requests across different protocol schemes (HTTPS vs HTTP) or to internal domains, without a TLD.
What additional changes are required to make this simple HTTP header to speak to a HTTPS enabled server.
GET /index.php HTTP/1.1
Host: localhost
[CR]
[CR]
EDIT
To add some context, all I'm trying to do is open a TCP port (443) and read the index page but the server seems to return a 400 - Bad request along with a message that goes "You're speaking plain HTTP to an SSL-enabled server port." I thought this probably meant altering the header in some fashion.
HTTP runs on top of secured channel. No adjustments are needed at all on HTTP level. You need to encrypt the whole traffic going to the socket (after it leaves HTTP client code) and decrypt the traffic coming from the socket before it reaches HTTP client.
You encrypt the payload with the information from the server to encrypt. This is done via handshake on a server by server basis, so you can't just fake it once have it work everywhere.
The payload includes the query string, cookies, form, etc.