I am using a third party (OS X) tool to help me process OFX financial data. It works, but I am interested in knowing what exactly is going on behind the scenes to make it work (the structure of the HTTP requests).
I setup Charles as an SSL proxy for all traffic in hopes that I could observe the requests being made by this tool, but the program runs and Charles gets nothing. No requests show up whatsoever. How is that possible? Is there something I am not understanding about how Charles or other packet sniffing tools work? What are some ways that web requests could be made that wouldn't show up in a tool like Charles?
Charles is not a packet sniffer. It's a proxy. The app initiating the connection has to "voluntarily" use the proxy for the proxy to see anything. If an app uses a high-level networking API like NSURLConnection then it will, by virtue of the frameworks, automatically pick up the system-wide proxy settings and use the proxy. If, instead, the app wrote their networking using low-level socket API, then they will not end up going through the proxy unless they specifically re-implement that functionality.
If you want to see everything, you will need a real promiscuous-mode packet sniffer, which Charles is not. Unfortunately, using a "real" packet sniffer will just show you the gibberish going over the wire for SSL connections, so that's probably not what you want either. If an app has "in-housed" its SSL implementation and is not using a properly configured system-wide proxy, sniffing its traffic unencrypted will be considerably harder (you'll probably have to use a debugger or some other runtime hooking approach.)
Related
I am trying to find (or write) a caching proxy tool that accepts all traffic from a specific container in my localhost (using Iptables). What I want to do with this traffic is to save it and cache the response, and later, if I see that a request was already sent to a server, return the cached response to the requesting party (and not sending the request to the server again, because a previous similar request was already sent).
Here's a diagram to demonstrate what I'm trying to do:
I'm not sure exactly how big is the problem I'm trying to deal with here. I want to do it for all traffic, including HTTP, TLS and other TCP based traffic (database connections and such). I tried to check mitmproxy, and it seems to deal pretty good with HTTP and the TLS part, but intercepting raw TCP traffic (for databases etc.) is not possible.
Any advices or resources I can use to accomplish that? (Not necessarily in Python). How complex do you think this problem is? Do you think I can find a generic solution?
Thanks in advance!
I am trying to capture an old application that didn't honour the system's proxy setting. The only config I can change is the server IP address.
Capturing the packets with Wireshark. Without the Charles reverse proxy, I can see requests after the first three handshake requests.
With the reverse proxy, the connection stuck after the handshake requests.
I notice that when Charles received a request and connecting to somewhere but it will just stuck there:
Following is the config of the reverse proxy (Remote host removed):
Any help, solution and workarounds would be appreciated!
First of all, your app uses neither HTTP nor HTTPS. Studying screen shot of successful connection gives some details on protocol used:
the first message after handhsake is originated by server contrary to common client-server approach, where client is responsible for sending query. This fact is enough to cross out HTTP and HTTPS.
payload data isn't human-readable, so it's a binary protocol.
based on PUSH flags, protocol is much more likely to be message-based rather than stream-based
So client establishes connection, immediately gets some command from server and replies it. Then communication continues. I can't guess exact protocol. Port number might be irrelevant, but even if it's not, there are only few protocols using 4321 port by default. Anyway, it can always be custom private protocol.
I'm not familiar with Charles, but forwarding arbitrary TCP stream is probably covered by its port forwarding feature rather than reverse proxy. However, I don't really see any benefits in sending traffic through Charles in this case, capturing data on your PC should be enough to study details.
If you are looking for traffic manipulation, for arbitrary TCP stream it's not an easy task, but it must be possible. I'm not aware of suitable tools, quick googling shows lots of utils, but some of them looks applicable to text based stream only, so deeper study is required.
Reason for Failure
It may be because you are requesting a local IP address from a remote scope, which Charles proxy doesn't applies. For POS(Proof Of Statement), please refer to the below link
https://www.charlesproxy.com/documentation/faqs/localhost-traffic-doesnt-appear-in-charles/
Solution
So In order to solve the problem for the current scenario, use
http://192.168.86.22.charlesproxy.com/
Note: The url that you request will only be proxied properly by Charles not any other proxy services.
Need to be able to continuously receive calls when a Chrome webpage is open. How do I do that even for users who are inside a strict enterprise network?
WebSockets? (but there's the proxy problems that doesn't know what wss:// is)
HTTP? (but will I have to poll?)
Other?
Since you included the "vLine" tag, I'll reply with some information on how our WebRTC platform will behave in an enterprise network. vline.js will use a secure WebSocket by default if the browser supports it and fall back to HTTPS long polling. As described here, the secure WebSocket may work depending on the exact proxy configuration. Feel free to test it out by using GitTogether or creating your own vLine service for testing.
I'm in the process of putting my whole website behind SSL because something requires it right now. It's the first time I do that. I normally debug my (php/asp.net) website via Wireshark, and I really worry about the fact this will be totally obscure/encrypted once running in SSL.
Should I say goodbye to fast, simple debugging with a packet sniffer, or there are well-known practices to debug it?
You can decrypt (some) SSL traffic with WireShark. Since you own the domain, you own the private key the site will use. Configure WireShark to use it to decrypt the traffic to/from your site.
http://wiki.wireshark.org/SSL/
https://wirewatcher.wordpress.com/2010/07/20/decrypting-ssl-traffic-with-wireshark-and-ways-to-prevent-it/
This might sound really naive but I would really find a descriptive answer helpful.
So, my question is this:
I can use Firebug to look at AJAX requests made from any website I visit. So, am I right in saying that I wouldn't be able to examine the same communication between the client and the server if the website choses to use Websockets? In other words, does this make it more secure?
No. Not at all. Just because the browser does not (yet) have a tool to show WebSocket traffic, doesn't make it any more secure. You can always run a packet sniffer to monitor the traffic, for example.
No, because there will be other ways beside the browser-build in tools to read your traffic.
Have a try: Install and run Wireshark and you will be able to see all packets you send and receive via Websockets.
Depends on the application. If you are fully Ajax without reloading the document for data then I would think websockets would provide a better authentication for data requests then a cookie session in regards to connection hijack. Given that you are using SSL of course.
Never rely on secrecy of algorithm cause it only gives you false sense of security. Wiki: Security by obscurity
Remember that browser is a program on my computer and I am the one who have a full control over what is send to you, not my browser.
I guess it's only matter of time (up to few months IMO) when developer tools such as Firebug will provide some fancy tool for browsing data send/received by WebSockets.
WebSockets has both an unencrypted (ws://) and encrypted mode (wss://). This is analogous to HTTP and HTTPS. WebSockets protocol payload is simply UTF-8 encoded. From a network sniffing perspective there is no advantage to using WebSockets (use wss and HTTPS for everything at all sensitive). From the browser perspective there is no benefit to using WebSockets for security. Anything running in the browser can be examined (and modified) by a sufficiently knowledgeable user. The tools for examining HTTP/AJAX requests just happen to be better right now.