How can I dump to file, using mitmdump or mitmproxy, the decrypted https traffic ?
Also is it possible to write on a file only a traffic come from to a particular ip address ?
Related
Researching on how a server can figure out a client's IP address, I see that one needs to inspect the X-Forwarded-For header chain.
I understand that the client, ISP, and then routers and proxies declare their IP addresses there.
However, the server handler also has access to req.RemoteAddr field to read the client's IP address. How is that RemoteAddr determined exactly? Is it based on a specific header in the request? If yes, which one(s)?
I have tried inspecting the usage of the field and how it is set but the implementation details are hidden behind an interface.
The net/http server sets RemoteAddr to the string form of the network connection's remote address. The string is typically in the format "IP:port".
In the case of a TCP connection (the typical scenario), the network connection remote address is taken from the IP source address and the TCP source port.
The address can be the address of the client or a proxy.
The net/http server does not consider the headers when setting RemoteAddr.
I am using a call to ftp.exe to upload file to a FTP Server.
This program is running since many years and uploads to number of servers, so far without problems.
After one of the receiver servers has been updated, uploads are no longer possible.
This is the command sequence:
open ftp.xxx.de
<user>
<pw>
>230 User logged in, proceed
cd upload
bin
put <filename>
and in response to this the server replies:
501 PORT IP is not the same as 10.100.244.5
150 File Status okay, about to open Connection
That is it, after this the connection is stuck and gets closed after a certain timeout period.
Funny enough, a google search for "PORT IP is not the same as" return exactly one result, which explains that the IP seen by the server is different from the one expected.
Also, when using WinSCP, FileZilla or other FTP utility programs, the connection has no problem and does transfer files just fine.
So, why does this appear and how to solve it?
The ftp.exe uses an old-fashioned active mode command PORT, which requires the client to specify its IP address to which the FTP servers needs to connect back to open a data transfer connection.
If your are behind a firewall or a NAT, the client may not know its external IP address and uses its local network address. This causes troubles. Either the server fails to connect back as it obviously cannot connect to the client's local network. Or the server rejects the PORT command straight away, if the specified IP address does not match the IP address, from which the FTP client connects to the server. This is a security measure as the difference may indicate a man-in-the-middle attack. Your server does the validation. Some servers might be configured to ignore the IP address specified in the PORT command and connect to a known IP address of the client.
Another way to solve this is, if the firewall/NAT can inspect the FTP traffic and seamlessly modify the IP address in the PORT command. This is obviously not happening.
You do not get the problem with WinSCP or FileZilla, as these clients default to the passive FTP mode, which does not have the problem. Also in the active mode these clients can be configured to use the external IP address. FileZilla also supports the modern EPRT command, that does not need to specify the IP address at all (the server uses the known IP address of the client).
See my article about active/passive FTP mode for details.
I do not think there's any way to make it working with the Windows ftp.exe. It neither supports the passive mode, nor can be configured to use the external IP address, nor supports the EPRT command.
So unless you can configure the FTP server not to do the check and connect to the known IP address of the client or configure your firewall/NAT to modify the IP address in the PORT command, you have to use another FTP client.
As you know that WinSCP works, see the guide for converting the Windows ftp.exe script to WinSCP script.
(I'm the author of WinSCP)
I'm trying to use Fiddler to capture traffic that comes to my machine on its external ip address, and redirect it to the loopback interface without affecting the host header.
I have added the following to the OnBeforeRequest method:
if (oSession.HostnameIs("MyMachineName")){
oSession.bypassGateway = true;
oSession["x-overrideHost"] = "localhost";
}
This works fine for http traffic: I do indeed see a request to http://MyMachineName hit the loopback adaptor with its host header intact.
However, when intercepting https traffic I get the following in the response raw view:
fiddler.network.https> HTTPS handshake to auth.time-wise.net failed. System.IO.IOException The handshake failed due to an unexpected packet format.
I have Fiddler configured to capture and decrypt https traffic.
Does anyone know why this problem occurs and how it can be remedied?
Edit: in response to Eric's request for more information
Fiddler is running as a proxy (i.e. as standard), listening on port 8888.
The clients are (currently) web browsers on the same machine, and so are automatically using the Fiddler proxy, as they've picked up the change in default proxy.
You've left out some important details (e.g. what port is Fiddler running on, and how did you configure the remote client to send its traffic to Fiddler?)
Having said that, you will probably want to change your use of x-overrideHost to x-overrideHostname such that the port number of the traffic being retargeted is preserved.
Can you please tell me specifically what kind of data flows through which port during an FTP connection?
To be specific, I'd like to know whether contents of the directory and the server response codes flow through port 20.
FTP uses two types of connections: (1) the control connection (default port 21), which is used to send commands to the server and receive status codes back, and (2) the data connection (default port 20), which is used to transfer the content requested from the server: the content of a file or a directory listing, for example. I recommend to use a network protocol analyzer to see it with your own eyes. FTP specification RFC959 is written in an easily understandable manner, don't afraid to have a look at it.
http://ipfuck.paulds.fr/
We've been recently getting hammered by this Firefox plug-in. It sends a fake IP in the headers so when our nginx web server picks up the IP it is a fake one.
Is there any way to get a real IP address or block out requests that have this plug-in installed?
There is actually no client IP entries in any HTTP Headers. There are only some un-official proxy headers which are added to a request, so that a proxy server can tell you the real ip of the connecting client (since the tcp socket will only reveal the IP address of the proxy server).
The plugin you linked to adds those proxy headers, to "fake" a proxy request, by adding a X-Real-IP: 1.2.3.4 or X-Forwarded-For: 1.2.3.4 header to the request. But no one forces you to use that IP address (which can be fake, like the 1.2.3.4 example here), you can always use the IP address of the socket that initiated the connection - which will be the client's real IP address if he uses the mentioned plugin.
Within the location section of your nginx configuration, you get the socket IP address through the $remote_addr variable. To retrieve the "fake" IP address, you can use $http_x_forwarded_for or $http_x_real_ip variable.
If you are using any application/cgi backend, you usually can examine the full headers and the socket IP address (i.e. in PHP you should check $_REQUEST and $_HEADERS variables)