LFTP active mode with servers that do not recognize the PORT command - ftp

I am using LFTP to transfer files from a server, which unfortunately does not recognize the PORT command. I do not have control over the server (do not know in detail what server is) and I have to use the active mode.
This is the command line as:
lftp -e 'debug 10;set ftp:passive-mode off; set ftp:auto-passive-mode no; ls; bye;' -u user,password ftp://ftp.site.com
This is the debug output:
<--- 200 Using default language en_US
---> OPTS UTF8 ON
<--- 200 UTF8 set to on
---> OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
<--- 200 OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
---> USER xxxxx
<--- 331 Password required for xxxxx
---> PASS xxxxxx
<--- 230 User xxxxx logged in
---> PBSZ 0
<--- 200 PBSZ 0 successful
---> PROT P
<--- 200 Protection set to Private
---> PORT 172,16,133,11,146,168
<--- 500 Illegal PORT command
---> LIST
---> ABOR
---- Closing aborted data socket
---- Chiusura del socket di controllo
It seems that LFTP renounces to connect to data socket because the remote server does not support the PORT command. Is there a way to convince LFTP can still connect to port 20? By FTP manual obviously no problem.

The issue, I think, is not that the FTP server doesn't support the PORT command (it does), but rather, it doesn't like the IP address/port that your FTP client is sending in the PORT command.
PORT 172,16,133,11,146,168
...tells the server to connect to address 172.16.133.11, port 37544*. The interesting part here is the IP address: it's an RFC 1918 address (i.e. it's a private network address). That, in turn, suggests that your FTP client is in a LAN somewhere, and is connecting to an FTP server using a public IP address.
That remote FTP server cannot connect to a private network address; by definition, RFC 1918 address are not publicly routable.
Thus it very well could be that the FTP server is trying to make a connection to the address/port given in your PORT command, fails, thus that is why the FTP server fails the command, saying:
500 Illegal PORT command
To make a PORT command work with that FTP server, you would need to discover the public IP address that that server can connect to, to reach your client machine. Let's say that this address is 1.2.3.4. Then you would need to tell lftp to use that address in its PORT command, using the ftp:port-ipv4 option.
Chances are, though, that public IP address is the address of a NAT/router/firewall, and that that NAT/router/firewall will not allow connections, from the outside world to a high numbered port (e.g. 37544), to be routed to a machine within the LAN. This is one of the issues with active FTP data transfers, i.e. FTP data transfers which use the PORT (or EPRT) commands: they are not considered "firewall-friendly".
Hope this helps!
* - why 146,168 translates to port 37544?
According to FTP's RFC959 those parameters are:
(...) 16-bit TCP port address. This address information is broken into
8-bit fields and the value of each field is transmitted as a decimal
number (in character string representation).
146 dec = 10010010 bin = A
168 dec = 10101000 bin = B
A B
10010010 10101000 bin = 37544 dec

Related

ssh client to show server-supported algorithms

In order to check that all the servers across a fleet aren't supporting deprecated algorithms, I'm (programmatically) doing this:
telnet localhost 22
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.0p1 Ubuntu-6build1
SSH-2.0-Censor-SSH2
4&m����&F �V��curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1Arsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519lchacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.comlchacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com�umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1�umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1none,zlib#openssh.comnone,zlib#openssh.comSSH-2.0-Censor-SSH2
Connection closed by foreign host.
Which is supposed to be a list of supported algorithms for the various phases of setting up a connection. (kex, host key, etc). Every time I run, I get a different piece of odd data at the start - always a different length.
There's an nmap plugin - ssh2-enum-algos - which returns the data in it's complete form, but I don't want to run nmap; I have a go program which opens the port, and sends the query, but it gets the same as telnet. What am I missing, and how do I fix it?
For comparison, here's the top few lines from the output of nmap script:
$ nmap --script ssh2-enum-algos super
Starting Nmap 7.80 ( https://nmap.org ) at 2019-12-27 22:15 GMT
Nmap scan report for super (192.168.50.1)
Host is up (0.0051s latency).
rDNS record for 192.168.50.1: supermaster
Not shown: 998 closed ports
PORT STATE SERVICE
22/tcp open ssh
| ssh2-enum-algos:
| kex_algorithms: (12)
| curve25519-sha256
| curve25519-sha256#libssh.org
| ecdh-sha2-nistp256
| ecdh-sha2-nistp384
| ecdh-sha2-nistp521
Opening a tcp connection to port 22, (in golang, with net.Dial) then accepting and sending connection strings leaves us able to Read() from the Reader for the connection. Thence the data is in a standard format described by the RFC. From this, I can list the algorithms supported in each phase of an ssh connection. This is very useful for measuring what is being offered, rather than what the appears to be configured (it's easy to configure sshd to use a different config file).
It's a useful thing to be able to do from a security POV.
Tested on every version of ssh I can find from 1.x on a very old solaris or AIX box, to RHEL 8.1.
In some cases you can specify an algorithm to use, and if you specify one that is not supported the server will reply with a list of supported algorithms.
For example, to check for supported key exchange algorithms you can use:
ssh 127.0.0.1 -oKexAlgorithms=diffie-hellman-group1-sha1
diffie-hellman-group1-sha1 is insecure and should be missing from most modern servers. The server will probably respond with something like:
Unable to negotiate with 127.0.0.1 port 22: no matching key exchange method found. Their offer: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
Exit 255
Typing: "ssh -Q cipher | cipher-auth | mac | kex | key"
will give you a list of the algorithms supported by your client
Typing: "man ssh"
will let you see what options you can specify with the -o argument, including Cipher, MACs, and KexAlgorithms

lftp 550 permission denied

i've tried to create a script that will upload some file to a ftp server using lftp, but without any luck so far. If I used build in ftp command in debian I manage to succsessfully connected and put the file.
Here is the debug output from lftp command:
lftp xxx.xxx.xxx.xxxx -e "put -O /out/ some_file_name" -d
---- using user `user01' and password from ~/.netrc
---- Resolving host address...
---- 1 address found:xxx.xxx.xxx.xxxx
---- Connecting to xxx.xxx.xxx.xxxx (xxx.xxx.xxx.xxxx) port 21
<--- 220 (vsFTPd 2.0.7)
---> FEAT
<--- 211-Features:
<--- EPRT
<--- EPSV
<--- MDTM
<--- PASV
<--- REST STREAM
<--- SIZE
<--- TVFS
<--- UTF8
<--- 211 End
---> OPTS UTF8 ON
<--- 200 Always in UTF8 mode.
---> USER user01
<--- 331 Please specify the password.
---> PASS XXXX
<--- 230 Login successful.
---> PWD
<--- 257 "/"
---> TYPE I
<--- 200 Switching to Binary mode.
---> EPSV
<--- 550 Permission denied.
---- Switching passive mode off
---- Closing data socket
---- Closing control socket
As you can see I'm using stored user name and password from .netrc file. I have another script that connect to the same server but uploads files and rename them inside the remote ftp folder using lftp again. Can someone help and explain why I cannot put with lfpt but can do it using ftp.
after add set ftp:passive-mode true and set ftp:prefer-epsv false to /etc/lftp.conf the error is changed
<--- 230 Login successful.
---> PWD
<--- 257 "/"
---> TYPE I
<--- 200 Switching to Binary mode.
---> PASV
<--- 227 Entering Passive Mode (xxx.xxx.xxx.xxx,76,92)
---- Connecting data socket to (xxx.xxx.xxx.xxx) port 19548
---- Data connection established
---> ALLO 710
<--- 550 Permission denied.
---> STOR out/my_file_name
---> ABOR
put: Access failed: 550 Permission denied. (/out/my_file_name)
---- Closing aborted data socket
---- Closing control socket
Okay I've understand what ALLO means
The ALLO command may be sent to a server that requires the necessary space for an uploaded to be reserved before the transfer takes place
so after a quick search in ftp man page, I've found a command to shut it down. After adding set ftp:use-allo false and with epsv false everything is fine now. Thanks alot :)
lftp -e "set ftp:use-allo false; set ftp:passive-mode true; set ftp:prefer-epsv false; mirror -R {local dir} {remote dir}" -u {username},{password} {host}
Use this single command to sync your file from local to server without 550 permission error.
While 550 Permission denied. is a strange response to the EPSV command it means that the server or some middlebox in between does not understand the EPSV command (likely a middlebox since the response to FEAT shows EPSV as supported). If you use the builtin ftp command instead of lftp it will probably use the older PASV command (IPv4 only) instead of the newer EPSV command (IPv4+IPv6 capable).
According to the man page there is a setting ftp:prefer-epsv which should default to false. Maybe some configuration is setting this value to true so that lftp will use EPSV instead of PASV. Check your settings (set -a inside lftp) and if it is true (expected) set it to false and try again, in the hope that it will then use PASV instead of EPSV.

Hostnames resolution fails with "unknown host" error for hostnames containing utf-8 characters

I am trying to ping a hostname "win-2k12r2-addc.阿伯测阿伯测ad.hai.com" from a linux client.
I see that DNS requests go over the wire with hostname being sent in utf-8 format
and i get a response from the DNS server also with the correct IP address.
But ping fails with the following error :
ping: unknown host win-2k12r2-addc.阿伯测阿伯测ad.hai.com
If i add an entry into /etc/hosts, it works fine
I have the following entries in /etc/hosts when it works.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
127.0.0.1 localhost ava-dev
::1 localhost
10.141.33.93 win-2k12r2-addc.阿伯测阿伯测ad.hai.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The /etc/nsswitch.conf file has the following entries for hosts.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hosts: files dns
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I somewhat suspect that getaddrInfo() call fails when we try to resolve the address i.e it is not able to handle the DNS responses correctly for hostnames
containing unicode characters.
Has anyone faced this issue before ?
Or has anyone tried resolving a unicode hostname from a linux client ?
The reason i m suspecting getaddrinfo() is because of the following.
Apart from ping, i m trying the following ldap command to the same host and it fails with the below mentioned error
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ldapsearch -d 255 -x -h win-2k12r2-addc.阿伯测阿伯测ad.hai.com
ldap_create
ldap_url_parse_ext(ldap://win-2k12r2-addc.%E9%98%BF%E4%BC%AF%E6%B5%8B%E9%98%BF%E4%BC%AF%E6%B5%8Bad.hai.com)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP win-2k12r2-addc.阿伯测阿伯测ad.hai.com:389
ldap_connect_to_host: getaddrinfo failed: Name or service not known
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In both the scenarios (ping / ldap), i see the DNS query request going to the DNS server and the correct response from the DNS server back to the linux client.
The following is the value of the hostname sent in the DNS query
win-2k12r2-addc.\351\230\277\344\274\257\346\265\213\351\230\277\344\274\257\346\265\213ad.hai.com: type A, class IN
It looks like you are trying to use UTF-8 or unicode within the DNS system while the DNS system really doesn't like that. It wants ascii (See RFCs 5890, 5891, 5892, 5893 - but mostly 5891). Escaping the utf-8 characters does not turn them into the required ascii encoding, called punycode (prefixed by "xn--"). You want to use the version of your IDN that has punycode instead of the UTF-8:
ping win-2k12r2-addc.xn--ad-tl3ca3569aba8944eca.hai.com

Force lftp to open binary connections?

The problem: I am trying to create a local mirror of a public FTP site. When I use lftp to do the job it creates a mirror without a problem, but when I try to update the mirror a few days later it becomes very slow due to getting stuck on several files.
Running lftp -d I can see that lftp makes several requests to RETR the file, but these requests result in several **** Timeout - reconnecting messages and after about 2-3 minutes I see
<--- 150 Opening BINARY mode data connection for {filename removed}.`
After this last command the file successfully downloads and lftp proceeds further.
From the manual I understand that BINARY is the default mode for lftp, but somehow it doesn't seem to work for the early requests. Can someone suggest how I can force lftp to always open BINARY mode data connection to download all files?
Here's a MWE:
``lftp -d -u anonymous,anonymous -c "open {url}; get {file}"``
And response from lftp -d:
---- Connecting to {url} ({IP}) port 21
<--- 220 (vsFTPd 3.0.3)
---> FEAT
<--- 211-Features:
<--- EPRT
<--- EPSV
<--- MDTM
<--- PASV
<--- REST STREAM
<--- SIZE
<--- TVFS
<--- 211 End
---> USER anonymous
<--- 331 Please specify the password.
---> PASS anonymous
<--- 230 Login successful.
---> TYPE I
<--- 200 Switching to Binary mode.
---> SIZE {file}
<--- 213 3321
---> MDTM {file}
--- 213 20160318190446
---> PASV
<--- 227 Entering Passive Mode ({IP}).
---- Connecting data socket to ({IP}) port 55380
---- Data connection established
---> RETR {file}
**** Timeout - reconnecting
---- Closing data socket
---- Closing control socket
lftp uses binary mode by default for all file transfers and ascii mode for directory listings. So the binary mode should not be a problem here.
Maybe you have a subtle connectivity problem, sometimes setting net:socket-maxseg to a lower value (e.g. 500) helps.
The ftp command for binary mode is bin so use that command before you get the file.
cd /direc/tory; bin; get file.xml
The problem is that I see
---> TYPE I
<--- 200 Switching to Binary mode.
in your output, so you're already in binary mode. I wonder if you have a different problem? I also see that you're using passive mode (PASV), and that's good because passive works around firewalls and NATs [1], so we need another reason why you see those timeouts.
Do you have any other clues, maybe from ping or netstat?

Telnet Not Working in Ruby

I need to Telnet into my associate's server for a project I'm working on:
telnet = Net::Telnet::new(
"Host" => "example_ip",
"Port" => 80,
"Timeout"=>90,
"Waittime" => 1,
"Output_log"=>"output_log.log",
"Dump_log"=> "dump_log.log"
)# => #<Net::Telnet:0x007f8142321c00 ...
telnet.cmd('ls') # => "HTTP/1.1 0 ERROR\n\nERROR 0\n"
My output_log looks like:
Trying example_ip...
Connected to example_ip.
HTTP/1.1 0 ERROR
ERROR 0
I know his server is running and I can connect to it. Why can't I connect from Ruby?
(from the OSX terminal)
$ telnet example_ip 80
Trying example_ip...
Connected to c-example_ip...comcast.net.
Escape character is '^]'.
...
$ ls
_~1 .TRA 22 4096
TRASHE~1. 12 0
FSEVEN~1. 12 0
20110221.LOG 0 73
CONFIG . 0 573
20110223.LOG 0 95
20110224.LOG 0 17
20110225.LOG 0 17
20110315.LOG 0 73
...
Edit:
I tried setting Binmode to false, Telnetmode to false, and some different values for the Prompt (but I don't know what the right one would be).
One problem is you're telnetting to port 80 in your code, which is the HTTPd port. That's why you are getting:
HTTP/1.1 0 ERROR
The server is expecting you to send a HTTP command, such as GET /index.html, but instead you're sending "ls".
You don't say what host OS you are using on your machine's command-line to connect to the server, but, it's possible the telnet command on that machine expects the port command to be separated from the host IP or FQDN using a colon:
telnet example_ip:80
Failing to use the colon on that type of client causes the connection to occur to the standard telnet port on port 23, which seems to be backed up by your ls command working. A HTTPd on port 80 wouldn't know what you meant by ls.
You can use the Telnet protocol to connect to a HTTPd, and you can use a telnet client app also. You have to use the right command-set with it though. Try adjusting your code to telnet.cmd("GET /index.html\r\n") and see if you get a response back.
EDIT:
Use OpenURI to experiment. It's a very simple and convenient way to poke at web servers:
require 'open-uri'
doc = open('http://example.com').read
puts doc[0, 100]
That will open the connection to port 80 at the host given, following redirects, and returning the document received. It then prints the first 100 characters which look like:
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-
You don't say why you "need" to telnet to a web server. They don't support the same commands you can issue via a normal telnet session, and won't give you access to the file system, unless code has been written for those purposes.

Resources