I'm trying to log into an ftps site. I've tried giving the login creds at the command line (and putting set parameters in ~/.lftprc, then opening an lftp session and typing those parameters with lftp job control statements. Regardless, I keep hitting the same roadblock:
421 Sorry, cleartext sessions are not accepted on this server.
Please reconnect using SSL/TLS security mechanisms.
I got furthest with the following parameters, but keep getting the error above.
How do I get lftp to use SSL/TLS security mechanism from the command line?
The objective is to script the access to this ftps site using bash (programming without using expect).
lftp
lftp :~> set ssl-allow false
lftp :~> set passive-mode yes
lftp :~> open ftp.abc.com
lftp ftp.abc.com:~> login theuser
Password:
lftp theuser#ftp.abc.com:~> cd
`cd' at 0 [Delaying before reconnect: 26]
CTRL-C
lftp theuser#ftp.abc.com:~> debug
lftp theuser#ftp.abc.com:~> cd
---- Connecting to ftp.abc.com (XX.XXX.XX.XX) port 21
<--- 220-Welcome to the Yahoo! Web Hosting FTP server
<--- 220-Need help? Get all details at:
<--- 220-http://help.yahoo.com/help/us/webhosting/gftp/
<--- 220-
<--- 220-No anonymous logins accepted.
<--- 220-Yahoo!
<--- 220-Local time is now 15:30. Server port: 21.
<--- 220-This is a private system - No anonymous login
<--- 220 You will be disconnected after 5 minutes of inactivity.
---> FEAT
<--- 211-Extensions supported:
<--- EPRT
<--- IDLE
<--- MDTM
<--- SIZE
<--- MFMT
<--- REST STREAM
<--- MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
<--- MLSD
<--- XDBG
<--- AUTH TLS
<--- PBSZ
<--- PROT
<--- TVFS
<--- ESTA
<--- PASV
<--- EPSV
<--- SPSV
<--- ESTP
<--- 211 End.
---> OPTS MLST type;size;modify;UNIX.mode;UNIX.uid;UNIX.gid;
<--- 200 MLST OPTS type;size;sizd;modify;UNIX.mode;UNIX.uid;UNIX.gid;unique;
---> USER theuser
<--- 421 Sorry, cleartext sessions are not accepted on this server.
Please reconnect using SSL/TLS security mechanisms.
It seems like lftp is not configured correctly on many systems, which makes it unable to verify server certificates (producing Fatal error: Certificate verification: Not trusted).
The web (and answers in this post) is full of suggestions to fix this by disabling certificate verification or encryption altogether. This is unsecure as it allows man-in-the-middle attacks to pass unnoticed.
The better solution is to configure certificate verification correctly, which is easy, fortunately. To do so, add the following line to /etc/lftp.conf (or alternatively ~/.lftp/rc, or ~/.config/lftp/rc):
set ssl:ca-file "/etc/ssl/certs/ca-certificates.crt"
ca-certificates.crt is a file that contains all CA certificates of the system. The location used above is the one from Ubuntu and may vary on different systems. To generate or update the file, run update-ca-certificates:
sudo update-ca-certificates
If your system does not have this command, you can create one manually like this:
cat /etc/ssl/certs/*.pem | sudo tee /etc/ssl/certs/ca-certificates.crt > /dev/null
lftp :~> set ssl-allow false
You've explicitly set ssl-allow to false. But this must be true if lftp should attempt to use SSL.
You might also need to
set ssl:verify-certificate no
My answer provides access for a single user on your system rather than a system-wide certificate.
lftp uses Transport Layer Security (TLS). So it’s essential to first grab the certificate from the FTP server.
openssl s_client -connect <ftp-hostname>:21 -starttls ftp
I include the entire certificate chain in a new file called cert.crt in my local ~/.lftp folder. At the very least, you're looking to include all the text of the certificate itself: -----BEGIN CERTIFICATE----- <...> -----END CERTIFICATE-----.
I create a file called rc in the local ~/.lftp folder and add the lines
set ssl:ca-file “cert.crt”
set ssl:check-hostname no (this prevents Fatal error: Certificate verification: certificate common name doesn't match requested host name ‘<ftp-hostname>’ when running a command like ls remotely)
Setting ftp:ssl-allow true didn't work for me.
By typing set:
lftp :~> set
I noticed this:
set ftp:ssl-allow true
set ftp:ssl-allow/XXX.XXX.XXX.XXX no
with XXX.XXX.XXX.XXX being the server, I was logging into.
So the final set of commands I needed was:
lftp :~> set ftp:ssl-allow true
lftp :~> set ftp:ssl-allow/XXX.XXX.XXX.XXX true
lftp :~> set ssl:verify-certificate no
lftp version must be >= 4.6.3 (Debian user)
What worked for me step by step with lftp:
get certificate of host with openssl s_client -connect <ftp_hostname>:21 -starttls ftp, at the begining of result I got something like -----BEGIN CERTIFICATE-----
MIIEQzCCAyu.....XjMO
-----END CERTIFICATE-----
copy that -----BEGIN CERTIFICATE-----
MIIEQzCCAyu.....XjMO
-----END CERTIFICATE----- into /etc/ssl/certs/ca-certificates.crt
Into lftp configuration reference this certificate file adding to /etc/lftp.conf for systemwide set ssl:ca-file "/etc/ssl/certs/ca-certificates.crt"
and then do your sync or whatever with lftp, on my case it is lftp -u "${FTP_USER},${FTP_PWD}" ${FTP_HOST} -e "set net:timeout 10;mirror ${EXCLUDES} -R ${LOCAL_SOURCE_PATH} ${REMOTE_DEST_PATH} ; quit"
This worked for me for a FTPS server connection (with port 990, but not necessary to specify) using lftp
code:
lftp ftps://USER:PASSWORD#server.com -c "set ssl:verify-certificate false;"
then:
do stuff
more info at:
how-to-avoid-lftp-certificate-verification-error
Related
I made this script to download a folder in a remote FTP with implicit SSL:
open -d ftps://USER:"PASS"#ftp.xxx.tld:990
mirror --verbose --continue /remote/folder $HOME/destination/folder;
bye
It connects, the features are:
<--- 220-Microsoft FTP Service
<--- 220 FTP SOMECOMPANY
---> FEAT
<--- 211-Extended features supported:
<--- LANG EN*
<--- UTF8
<--- AUTH TLS;TLS-C;SSL;TLS-P;
<--- PBSZ
<--- PROT C;P;
<--- CCC
<--- HOST
<--- SIZE
<--- MDTM
<--- REST STREAM
<--- 211 END
But when the mirror command tries to do its thing, it gets:
<--- 257 "/" is current directory.
---> PBSZ 0
---- CWD path to be sent is `/remote/folder'
<--- 200 PBSZ command successful.
---> CWD remote
<--- 550 Access is denied.
mirror: Access failed: 550 Access is denied. (/remote/folder)
---> CWD folder
<--- 550 The system cannot find the file specified.
---> QUIT
When I manually connect with lftp I do have ls access for example but cannot change directory, same error, 550.
On FileZila it works perfectly, I do have access to the folder and can download files. I'm puzzled here. Already tried cryptic things such as using:
lftp -e "set ftp:use-allo false; set ftp:passive-mode true; set ftp:prefer-epsv false;...
to no avail. Any tips? Thanks!
You probably have to force TVFS here by using “set ftp:use-tvfs yes”. It enables Unix-like paths even if the server forgets to announce their support.
As the ftp server is hosted in a Windows envirionment, I had to use \ instead of / for defining paths.
i've tried to create a script that will upload some file to a ftp server using lftp, but without any luck so far. If I used build in ftp command in debian I manage to succsessfully connected and put the file.
Here is the debug output from lftp command:
lftp xxx.xxx.xxx.xxxx -e "put -O /out/ some_file_name" -d
---- using user `user01' and password from ~/.netrc
---- Resolving host address...
---- 1 address found:xxx.xxx.xxx.xxxx
---- Connecting to xxx.xxx.xxx.xxxx (xxx.xxx.xxx.xxxx) port 21
<--- 220 (vsFTPd 2.0.7)
---> FEAT
<--- 211-Features:
<--- EPRT
<--- EPSV
<--- MDTM
<--- PASV
<--- REST STREAM
<--- SIZE
<--- TVFS
<--- UTF8
<--- 211 End
---> OPTS UTF8 ON
<--- 200 Always in UTF8 mode.
---> USER user01
<--- 331 Please specify the password.
---> PASS XXXX
<--- 230 Login successful.
---> PWD
<--- 257 "/"
---> TYPE I
<--- 200 Switching to Binary mode.
---> EPSV
<--- 550 Permission denied.
---- Switching passive mode off
---- Closing data socket
---- Closing control socket
As you can see I'm using stored user name and password from .netrc file. I have another script that connect to the same server but uploads files and rename them inside the remote ftp folder using lftp again. Can someone help and explain why I cannot put with lfpt but can do it using ftp.
after add set ftp:passive-mode true and set ftp:prefer-epsv false to /etc/lftp.conf the error is changed
<--- 230 Login successful.
---> PWD
<--- 257 "/"
---> TYPE I
<--- 200 Switching to Binary mode.
---> PASV
<--- 227 Entering Passive Mode (xxx.xxx.xxx.xxx,76,92)
---- Connecting data socket to (xxx.xxx.xxx.xxx) port 19548
---- Data connection established
---> ALLO 710
<--- 550 Permission denied.
---> STOR out/my_file_name
---> ABOR
put: Access failed: 550 Permission denied. (/out/my_file_name)
---- Closing aborted data socket
---- Closing control socket
Okay I've understand what ALLO means
The ALLO command may be sent to a server that requires the necessary space for an uploaded to be reserved before the transfer takes place
so after a quick search in ftp man page, I've found a command to shut it down. After adding set ftp:use-allo false and with epsv false everything is fine now. Thanks alot :)
lftp -e "set ftp:use-allo false; set ftp:passive-mode true; set ftp:prefer-epsv false; mirror -R {local dir} {remote dir}" -u {username},{password} {host}
Use this single command to sync your file from local to server without 550 permission error.
While 550 Permission denied. is a strange response to the EPSV command it means that the server or some middlebox in between does not understand the EPSV command (likely a middlebox since the response to FEAT shows EPSV as supported). If you use the builtin ftp command instead of lftp it will probably use the older PASV command (IPv4 only) instead of the newer EPSV command (IPv4+IPv6 capable).
According to the man page there is a setting ftp:prefer-epsv which should default to false. Maybe some configuration is setting this value to true so that lftp will use EPSV instead of PASV. Check your settings (set -a inside lftp) and if it is true (expected) set it to false and try again, in the hope that it will then use PASV instead of EPSV.
Haven't written program for quite long time, I am scratching my head for this one. I did some research first but nothing I find seems working in my case so far. So here's my task:
1) First command:
openssl s_client -crlf -quiet -connect email-smtp.us-east-1.amazonaws.com:465
which will open a separate shell(not sure if it's the correct terminology to called it shell yet that's what comes in to my mind) and output as follow'
bitnami#ip-172-31-49-138:~/script$ openssl s_client -crlf -quiet -connect email-smtp.us-east-1.amazonaws.com:465
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify error:num=20:unable to get local issuer certificate
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-1866227133 zmydXvw5oa7oYxVcuSDr
2) now enter the 2nd command: EHLO ip-172-31-49-138
which yields the following output --
bitnami#ip-172-31-49-138:~/script$ openssl s_client -crlf -quiet -connect email-smtp.us-east-1.amazonaws.com:465
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify error:num=20:unable to get local issuer certificate
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-1866227133 zmydXvw5oa7oYxVcuSDr
EHLO ip-172-31-49-138
250-email-smtp.amazonaws.com
250-8BITMIME
250-SIZE 10485760
250-AUTH PLAIN LOGIN
250 Ok
so I need to write a bash script to automate those two commands(there're more commands but for simplicity I only list the first two).
no matter I use &, or grouping {}, or insert enter line, the script will always execute the first command first, which seems waiting for something until time out, then called the 2nd command at which point the system will not recognize it because it only available while still in the shell bring up by the first command:
bitnami#ip-172-31-49-138:~/script$ ./sendMail
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify error:num=20:unable to get local issuer certificate
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-1866227133 CuOz95oNth2yafnNOxp4
421 Timeout waiting for data from client.
./sendMail: line 3: EHLO: command not found
bitnami#ip-172-31-49-138:~/script$
Hope someone can shed some light into it.
Your second "command" isn't a command at all - it's input to your first command. As such, you need to pipe it in. The following will work:
echo EHLO ip-172-31-49-138 | openssl s_client -crlf -quiet -connect email-smtp.us-east-1.amazonaws.com:465
The problem: I am trying to create a local mirror of a public FTP site. When I use lftp to do the job it creates a mirror without a problem, but when I try to update the mirror a few days later it becomes very slow due to getting stuck on several files.
Running lftp -d I can see that lftp makes several requests to RETR the file, but these requests result in several **** Timeout - reconnecting messages and after about 2-3 minutes I see
<--- 150 Opening BINARY mode data connection for {filename removed}.`
After this last command the file successfully downloads and lftp proceeds further.
From the manual I understand that BINARY is the default mode for lftp, but somehow it doesn't seem to work for the early requests. Can someone suggest how I can force lftp to always open BINARY mode data connection to download all files?
Here's a MWE:
``lftp -d -u anonymous,anonymous -c "open {url}; get {file}"``
And response from lftp -d:
---- Connecting to {url} ({IP}) port 21
<--- 220 (vsFTPd 3.0.3)
---> FEAT
<--- 211-Features:
<--- EPRT
<--- EPSV
<--- MDTM
<--- PASV
<--- REST STREAM
<--- SIZE
<--- TVFS
<--- 211 End
---> USER anonymous
<--- 331 Please specify the password.
---> PASS anonymous
<--- 230 Login successful.
---> TYPE I
<--- 200 Switching to Binary mode.
---> SIZE {file}
<--- 213 3321
---> MDTM {file}
--- 213 20160318190446
---> PASV
<--- 227 Entering Passive Mode ({IP}).
---- Connecting data socket to ({IP}) port 55380
---- Data connection established
---> RETR {file}
**** Timeout - reconnecting
---- Closing data socket
---- Closing control socket
lftp uses binary mode by default for all file transfers and ascii mode for directory listings. So the binary mode should not be a problem here.
Maybe you have a subtle connectivity problem, sometimes setting net:socket-maxseg to a lower value (e.g. 500) helps.
The ftp command for binary mode is bin so use that command before you get the file.
cd /direc/tory; bin; get file.xml
The problem is that I see
---> TYPE I
<--- 200 Switching to Binary mode.
in your output, so you're already in binary mode. I wonder if you have a different problem? I also see that you're using passive mode (PASV), and that's good because passive works around firewalls and NATs [1], so we need another reason why you see those timeouts.
Do you have any other clues, maybe from ping or netstat?
I have a one line bash command which gets me an HTML site over an SSL encrypted HTTPS connection:
echo "GET / HTTP/1.1\nHost: www.example.com\n\n" | openssl s_client -connect www.example.com:443 -quiet 2> /dev/null
The site is being loaded but with HTTP headers like:
HTTP/1.1 200 OK
Date: Fri, 01 Feb 2013 13:15:59 GMT
Server: Apache/2.2.20 (Ubuntu)
and more like this. With 2> /dev/null I can hide the output of wrong SSL certificates and more.
I do not want to take another script because curl does not what I want to do.
It is not possible due to the nature of openssl s_client which gives you the direct and plain output from a service which runs behind the connecting port (443 in my example from the question where I want to get / on a webserver with SSL).
telnet would also give you the plain output from the HTTP protocol and curl would show me the HTML site without headers and with HTTPS but does not allow self written commands to the web server.
You may use expect command to do the interaction with openssl. I'm not sure it will work but it worth a try.