After downloading files from a remote UNIX FTP server, you want to verify that you have downloaded all the files correctly. Minimal you will get information similar to "dir /s" command in Windows command prompt. The FTP client runs on Windows.
Sadly this was written for Unix/Linux users :/
Personally, I would install CYGWIN just to get Linux binaries of LFTP/RSYNC to work on windows, as there appears not to be anything that competes with it.
As #zadok.myopenid.com
mentioned rsync, this appears to be a windows build for it using CYGWIN ( if you manage to be able to get ssh access to the box eventually )
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
Rsync is handy in that it will compare everything with check sums, and optimally transfer partial change blocks.
If you get CYGWIN/Linux:
http://lftp.yar.ru/ is my favorite exploration tool for this.
It can do almost everything bash can do, albeit remotely.
Example:
$ lftp mirror.3fl.net.au
lftp mirror.3fl.net.au:~> ls
drwxr-xr-x 14 root root 4096 Nov 27 2007 games
drwx------ 2 root root 16384 Apr 13 2006 lost+found
drwxr-xr-x 15 mirror mirror 4096 Jul 15 05:20 pub
lftp mirror.3fl.net.au:/> cd games/misc
lftp mirror.3fl.net.au:/games/misc>find
./
./dreamchess/
./dreamchess/full_game/
./dreamchess/full_game/dreamchess-0.2.0-win32.exe
./frets_on_fire/
./frets_on_fire/full_game/
./frets_on_fire/full_game/FretsOnFire-1.2.451-macosx.zip
./frets_on_fire/full_game/FretsOnFire-1.2.512-win32.zip
./frets_on_fire/full_game/FretsOnFire_ghc_mod.zip
./gametap_setup.exe
......
lftp mirror.3fl.net.au:/games/misc> du gametap_setup.exe
32442 gametap_setup.exe
lftp mirror.3fl.net.au:/games/misc> du -sh gametap_setup.exe
32M gametap_setup.exe
lftp mirror.3fl.net.au:/games/misc>
Do this :
ls -lR
..................
If you have ssh access, use rsync instead. It is a far better data transfer app.
Grab fuse for your OS and load ftpfs. This will let you mount the remote ftp directory locally and you can use dir /s or any other application you want on it.
Assuming you are using simple ftp via command line,
Use dir command with -Rl option to search recursively and copy it to a file and then search the file using grep, find or whatever way is supported on your OS.
ftp> dir -Rl education.txt
output to local-file: education.txt? y
227 Entering Passive Mode (9,62,119,15,138,239)
150 Opening ASCII mode data connection for file list
226 Transfer complete
You can use ftp.listFiles("directory") from apache-commons-net and can write your own BFS or DFS to fetch all the files recursively.
Related
I'm attempting to move multiple files on an FTP server to a different directory on the same server. So far, I've written a bash script that will login and retrieve any new files in the remote directory but the ftp command doesn't support a 'mv' command. Essentially the script would download the new file(s) and then once downloaded move the file(s) to a different directory on the same server. Please Note that the filenames will be different every time so the use of wildcards is important here.
Before you answer please note that this needs to be automated so using a GUI like Filezilla wouldn't help me as I would have to login to various ftp sites and move the files manually, also, keep in mind that I'm unable to ssh into any of the servers as they are managed by other company's and ftp access is all I'm able to get. Last thing, I won't know what the file names are so using a wildcard would be helpful.
Any help or guidance is truly appreciated.
Thank you!
Perhaps the rename command in ftp could work for you?
rename [from [to]]
Rename the file from on the remote machine, to the file to.
I gave it a bash with an old file I had sitting on a server and it seemed to do what you want:
ftp> ls tmp/test*
229 Entering Extended Passive Mode (|||55572|)
150 Accepted data connection
-rw-r--r-- 1 sinasohn sinasohn 21 Mar 31 16:37 tmp/testfile01
226-Options: -a -l
226 1 matches total
ftp> ls tmp2/test*
229 Entering Extended Passive Mode (|||64715|)
150 Accepted data connection
226-Options: -a -l
226 0 matches total
ftp> rename tmp/testfile01 tmp2/testfile01
350 RNFR accepted - file exists, ready for destination
250 File successfully renamed or moved
ftp> ls tmp/test*
229 Entering Extended Passive Mode (|||56698|)
150 Accepted data connection
226-Options: -a -l
226 0 matches total
ftp> ls tmp2/test*
229 Entering Extended Passive Mode (|||50239|)
150 Accepted data connection
-rw-r--r-- 1 sinasohn sinasohn 21 Mar 31 16:37 tmp2/testfile01
226-Options: -a -l
226 1 matches total
ftp>
I put blank lines in between commands here for clarity.
Hope this helps!
full script to achieve move more than one file
1. get file list from ftp server with mls command
2. generate to do list file
2.1 get file
2.2 rename (move file)
3. execute ftp command with to do list file
#!/bin/sh
clear
# change local directory
cd [local-directory]
#collect file names
ftp -ni ftp.abccompany.com <<EOF
user [user] [password]
cd /OUT
mls abc*.* list.txt
quit
EOF
# create ftp action list
echo >>todo.lst user [user] [password]
while read N
do
echo >>todo.lst cd /OUT
echo >>todo.lst get $N
echo >>todo.lst rename $N ARCHIVE/$N
done <list.txt
echo >>todo.lst quit
# ftp transfer process
ftp -nv ftp.abccompany.com <todo.lst
# cleanup
rm todo.lst
I am new to FTP configuration. What I am trying to do is as follows:
I am running a shell script on my localhost and downloading some files to my machine. Now I want a functionality where the files which I downloaded should be stored in a temporary directory, and then it should be transferred to a location(other directory) which I specify. I feel this mechanism is achievable by FTP communication and will be helpful when I host this on a domain, but I am not getting resources from where I can teach myself how to set this up.
OK, having visited many sites, here are some resources you might find handy:
For configuring vsftpd, here's a manual of how to install, configure and use.
About receiving many files recursively via FTP, you can use wget (extracted from this site):
cd /tmp/ftptransfer
wget --mirror --username=foo --password=bar ftp://ftp.originsite.com/path/to/folder
About sending many files recursively, many people find the only way of doing so by tar-n-send; the only problem is that the files will remain tarred until you extract them by going to the other machine (remotely or via ssh) to extract the manually. There is an alternative, not using FTP, but using ssh and pipes which lets you have files extracted on target machine:
tar -cf - /tmp/ftptransfer | ssh geek#targetsite "cd target_dir; tar -xf -"
Explained:
tar is the application to make tar files
-c: create file
-f -: file name is "stdout"
/tmp/ftptransfer: include this folder and all subdirectories in the tar
|: Make a pipe to the next program (connect stdout to stdin)
ssh: Secure Shell program
geek#targetsite: username # machinename where you want to connect to
"..." command to send to the remote host
cd target_dir: changes the dir of output
tar -xf -: extracts the file received by "stdin"
For configuring SSH on Ubuntu, have a look here.
If you need more help, don't be afraid to ask! :)
I have developed a shell script to copy the files from source to destination and simultaneously to delete the copied files in source. I can copy the files but the files cannot be deleted in source side.
files='ls filename'
for file in $files
do
ftp -vn $hostname <<EOFD
quote USER $username
quote PASS $password
binary
cd $start_dir
rm -rf $file
quit
EOFD
done
I got errors as 'No such files or directories found'
By putting ftp outside the forloop also i got error as 'invalid command'
I also tried in ssh but it prompting for username and password
files=`ls filename`
Put backticks, not simple quotes around the command to get its output.
I also tried in ssh but it prompting for username and password - check SSH-Login without password.
Scripting FTP commands using input stream directly to ftp is usually a bad idea: it lacks any error handling, it can go totally wrong and you have no chance to control it. If you have any chance to use saner command-line client, such as lftp, curl or a similar scriptable one.
Also, it's a very bad idea to iterate over files using
files=`ls files`
for file in $files
A slightly better solution is:
for file in *
but it doesn't scale: if * (or ls output) would expand more than command line buffer, it will fail. A fairly scalable solution is something like:
find . | while read file do
do_something_with $file
done
...and yet it's not probably what you want. In fact, if you just want to transfer files from source to destination and then delete files at source, you can just use lftp with mput command and -E option to delete file after transfer, or something similar with rsync --remove-source-files.
Full-proof solution:
Replace the line
`rm -rf $file`
with
`!rm -rf $file`
This is because, at that place in the code you are on the ftp console until the EOFD string is reached, so to run any command on local system(source), you need ! to be prefixed.
Best way to test is manually executing the commands. Here's what I have tested:
mtk4#laptop:~$ ftp XXX.XXX.XXX
Connected to static-XX-XX-XX-XX.XXXXXX.com.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 2 of 50 allowed.
220-Local time is now 07:52. Server port: 21.
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (XXXXXX.XXX:XXX): XXXXXXX
331 User XXXXXXX OK. Password required
Password:
230 OK. Current restricted directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> lcd test/
Local directory now /home/mtk4/test
ftp> pwd
257 "/" is your current location
ftp> !pwd
/home/mtk4/test
ftp> !ls
sample.txt
ftp> !rm sample.txt
ftp> !ls
ftp> bye
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.
mtk4#laptop:~$
Or another solution,
use the same for loop again, after the complete ftp is done to iterate over the same set of files and delete them.
i am connecting remote server via ftp and
i am sending ls -t command but it's outputting files sorted by name
how can i get last modified file via ftp ?
Note: i am connecting windows ftp server from linux machine
ls -t will give you the last modified file on top
You can confirm this by viewing with full timestamps
ls -lt
ftp -n server <<EOF|awk 'END{for(i=9;i<=NF;i++)printf "%s ",$i}'
user username password
ls -ltr
EOF
In most Unix/Linux based ftp servers, the ls command is linked to the actual ls command. This is why all the other answers are saying to use ls -t with maybe a few more parameters thrown in.
However, since you're using a Windows machine as your server, it's much harder to say how exactly the command will work. I don't believe Windows comes with a default FTP server service. I know many sites use third party FTP services on their Windows machines. It's going to depend on the software your Windows machine is using, and how it's been setup:
Try something like this:
ftp> dir /O:D
or
ftp> ls /O:D
These use the Windows parameters for the built in dir command.
try this one, it worked for me.
ls -t1 | head -1
How could I use grep and ls in FTP client...
I mean if I want to find some specific file I could use:
ls -l | grep pattern
With the usual Unix commandline interactive ftp, one approach is:
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir . foobar
output to local-file: foobar [anpqy?]? y
500 Unknown command
227 Entering Passive Mode (62,149,140,15,195,159)
150 Accepted data connection
11966 5.26 KB/s
226-Options: -a -l
226 156 matches total
ftp> !grep con foobar
-rwxr-xr-x 1 11050207 users 911007 Sep 13 2007 accu_pyconc.pdf
-rwxr-xr-x 1 11050207 users 9805405 Mar 25 2009 pycon_abst.pdf
i.e., get the dir results into a local file first, then run grep locally. Incidentally, this lets you run multiple greps after paying for just one dir data transfer;-).
lftp can, exactly the way you typed.
On Windows, you can do this with WinSCP scripting for any protocol, including the FTP:
winscp.com /command ^
"open ftp://username:password#example.com" ^
"ls /path/*.txt" ^
"exit"
References:
https://winscp.net/eng/docs/scripting
https://winscp.net/eng/docs/scriptcommand_ls
You can mount an sftp folder to a host where you have permissions to use grep or any tool.