FTP get and delete multiple files - ftp

Question

I have to get and delete multiple files via FTP so I wrote this script:
open ftp.myftpserver.org
user
pass
cd folder
lcd E:\localdir
mget *
mdel *
bye
This works but is not safe since the folder is being fed from other sources and the mdel * step may delete files uploaded in the meanwhile.
I guess a solution could be moving the files remotely to a different folder, building a filelist at the beginning of the process, but i have no idea how to make it.
Is it possibile?

Answer 1

FTR I followed the nice hint and I managed to succesfully made something working, maybe not elegant but works:
First step to get the file list:
getfilelist.bat
open ftp.myserver.it
myuser
pass1234
cd ftpfolder
prompt n
lcd E:\localdir
ls *.??? filelist.txt
bye
Second step to download and delete the above files
movefiles.bat
#echo off
setlocal enableextensions
setlocal enabledelayedexpansion
echo open ftp.myserver.it>>myscript
echo user myuser pass1234>>myscript
echo cd ftpfolder>>myscript
echo prompt n>>myscript
echo ascii>>myscript
echo lcd E:\downloaddir>>myscript
for /F "usebackq tokens=1,2* delims=," %%G IN ("E:\localdir\filelist.txt") DO ECHO rename %%G %%G_TMP>>myscript
echo mget *_TMP>>myscript
echo mdelete *_TMP>>myscript
echo bye>>myscript
ftp -n -s:myscript
del filelist.txt
del myscript
e:
cd E:\downloaddir
ren *.???_TMP *.???
A bat file to recall the above steps:
E:
cd E:\localdir
ftp -i -s:E:\localdir\getfilelist.bat
E:\localdir\movefiles.bat
hope it helps

Answer 2

I had a similar problem. It looks a lot of people struggle here ;-)
I have to download multiple files and remove them after successfully downloading them on the remote Server to avoid double processing. There is a very small chance that while doing an mget the remote System adds further files. So a mdelete might delete untransferred files.
My approach was not to use anything else than FTP commands. Here we go:
cd <dir on remote server>
mkdir Transfer
mv *.* Transfer
mget Transfer\*.*
del Transfer\*.*
rmdir Transfer
At a glance I move all files to an extra directory on the remote server. Should be very fast as this happens locally. Once the files are shifted to the transfer dir, I perform the real mget download. This might take longer and meanwhile further files can be safely uploaded on the remote Server main dir.
After the download is done, I delete all files in the transfer dir and remove the dir itself.

Answer 3

Don't know if you can do that with a script from with ftp client.
Might be better to do it as a program or scipt using a language of your choice with an FTP library so you have much more control of the FTP operations. e.g. perl with Net::ftp, java etc.
You could then implement a algorithum like:
remote cd to required folder
localcd to required folder
list current files in remote folder
for each file in list that matchs required pattern
get file
if get ok then
delete file
else
log error,exit or whatever error handling you want
endif
endfor
Also need to make sure you don't try to get a file that is in the process of being written, depending on o/s this might be handled for you with file locks, but you need to make sure files are written to the remote dir in a two stage process. First too a temporay directory or file name that does not match the file pattern you are checking for, then renamed or moved into the correct location and name that you will detect.

Answer 4

ok, then i think you can implement the algorithum i've described as a batch script using two seperate ftp calls and scripts. The first to list the files to be transfered from the remote dir, the second to get and delete a single file from the list.
The file get script would have to be created dynamically on each loop iteration to get the next file.
cd to working dir
Run FTP with script to:
cd to remote folder
list files matching required pattern and store in local file
(e.g: ls files_*.txt filelist.txt)
for each file in file list created above (e.g. use 'for /f ...' command to loop through filelist.txt)
create temp ftp script to:
cd to remote dir
lcd to local dir
get file
del file
run FTP with temp script
endfor
This site has an example of doing similer (note script as shown dosn't work, comments give details of corrections needed).
http://www.computing.net/answers/programming/batch-ftp-script-list-rename-get-delete/25728.html