I searched a lot and googled a lot too but I didn't find anything...
I'm using bash on linux...
I have to download a certain folder from a ftp server (i know ftp is deprecated, can't use ftps or sftp right now btw i'm in a local network).
I want to do a sort of integrity check of the downloaded folder, which has a lot of subfolders and files, so i choose to compare folder size as a test.
I'm downloading through wget but my question is...how can I check the folder size BEFORE downloading it so that i can store the size in a file and then compare with the downloaded one? In ftp, so...
I tried with a simply curl to the parent directory but there is no size information there...
Thanks for the help!!
wget will recursively download all directory content one at a time. There's no way to determine the size of an entire directory using ftp. Though it's possible using ssh.
I recommend installing ssh server on your machine and after gaining an access, you can use the following command to get the size of the desired directory:
du -h desired_directory | tail -n 1
I do not recommend this method though, it's more reliable to get the hash checksum of the remote content and compare them with your downloaded content. It's far more reliable and it's already used by many download clients to check the integrity of the files.
It basically depends on what your ftp client and your ftp server can do. With some I know, the default ls does the job and they even have a size command:
ftp> help size
size show size of remote file
ftp> size foo
213 305
ftp> ls foo
200 PORT command successful.
150 Opening ASCII mode data connection for file list
-rw-r--r-- 1 foo bar 305 Aug 1 2013 foo
226 Transfer complete.
You can't check folder size via protocol FTP. Try to connect remote folder on the main server via curlftpfs (if it possible).
Related
Im trying to copy a zip file located on a server by a ssh2 library.
the way i'm about to do is using less command and write it down on client side.
Less -r -L -f zipfile
but the output file is bigger than the original.
i know this not a good practice but i have to.
so how can i handle this to have my zip file on the client machine?
Is Less an mandatory command to do that ?
You can simply use scp to achieve that by providing user and host and then typing the directory, where to copy the file from the server to local host, like on the example below:
scp your_username#remotehost.edu:foobar.txt /some/local/directory
I am trying to mirror a public FTP to a local directory. When I use wget -m {url} then wget quite quickly skips lots of files that have been already downloaded (and no newer version exists), when I use lftp open -u user,pass {url}; mirror then lftp sends MDTM for every file before deciding whether to download the file or not. With 2 million+ files in 50 thousand+ directories this is very slow, besides I get error messages that MDTM of directories could not be obtained.
In the manual it says that using set sync-mode off will result in sending all requests at once, so that lftp doesn't wait for each response. When I do that, I get error messages from the server saying there are too many connections from my IP address.
I tried running wget first to download only the newer files, but this does not delete the files which were removed from the FTP server, so I follow up with lftp to remove the old files, however lftp still sends MDTM on each file, which means that there is no advantage to this approach.
If I use set ftp:use-mdtm off, then it seems that lftp just downloads all files again.
Could someone suggest the correct setting for lftp with large number of directories/files (specifically, so that it skips directories which were not updated, like wget seems to do)?
Use set ftp:use-mdtm off and mirror --ignore-time for the first invocation to avoid re-downloading all the files.
You can also try to upgrade lftp and/or use set ftp:use-mlsd on, in this case lftp will get precise file modification time from the MLSD command output (provided that the server supports the command).
I'm using a (cheap branded) local media station as an FTP server and I'm using FIleZilla to transfer files to it.
When I try to move or rename a file located on the media station, I'm getting
Command: RNFR [filename]
Response: 503 Command not understood.
I don't know whether this is because of an old or corrupted FTP version (it's a device older than 5 years and I think there are no updates available).
Is there an alternative to perform FTP rename or move commands?
Is there an alternative to perform FTP rename or move commands?
If you have telnet or SSH access to the machine you could do the renaming their. If not you might try to use the FTP SITE command with "mv from-name to-name". But I doubt that the server will support this if it does not even support the standard way of FTP to rename files.
Apart from that the only alternative is probably to download the file, remove it on the server and upload it again with a different name.
Usually I use rsync based backup.
But now I have to make backup script from Windows server to linux.
So, there is no rsync - only FTP.
I like ideas of hard links using to save disk space and incremental backup to minimize traffic.
Is there any similar backup script for ftp instead of rsync?
UPDATE:
I need to backup Windows server through FTP. Backup script executes at Linux backup server.
SOLUTION:
I found this useful script to backup through FTP with hard links and incremental feature.
Note for Ubuntu users: there is no md5 command in Ubuntu. Use md5sum instead.
# filehash1="$(md5 -q "$curfile"".gz")"
# filehash2="$(md5 -q "$mysqltmpfile")"
filehash1="$(md5sum "$curfile"".gz" | awk '{ print $1 }')"
filehash2="$(md5sum "$mysqltmpfile" | awk '{ print $1 }')"
Edit, since the setup was not clear enough for me from the original question.
Based on the update of the question the situation is, that you need to pull the data on the backup server from the windows system via ftp. In this case you could adapt the script you find yourself (see comment) or use a similar idea like:
Use cp -lr to clone the previous backup with hard links.
Use lftp --mirror to overwrite this copy with anything which got updated on the remote system.
But I assumed initially that you need to push the data from the windows system to the backup server, that is the FTP server is on the backup system. This case can not handled this way (original answer follows):
Since FTP has no idea of links at all any transfers will only result in new or overwritten files. The only way would be to using the SITE command to issue site specific commands and deal this way with hard links. But site specific commands are usually restricted heavily so that you can do something like change permissions but not do anything with hard links.
And even if you could support hard links with SITE you have to implement the logic which decides when to use such links. With rsync this logic is built into the rsync server and executed on the server site. With FTP you have to built all the logic at the client site, which means that you would have to download a file to compare it with a local file and then decide if you would need to upload the new file or if a hard link to an existing file could be used.
I want to list images by SFTP and save this list, so another script may further process it.
Unfortunately, there are also many other files there, so I need to identify which are images. I am filtering out everything with wrong file extension, but I would like to go a step further and check also the content of the file.
Downloading everything to check it with file --mime-type on local machine is too slow. Is there a way how to check MIME type of a file on remote SFTP before the download?
We found a way, downloading only first 64 bytes. It is a lot faster than downloading whole file, but still enough to see if it looks like an image:
curl "sftp://sftp.example.com/path/to/file.png" -u login:pass -o img.tmp -r 0-64
file --mime-type img.tmp
MIME type is supported by SFTP version 6 and newer only.
Most SFTP clients and servers, including the most widespread one, OpenSSH, support SFTP version 3 only.
Even the servers that I know of to support SFTP version 6, like Bitvise or ProFTPD mod_sftp, do not support the "MIME type" attribute.
So while in theory it's possible to determine MIME type of remote files over SFTP, in practice, you won't be able to do it.
You can run any command remotely using ssh:
ssh <destination> '<command_to_run>'
In this case that would be something like:
ssh <remote_machine_name_or_ip_address> 'file --mime-type ./*'