I'm trying to download a large file over a ftp. (5GB file). Here is my script.
read ZipName
wget -c -N -q --show-progress "ftp://Password#ftp.server.com/$ZipName"
unzip $ZipName
The files downloads at 100% but never goes to the unzip command. No special error message, no outputs in the terminal. Just blank new line. I have to send CTRL + c and run back to script to unzip since wget detects that the file is fully downloaded.
Why does is hangs out like this? Is it because of the large file, or passing an argument in command?
By the way I can't use ftp because it's not on the VM i'm working on, and it's a temporary VM so no root privilege to install anything.
I've made some tests, and I think that size of the disk was the reason.
I've tried with curl -O and it worked for the same disk space.
Related
I want to download all files from a http server like:
http://dl.mysite.com/files/
and I also want to go inside each folder inside that folder.
But I do want to download only those files that have "480p" in their name.
What is the easiest solution for that using wget?
edit:
I want to have that script to be run each night from 2am to 6am to sync those files from that server to my PC.
The following wget command should work with the following flags:
wget -A "*480p*" -r -np -nc --no-check-certificate -e robots=off http://dl.mysite.com/files/
Explanation:
-A "480p" your pattern
-r, recursively recursively look through the folders
-np, --no-parent ignore links to a higher directory
-nc, --no-clobber If a file is downloaded more than once in the same directory, Wget’s behavior depends on a few options, including ‘-nc’. In certain cases, the local file will be clobbered, or overwritten, upon repeated download. In other cases it will be preserved.
--no-check-certificate Don’t check the server certificate against the available certificate authorities.
-e, --execute command A command thus invoked will be executed after the commands in .wgetrc
robots=off robot exclusion
More information on wget flags can be found at the official GNU manual page: https://www.gnu.org/software/wget/manual/wget.html
With regards to it being run once per day, you may want to read up on Cron jobs. Taken from the documentation page at: https://help.ubuntu.com/community/CronHowto
A crontab file is a simple text file containing a list of commands meant to be run at specified times. It is edited using the crontab command. The commands in the crontab file (and their run times) are checked by the cron daemon, which executes them in the system background.
So basically you need to put your wget command into a file, and set the cron to run this file at the specified time.
Note: Windows does not have a native implementation of Cron, but you can achieve the same effect using the Windows Task Scheduler.
I have a device installed remotely that has Internet access. As i cannot SSH directly to it, the device downloads updates from a .txt file located in a server. This .txt file is interpreted by the device as a sequence of bash instructions.
Now, i'm planning an update that requires re-compiling a C program in the device after downloading and overwritting some files. The content of the .txt file for this update looks like:
#!/bin/bash
curl -o /path-to-file/newfile.c http://myserver/newfile.c
sleep 10 #wait so it can download
cd /path-to-file
make
sleep 10 #wait to make
sudo /path-to-file/my-program #test if it worked
I previously tested this method and it worked as expected, but never tested make. A couple of questions:
Should it work?
Is the sleep after make necessary?
Here is an example of how to retrieve a source code file into another directory, change to that directory, compile the source code with make and then execute the resulting binary:
mkdir -p path-to-file/
curl -o path-to-file/newfile.c http://www.csit.parkland.edu/~cgraff1/src/hello_world.c
cd path-to-file/
make newfile
./newfile
The cd is really not an integral part of the process, but it seems as if the question specifically pertains to performing the work in a directory other than the present working directory.
On a cluster I zipped a large (61GB, 9.2GB when zipped) directory.
zip -r zzDirectory Directory
I then scp the zzDirectory on my personal computer.
scp -r name#host.com:/path/to/zzDirectory.zip path/in/my/computer/zzDirectory.zip
And finally I unzipped it. I tried to unzip from the bash but it failed
warning [zzDirectory.zip]: 5544449626 extra bytes at beginning or within zipfile
(attempting to process anyway)
error [zzDirectory.zip]: start of central directory not found;
zipfile corrupt.
(please check that you have transferred or created the zipfile in the
appropriate BINARY mode and that you have compiled UnZip properly)
So I doubled click on the icon from the finder and the system started to unzip zzDirectory.zip. However, some files are missing and it looks like (I am not 100% sure yet) that some newline characters (\n) are missing as well. unzip used to work fine on my computer before.
In order to investigate where the problem come from, I unzipped zzDirectory.zip on the cluster and everything seem to work fine (no missing files).
I repeated the transfer and unzipped again but the problem persists. Note that transfers are made via internet. My OS is Mac OSX Yosemite 10.10.2.
How can I solve this issue? I would prefer not to transfer data that are not zipped because of band width issue. Do you think I should try to tar or should I use specific options that goes with the unzip command line?
On OS X you could try:
ditto -x -k the_over4gb.zip /path/to/dir/where/want/unzip
e.g:
ditto -x -k zzDirectory.zip .
I keep seeing this phrase "download or create" in tutorials right after tofrodos. What would be an example of how to download or create? I just get stuck in
^
^
^
mode.
apt-get -y install tofrodos
Download or create ZPX_ubuntu_12-04_auto_installer.sh
The author means you need to transfer the "ZPX_ubuntu_12-04_auto_installer.sh" shell script to the server, and if necessary, change to the directory you saved the script in using the cd command, before entering the commands below:
fromdos ZPX_ubuntu_12-04_auto_installer.sh
chmod +x ZPX_ubuntu_12-04_auto_installer.sh
./ZPX_ubuntu_12-04_auto_installer.sh
He seems to be referring to a particular script included in a .zip file posted on a web forum.
You may be able to download the .zip file to your computer, extract it, and then use the sftp or scp program to transfer just the shell script to the server. Alternatively, you could use wget or curl to download the .zip file to the server and then the unzip command to unzip it.
A graphical SFTP client like FileZilla may help for the former approach. The Firefox add-on cliget may help for the latter, especially because the file is hosted on a password-protected web forum.
fromdos is just a utility program to convert a text file from DOS/Windows format to Unix format by stripping out all the carriage return characters. Perhaps using this command is necessary because the author of the script used a Windows text editor or an ASCII-mode FTP transfer before zipping up the file. Of course, you need the file on the server if you are trying to run the command on the server.
I created this simple script that does a backup, I wrote and tested it in Linux, then I copied it in my WebApp WEB-INF/scripts directory so that I could be run via Java Runtime.exec().
#!/bin/bash
JACCISE_FOLDER="/var/jaccise"
rm $JACCISE_FOLDER/jaccisebackup.zip
zip -r jaccisefolder.zip $JACCISE_FOLDER
mysqldump -ujacc -pxxx jacciseweb > jaccisewebdump.sql
zip jaccisebackup.zip jaccisewebdump.sql
zip jaccisebackup.zip jaccisefolder.zip
rm jaccisewebdump.sql
rm jaccisefolder.zip
cp jaccisebackup.zip $JACCISE_FOLDER
But it doesn't. So I tried to copy it from WEB-INF/scripts to my user dir and run it to roubleshoot it. The result is that it comes out with: ": File o directory non esistente" (Means "Unknown file or directory" notice the colon at the beginning). I created another file from scratch, copied and pasted the whole script and it works. I may think that this is related to:
Text encoding
\n\r differences between windows (I use Eclipse on windows to edit everything) and Linux.
How do I solve this deploy problem?
You should check if the file is executable (chmod +x). Then you should check, if your web server allows the execution of external programs. This might be a security problem and it is likely that the web server prevents the execution. Check the logs of the web server. The encoding of the file can be changed with the dos2unix command. In order to debug your script you can add an "set -x" at the beginning, but I think the script does not start at all.