I have a file named files.txt with all files which I want download.
files.txt
http://file/to/download/IC_0000.tpl
http://file/to/download/IC_0001.tpl
If I use
cat files.txt | egrep -v "(^#.*|^$)" | xargs -n 1 wget
all files are downloaded.
But I dont know how to use If files.txt contains only files without http
files.txt
IC_0000.tpl
IC_0001.tpl
I have "wget" only with this paramter:
Usage: wget [-c|--continue] [-s|--spider] [-q|--quiet] [-O|--output-document FILE]
[--header 'header: value'] [-Y|--proxy on/off] [-P DIR]
[--no-check-certificate] [-U|--user-agent AGENT] [-T SEC] URL...
Can you help me, please.
Many thanks.
Simply try wget -i files.txt (see http://www.gnu.org/software/wget/manual/wget.html#Logging-and-Input-File-Options)
If you don't have the host in the file, try:
for i in `cat files.txt`; do wget "${HOST}/${i}"; done
Just leave it here...
for MacOSX
file_name=newMP3List.txt && cur_path=$(pwd) && split -l 50 $file_name PART && find . -name "PART*" -print0 | xargs -0 -I f osascript -e "tell application \"Terminal\" to do script \"cd $cur_path && cat f | while read CMD; do curl -O \\\"\$CMD\\\"; done; rm f;\""
for linux
cat newMP3List.txt | while read CMD; do curl -O $CMD; done;
need to replace newMP3List.txt with your filename
Related
I would like to:
Find latest modified file in a folder
Change some files in the folder
Find all files modified after file of step 1
Run a script on these files from step 2
This this where I've end up:
#!/bin/bash
var=$(find /home -type f -exec stat \{} --printf="%y\n" \; |
sort -n -r |
head -n 1)
echo $var
sudo touch -d $var /home/foo
find /home/ -newer /home/foo
Can anybody help me in achieving these actions ?
Use inotifywait instead to monitor files and check for changes
inotifywait -m -q -e modify --format "%f" {Path_To__Monitored_Directory}
Also, you can make it output to file, loop over it's contents and run your script on every entry.
inotifywait -m -q -e modify --format "%f" -o {Output_File} {Path_To_Monitored_Directory}
sample output:
file1
file2
Example
We are monitoring directory named /tmp/dir which contains file1 and file2.
The following script which monitor the whole directory and echo the file name:
#!/bin/bash
while read ch
do
echo "File modified= $ch"
done < <(inotifywait -m -q -e modify --format "%f" /tmp/dir)
Run this script and modify file1 echo "123" > /tmp/dir/file1, the script will output the following:
File modified= file1
Also you can look at this stackoverflow answer
I am trying to pipe the output of fswatch to several commands in a shell script with following technique:
$ fswatch -0 [opts] [paths] | xargs -0 -n 1 -I {} [command]
Instead of [command] I put the shell script path. Here is my command line:
fswatch -0 -Ie ".*\.*$" -i ".*\.mp4$" ~/Desktop/encoding\ tests | xargs -0 -n 1 -I {} ~/Desktop/s3cmd.sh
The script is following:
#!/bin/sh
terminal-notifier -message "s3cmd Upload {}" ;
s3cmd sync --acl-public -m video/mp4 --add-header=Cache-Control:public,max-age=2052000 {} s3://saltanat-test/ &&
terminal-notifier -message "s3cmd Upload of {} done"
Sorry I am not experienced with shell scripting.
How can I pipe the the fswatch output into the script?
Thank you.
#!/bin/bash
fswatch -0 -x --event Created --event Updated --event Renamed -Ie '.*\.*$' -i '.*\.mp4$' ~/Desktop/encoding\ tests \
| xargs -0 -n 1 ~/Desktop/s3cmd-3.sh
This works. Thanks.
For anyone in the future with this question :
fswatch command:
fswatch -0 /path/to/watch | xargs -0 -n 1 -I {} ~/yourfile.sh {}
.sh file :
#!/bin/bash
echo $1
Now $1 will be equal to the output of fswatch, which is the full path to the file that was changed.
I have managed to do this separately using
grep -r "zone 19" path
mkdir zone19
find . -name "ListOfFilesfromGrep" -exec mv -i {} zone19 \;
I just don't know how to combine the two, that is, how to input the list of files I get from grep into the find command.
You should use grep from within find:
find /path/to/dir -type f -exec grep -q "zone 19" {} \; -exec mv -i {} zone19 \;
You could try
grep -lr "zone 19" path | while read in ; do mv -i "$in" zone19; done
-l prints the filenames with matched string; while ... done move the files one by one.
Using GNU versions of the standard tools:
grep -l will give you the filenames.
mv -t will move to a given directory.
xargs -r will invoke a command using arguments from stdin, but only if there's at least one.
Combine them like this:
grep -l -r -e 'zone 19' path | xargs -r mv -i -t 'zone19'
Or (if your filenames might contain newlines etc):
grep -lZr -e 'zone 19' path | xargs -0r mv -it 'zone19'
You can pipe the result from grep and use xargs:
grep -lr "zone 19" path | xargs <command>
<command> will be applied on each result of grep. Note thta -o flag tells grep to show only matching parts.
Below is the command to move all files containing string "Hello" to folder zone19.
grep Hello * |cut -f1 -d":"|sort -u|xargs -I {} mv {} zone19
I am trying to move all my video files that are in my pictures directory to my movies Directory. This is on a Mac by the way.
I thought I could simple Recurse through all my picture directories with an "ls -R"
Then I pipe that to grep -i ".avi" This give me all the movie files.
Now I pipe these values to "mv -n $1 ~/Movies" this I am hoping would move the files to the Movies folder.
I have a few Problems.
1. The "ls -R" does not list the path when listing the files. So I think I may fail to move the file.
2. I can not seem to get the file name to assign to the $1 in the mv command.
All together my command looks like this: Note I am running this from ~/Pictures
ls -R | grep -i ".avi" | mv -n $1 ~/Movies
So right now I am not sure which part is failing but I do get this error:
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
If I remove the 'mv' command I get a listing of avi files with out the path. Example Below:
4883.AVI
4884.AVI
4885.AVI
4886.AVI
4887.AVI
...
Any one have any ideas on how I can get the path in the 'ls' or how to pass a value in between the '|' commands.
Thanks.
It's better if you use the find command:
$ find -name "*.avi" -exec mv {} ~/Movies \;
you should create simple copy.sh like this
#!/bin/bash
cp $1 ~/Movies/
An run command ./copy.sh "$(ls | grep avi)"
The bash for loop can help you find all the avi files easily
shopt -s nullglob
for file in *.avi
do
mv "$file" "$file" ~/Movies/"$file"
done
you can achieve this in many ways, one of it in my openion:
ls -R | grep -i ".avi" | while read movie
do
echo " moving $movie"
mv $movie ~/Movies/
done
Use backticks
mv `ls *.avi` ~/Movies
This scripts will sort the files by date then move the first 2500 files to another directory.
When I run below scripts, system prompt out Argument list too long msg. Anyone can help me enhance the scripts ? Thanks
NUM_OF_FILES=2500
FROM_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in
DESTINATION_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in_load
if [ ! -d $DESTINATION_DIRECTORY ]
then
echo "unused_file directory does not exist!"
mkdir $DESTINATION_DIRECTORY
echo "$DESTINATION_DIRECTORY directory created!"
else
echo "$DESTINATION_DIRECTORY exist!"
fi
echo "Moving $NUM_OF_FILES oldest files to $DESTINATION_DIRECTORY directory"
ls -tr $FROM_DIRECTORY/MSCERC*.Z|head -$NUM_OF_FILES |
xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"
You didn't say, but I assume this is where the problem occurs:
ls -tr $FROM_DIRECTORY/MSCERC*.Z|head -2500 | \
xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"
(You can verify it by adding "set -x" to the top of your script.)
The problem is that the kernel has a fixed maximum size of the total length of the command line given to a new process, and your exceeding that in the ls command. You can work around it by not using globbing and instead using grep:
ls -tr $FROM_DIRECTORY/ | grep '/MSCERC\*\.Z$' |head -2500 | \
xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"
(grep uses regular expressions instead of globs, so the pattern looks a little bit different.)
Change
ls -tr $FROM_DIRECTORY/MSCERC*.Z|head -2500 | \
xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"
do something like the following:
find "$FROM_DIRECTORY" -maxdepth 1 -type f -name 'MSCERC*.Z' -printf '%p\t%T#\n' | sort -k2,2 -r | cut -f1 | head -$NUM_OF_FILES | xargs mv -t "$DESTINATION_DIRECTORY"
This uses find to create a list of files with modification timestamps, sorts by the timestamp, then removes the unneeded field before passing the output to head and xargs
EDIT
Another variant, should work with non GNU utils
find "$FROM_DIRECTORY" -type f -name 'MSCERC*.Z' -printf '%p\t%T#' |sort -k 2,2 -r | cut -f1 | head -$NUM_OF_FILES | xargs -i mv \{\} "$DESTINATION_DIRECTORY"
First of create a backup list of the files to be treated. Then read the backup file line-by-line and heal it. For example
#!/bin/bash
NUM_OF_FILES=2500
FROM_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in
DESTINATION_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in_load
if [ ! -d $DESTINATION_DIRECTORY ]
then
echo "unused_file directory does not exist!"
mkdir $DESTINATION_DIRECTORY
echo "$DESTINATION_DIRECTORY directory created!"
else
echo "$DESTINATION_DIRECTORY exist!"
fi
echo "Moving $NUM_OF_FILES oldest files to $DESTINATION_DIRECTORY directory"
ls -tr $FROM_DIRECTORY/MSCERC*.Z|head -2500 > list
exec 3<list
while read file <&3
do
mv $file $DESTINATION_DIRECTORY
done
A quick way to fix this would be to change to $FROM_DIRECTORY, so that you can refer the files using (shorter) relative paths.
cd $FROM_DIRECTORY &&
ls -tr MSCERC*.Z|head -2500 |xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"
This is also not entirely fool-proof, if you have too many files that match.