I would like to create thumbnails of images I have organised into a set of nested subdirectories into a mirror of the file structure so that a command of the type:
./imageresize.sh Large Small 10
...would convert any .jpg or .JPG files, in the directories nested under "./Large":
./Large/Holidays/001.jpg
./Large/Holidays/002.jpg
./Large/Holidays/003.jpg
./Large/Pets/Dog/001.jpg
./Large/Pets/Dog/002.jpg
./Large/Pets/Cat/001.jpg
into thumbnails of 10% to a mirror destination with a different top directory ("Small" instead of "Large" in this e.g.):
./Small/Holidays/001.jpg
./Small/Holidays/002.jpg
./Small/Holidays/003.jpg
./Small/Pets/Dog/001.jpg
./Small/Pets/Dog/002.jpg
./Small/Pets/Cat/001.jpg
This is what I have so far, but I can't seem to get it working. The $newfile variable seems invalid but I don't know why, and when testing, it outputs the result of the 'convert' command to the screen. Any help/suggestions greatly appreciated.
#!/bin/bash
#manage whitespace and escape characters
OIFS="$IFS"
IFS=$'\n'
#create file list
filelist=$(find ./$1/ -name "*.jpg" -o -name "*.JPG")
for file in $filelist
do
#create destination path for 'convert' command
newfile= ${file/$1/$2}
convert "$file" -define jpeg:extent=10kb -scale $3% "$newfile"
done
Don't know if it's just a copy/paste error or if it's actually in your script, but this line:
newfile= ${file/$1/$2}
would be an invalid assignment in Bash as spaces around = isn't allowed when assigning.
Try this instead:
newfile=${file/$1/$2}
As a side-note. find has a case-insensitive search too, -iname, so you could do:
filelist=$(find ./$1/ -iname "*.jpg")
It also has -exec for executing commands on the result set. It's explained very well here: https://unix.stackexchange.com/questions/12902/how-to-run-find-exec So you may be able to do it in one single find. (Note: That's not necessarily better, in most cases it's just a matter of preference, I just mention it as a possibility.)
So - revised working script with corrections suggested by madsen and 4ae1e1 (thanks both), and with rsync commands to create directory structure first, (then clean extraneous files from the destination) :). I've added an extra parameter and parameter checker so that now you can specify source, destination, approx destination file size in kb and percentage of original. Hope it helps someone else. :)
#!/bin/bash
#manage whitespace and escape characters
OIFS="$IFS"
IFS=$'\n'
#check parameters
if [ $# -lt 4 ] || [ "$1" = "--help" ]
then # if no parameters or '--help' as $1 - show help
echo "______________________________________________________________"
echo ""
echo "Useage: thumbnailmirror [Source] [Destination] [Filesize] [Percentage]"
echo "Source - e.g. 'Fullsize' (directory must exist)"
echo "Destination - e.g. 'Thumnail' (directory must exist)"
echo "Filesize - approx filesize in kb e.g. '10'"
echo "Percentage - % of reduction (1-100)"
echo "e.g. thumbnailmirror Fullsize Thumnail 18 7"
echo "______________________________________________________________"
else # parameters exist
#syncronise directory structure (directories only)
rsync -a --include '*/' --exclude '*' ./$1/ ./$2/
# delete any extraneous files and directories at destination
rsync -a --delete --existing --ignore-existing ./$1/ ./$2/
#create file list ( -iname means not case sensitive)
filelist=$(find ./$1/ -iname "*.jpg")
for file in $filelist
do
#define destination filename for 'convert' command
newfile=${file/$1/$2}
if [ ! -f "$newfile" ] #if file doesn't exists create it
then
convert "$file" -define jpeg:extent=$3kb -quality 100 -scale $4% "$newfile"
echo "$file resized"
else #skip it
echo "Skipping $file - exists already"
fi
done
fi
Related
I have a large set of working files of the form a.mp4 b.txt c.avi d.doc etc (the extension is irrelevant to the question). I also have a set of files which include the same named files except with a common specific prefix "broken_" eg broken_a.mp4, broken_b.txt
If I have a.mp4 and broken_a.mp4, I want to move the broken_a.mp4 to a holding directory. If I have broken_d.mp4 but no matching d.mp4, then leave it alone.
I have some code successfully used to identify and move files with the same extension which I'd like to modify
This is the form of working example code for same extension files (kudos to the original author) which I'd like to modify if possible to do the job
#!/bin/bash
# Name of source directory
SOURCE_DIR=.
# Name of destination directory
DEST_DIR=already_converted_m4v
# Create the destination directory for the moved files, if it doesn't already exist.
[ ! -d $DEST_DIR ] && mkdir -p $DEST_DIR
find $SOURCE_DIR -maxdepth 1 -type f -iname "*.avi" | while read fin
do
#echo "m4v doing avi"
fm4v=${fin/.avi/.m4v}
[ -f "$fm4v" ] && gmv -v --backup=numbered "$fin" $DEST_DIR/
done
My garbage first attempt which clearly doesnt work looks horribly like:
#!/bin/bash
# Name of source directory
SOURCE_DIR=.
# Name of destination directory
DEST_DIR=Already_broken
# Create the destination directory for the moved files, if it doesn't already exist.
[ ! -d $DEST_DIR ] && mkdir -p $DEST_DIR
find $SOURCE_DIR -maxdepth 1 -type f -iname "*" | while read fin
do
#echo "working to find existing broken and unbroken files"
filetest_basename=$(basename "$fin" )
filetest_extension=$(extension "$fin" )
echo $filetest_basename
echo $filetest_extension
fileok=${filetest_basename/!broken_/broken_}
[ -f "$fileok" ] && gmv -v --backup=numbered "$fin" $DEST_DIR/
done
Grateful for help
find is irrelevant here, a simple shell loop would suffice:
SRCDIR='.'
DSTDIR='Already_broken'
if ! [ -d "$DSTDIR" ]; then
mkdir -p -- "$DSTDIR"
fi
for broken in "$SRCDIR"/broken_*; do
if [ -f "${broken%"${broken##*/}"}${broken##*/broken_}" ]; then
echo gmv -v --backup=numbered "$broken" -- "$DSTDIR"
fi
done
If its output looks good, remove echo.
Some notes:
We can't use ${broken/broken_} here for SRCDIR might contain broken_ in the future.
Nested PE (${broken##*/}) needs to be quoted for its result might contain metacharacters and that would bring about undesired results.
I am working on Linux machine.
I have a lot of files named the same, with a directory structure like this:
P45_input_foo/result.dat
P45_input_bar/result.dat
P45_input_tar/result.dat
P45_input_cool/result.dat ...
It is difficult to copy them one by one. I want to copy them into another folder named as data with similar folder names and file names:
/data/foo/result.dat
/data/bar/result.dat
/data/tar/result.dat
/data/cool/result.dat ...
In stead of copy them one by one what I should do?
Using a for loop in bash :
# we list every files following the pattern : ./<somedirname>/<any file>
# if you want to specify a format for the folders, you could change it here
# i.e. for your case you could write 'for f in P45*/*' to only match folders starting by P45
for f in */*
do
# we strip the path of the file from its filename
# i.e. 'P45_input_foo/result.dat' will become 'P45_input_foo'
newpath="${f%/*}"
# mkdir -p /data/${newpath##*_} will create our new data structure
# - /data/${newpath##*_} extract the last chain of character after a _, in our example, 'foo'
# - mkdir -p will recursively create our structure
# - cp "$f" "$_" will copy the file to our new directory. It will not launch if mkdir returns an error
mkdir -p /data/${newpath##*_} && cp "$f" "$_"
done
the ${newpath##*_} and ${f%/*} usage are part of Bash string manipulation methods. You can read more about it here.
You will need to extract the 3rd item after "_" :
P45_input_foo --> foo
create the directory (if needed) and copy the file to it. Something like this (not tested, might need editing):
STARTING_DIR="/"
cd "$STARTING_DIR"
VAR=$(ls -1)
while read DIR; do
TARGET_DIR=$(echo "$DIR" | cut -d'_' -f3)
NEW_DIR="/data/$DIR"
if [ ! -d "$NEW_DIR" ]; then
mkdir "$NEW_DIR"
fi
cp "$DIR/result.dat" "$NEW_DIR/result.dat"
if [ $? -ne 0 ];
echo "ERROR: encountered an error while copying"
fi
done <<<"$VAR"
Explanation: assuming all the paths you've mentioned are under root / (if not change STARTING_PATH accordingly). With ls you get the list of the directories, store the output in VAR. Pass the content of VAR to the while loop.
A bit of find and with a few bash tricks, the below script could do the trick for you. Remember to run the script without the mv and see if "/data/"$folder"/" is the actual path that you want to move the file(s).
#!/bin/bash
while IFS= read -r -d '' file
do
fileNew="${file%/*}" # Everything before the last '\'
fileNew="${fileNew#*/}" # Everything after the last '\'
IFS="_" read _ _ folder <<<"$fileNew"
mv -v "$file" "/data/"$folder"/"
done < <(find . -type f -name "result.dat" -print0)
Currently learning some bash scripting and having an issue with a question involving listing all files in a given directory and stating if they are a file or directory. The issue I am having is that I only get either my current directory or if a specify a directory it will just say that it is a directory eg. /home/user/shell_scripts will return shell_scipts is a directory rather than the files contained within it.
This is what I have so far:
dir=$dir
for file in $dir; do
if [[ -d $file ]]; then
echo "$file is a directory"
if [[ -f $file ]]; then
echo "$file is a regular file"
fi
done
Your line:
for file in $dir; do
will expand $dir just to a single directory string. What you need to do is expand that to a list of files in the directory. You could do this using the following:
for file in "${dir}/"* ; do
This will expand the "${dir}/"* section into a name-only list of the current directory. As Biffen points out, this should guarantee that the file list wont end up with split partial file names in file if any of them contain whitespace.
If you want to recurse into the directories in dir then using find might be a better approach. Simply use:
for file in $( find ${dir} ); do
Note that while simple, this will not handle files or directories with spaces in them. Because of this, I would be tempted to drop the loop and generate the output in one go. This might be slightly different than what you want, but is likely to be easier to read and a lot more efficient, especially with large numbers of files. For example, To list all the directories:
find ${dir} -maxdepth 1 -type d
and to list the files:
find ${dir} -maxdepth 1 -type f
if you want to iterate into directories below, then remove the -maxdepth 1
This is a good use for globbing:
for file in "$dir/"*
do
[[ -d "$file" ]] && echo "$file is a directory"
[[ -f "$file" ]] && echo "$file is a regular file"
done
This will work even if files in $dir have special characters in their names, such as spaces, asterisks and even newlines.
Also note that variables should be quoted ("$file"). But * must not be quoted. And I removed dir=$dir since it doesn't do anything (except break when $dir contains special characters).
ls -F ~ | \
sed 's#.*/$#/& is a Directory#;t quit;s#.*#/& is a File#;:quit;s/[*/=>#|] / /'
The -F "classify" switch appends a "/" if a file is a directory. The sed code prints the desired message, then removes the suffix.
for file in $(ls $dir)
do
[ -f $file ] && echo "$file is File"
[ -d $file ] && echo "$file is Directory"
done
or replace the
$(ls $dir)
with
`ls $`
If you want to list files that also start with . use:
for file in "${dir}/"* "${dir}/"/.[!.]* "${dir}/"/..?* ; do
I have several folders, each with between 15,000 and 40,000 photos. I want each of these to be split into sub folders - each with 2,000 files in them.
What is a quick way to do this that will create each folder I need on the go and move all the files?
Currently I can only find how to move the first x items in a folder into a pre-existing directory. In order to use this on a folder with 20,000 items... I would need to create 10 folders manually, and run the command 10 times.
ls -1 | sort -n | head -2000| xargs -i mv "{}" /folder/
I tried putting it in a for-loop, but am having trouble getting it to make folders properly with mkdir. Even after I get around that, I need the program to only create folders for every 20th file (start of a new group). It wants to make a new folder for each file.
So... how can I easily move a large number of files into folders of an arbitrary number of files in each one?
Any help would be very... well... helpful!
Try something like this:
for i in `seq 1 20`; do mkdir -p "folder$i"; find . -type f -maxdepth 1 | head -n 2000 | xargs -i mv "{}" "folder$i"; done
Full script version:
#!/bin/bash
dir_size=2000
dir_name="folder"
n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1))
for i in `seq 1 $n`;
do
mkdir -p "$dir_name$i";
find . -maxdepth 1 -type f | head -n $dir_size | xargs -i mv "{}" "$dir_name$i"
done
For dummies:
create a new file: vim split_files.sh
update the dir_size and dir_name values to match your desires
note that the dir_name will have a number appended
navigate into the desired folder: cd my_folder
run the script: sh ../split_files.sh
This solution worked for me on MacOS:
i=0; for f in *; do d=dir_$(printf %03d $((i/100+1))); mkdir -p $d; mv "$f" $d; let i++; done
It creates subfolders of 100 elements each.
This solution can handle names with whitespace and wildcards and can be easily extended to support less straightforward tree structures. It will look for files in all direct subdirectories of the working directory and sort them into new subdirectories of those. New directories will be named 0, 1, etc.:
#!/bin/bash
maxfilesperdir=20
# loop through all top level directories:
while IFS= read -r -d $'\0' topleveldir
do
# enter top level subdirectory:
cd "$topleveldir"
declare -i filecount=0 # number of moved files per dir
declare -i dircount=0 # number of subdirs created per top level dir
# loop through all files in that directory and below
while IFS= read -r -d $'\0' filename
do
# whenever file counter is 0, make a new dir:
if [ "$filecount" -eq 0 ]
then
mkdir "$dircount"
fi
# move the file into the current dir:
mv "$filename" "${dircount}/"
filecount+=1
# whenever our file counter reaches its maximum, reset it, and
# increase dir counter:
if [ "$filecount" -ge "$maxfilesperdir" ]
then
dircount+=1
filecount=0
fi
done < <(find -type f -print0)
# go back to top level:
cd ..
done < <(find -mindepth 1 -maxdepth 1 -type d -print0)
The find -print0/read combination with process substitution has been stolen from another question.
It should be noted that simple globbing can handle all kinds of strange directory and file names as well. It is however not easily extensible for multiple levels of directories.
The code below assumes that the filenames do not contain linefeeds, spaces, tabs, single quotes, double quotes, or backslashes, and that filenames do not start with a dash. It also assumes that IFS has not been changed, because it uses while read instead of while IFS= read, and because variables are not quoted. Add setopt shwordsplit in Zsh.
i=1;while read l;do mkdir $i;mv $l $((i++));done< <(ls|xargs -n2000)
The code below assumes that filenames do not contain linefeeds and that they do not start with a dash. -n2000 takes 2000 arguments at a time and {#} is the sequence number of the job. Replace {#} with '{=$_=sprintf("%04d",$job->seq())=}' to pad numbers to four digits.
ls|parallel -n2000 mkdir {#}\;mv {} {#}
The command below assumes that filenames do not contain linefeeds. It uses the implementation of rename by Aristotle Pagaltzis which is the rename formula in Homebrew, where -p is needed to create directories, where --stdin is needed to get paths from STDIN, and where $N is the number of the file. In other implementations you can use $. or ++$::i instead of $N.
ls|rename --stdin -p 's,^,1+int(($N-1)/2000)."/",e'
I would go with something like this:
#!/bin/bash
# outnum generates the name of the output directory
outnum=1
# n is the number of files we have moved
n=0
# Go through all JPG files in the current directory
for f in *.jpg; do
# Create new output directory if first of new batch of 2000
if [ $n -eq 0 ]; then
outdir=folder$outnum
mkdir $outdir
((outnum++))
fi
# Move the file to the new subdirectory
mv "$f" "$outdir"
# Count how many we have moved to there
((n++))
# Start a new output directory if we have sent 2000
[ $n -eq 2000 ] && n=0
done
The answer above is very useful, but there is a very import point in Mac(10.13.6) terminal. Because xargs "-i" argument is not available, I have change the command from above to below.
ls -1 | sort -n | head -2000| xargs -I '{}' mv {} /folder/
Then, I use the below shell script(reference tmp's answer)
#!/bin/bash
dir_size=500
dir_name="folder"
n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1))
for i in `seq 1 $n`;
do
mkdir -p "$dir_name$i";
find . -maxdepth 1 -type f | head -n $dir_size | xargs -I '{}' mv {} "$dir_name$i"
done
This is a tweak of Mark Setchell's
Usage:
bash splitfiles.bash $PWD/directoryoffiles splitsize
It doesn't require the script to be located in the same dir as the files for splitting, it will operate on all files, not just the .jpg and allows you to specify the split size as an argument.
#!/bin/bash
# outnum generates the name of the output directory
outnum=1
# n is the number of files we have moved
n=0
if [ "$#" -ne 2 ]; then
echo Wrong number of args
echo Usage: bash splitfiles.bash $PWD/directoryoffiles splitsize
exit 1
fi
# Go through all files in the specified directory
for f in $1/*; do
# Create new output directory if first of new batch
if [ $n -eq 0 ]; then
outdir=$1/$outnum
mkdir $outdir
((outnum++))
fi
# Move the file to the new subdirectory
mv "$f" "$outdir"
# Count how many we have moved to there
((n++))
# Start a new output directory if current new dir is full
[ $n -eq $2 ] && n=0
done
Can be directly run in the terminal
i=0;
for f in *;
do
d=picture_$(printf %03d $((i/2000+1)));
mkdir -p $d;
mv "$f" $d;
let i++;
done
This script will move all files within the current directory into picture_001, picture_002... and so on. Each newly created folder will contain 2000 files
2000 is the chunked number
%03d is the suffix digit you can adjust (currently 001,002,003)
picture_ is the folder prefix
This script will chunk all files into its directory (create subdirectory)
You'll certainly have to write a script for that.
Hints of things to include in your script:
First count the number of files within your source directory
NBFiles=$(find . -type f -name *.jpg | wc -l)
Divide this count by 2000 and add 1, to determine number of directories to create
NBDIR=$(( $NBFILES / 2000 + 1 ))
Finally loop through your files and move them accross the subdirs.
You'll have to use two imbricated loops : one to pick and create the destination directory, the other to move 2000 files in this subdir, then create next subdir and move the next 2000 files to the new one, etc...
I have a directory with sub-directories and files with names that start with a string similar to the sub-directories; e.g.
bar/
foo-1/ (dir)
foo-1-001.txt
foo-1-002.txt
foo-1-003.txt
foo-2/ (dir)
foo-2-001.txt
foo-2-002.txt
foo-2-003.txt
foo-3/ (dir)
foo-3-001.txt
foo-3-002.txt
foo-3-003.txt
etc.
All files are currently at the same level. I'd like to move the corresponding .txt files into their similarly-named directories with a script (there are > 9500 in my current situation).
I've written the following, but I'm missing something, as I can't get the files to move.
#!/bin/sh
# directory basename processing for derivatives
# create directory list in a text file
find ./ -type d > directoryList.txt
# setup while loop for moving text files around
FILE="directoryList.txt"
exec 3<&0
exec 0<$FILE
while read line
do
echo "This is a directory:`basename $line`"
filemoves=`find ./ -type f -name '*.txt' \! -name 'directoryList.txt' | sed 's|-[0-9]\{3\}\.txt$||g'`
if [ "`basename $filemoves`" == "$line" ]
then
cp $filemoves $line/
echo "copied $filemoves to $line"
fi
done
exec 0<&3
Things seem to work OK until I get to the if. I'm working across a number of *nix, so I have to be careful what arguments I'm throwing around (RHEL, FreeBSD, and possibly Mac OS X, too).
Assuming your files really match the pattern above (everything before the last dash is the directory name) this should do it:
for thefile in *.txt ; do mv -v $thefile ${thefile%-*}; done
and if it tells you command line too long (expanding *.txt into 4900 files is a lot) try this:
find . -name '*.txt' | while read thefile ; do mv -v $thefile ${thefile%-*} ; done
I'm not a shell script expert but I'm aware that in many shells (and according to this page on the internet: http://www.vectorsite.net/tsshell.html this includes SH), string comparison is done with the "=" operator, not "==".
[ "$shvar" = "fox" ] String comparison, true if match.
[code block removed]
Reason 1. Used ls instead of globbing
Reason 2. Used mv $VAR1 $VAR2 style moving without quoting variables