Mass rename folders and their files in bash/terminal - bash

I am trying to mass rename folders and their respective files by use of the command "mv" but its not going very well, I would like the folders to be renamed into:
1/
2/
3/
etc.
The files in each folder should in this process also be renamed while keeping their file extension.
1.png
2.gif
3.jpg
etc
Very thankful for your help

EDIT: Better approach with a function rather than recursive calls to the same script.
Here's hoping I got all corner cases. This descends into directories recursively to handle deeply nested directory trees.
Caveat: Because the script takes care to not overwrite existing files, gaps may appear in the numbering in some corner cases -- if there is a file 0.txt in a directory and the first file that is handled in that directory is a .txt file, it will be moved to 1.txt. This also happens if the first file that is handled is 0.txt, so running the script twice will change the numbering, and running it again will change it back.
So here's the code:
#!/bin/bash
handle_directory() {
local counter=0
for i in *; do
# if the file is a directory (but not a symlink),
# handle it before moving
if [ -d "$i" ] && ! [ -h "$i" ]; then
cd "$i"
handle_directory
cd ..
fi
# extract suffix
suffix="${i##*.}"
if [ "$suffix" != "$i" ]; then
extension=".$suffix"
else
# If there is no filename extension, the counter
# is the whole filename. Without this, we'd get
# 0.Makefile and suchlike.
extension=""
fi
# find a filename that isn't already taken
# this may lead to gaps in the numbering.
while dest="$counter$extension" && [ -e "$dest" ]; do
let ++counter
done
echo mv "$i" "$dest"
let ++counter
done
}
# if a parameter was given, go there to handle it.
# otherwise handle the local directory.
if ! [ -z "$1" ] && ! cd "$1"; then
echo "Could not chdir to directory $1"
exit -1
fi
handle_directory
The general idea is a depth-first search of the directory tree in question. Like any tree, the directory tree is best handled recursively, and the function essentially boils down to: Walk through all things in this directory, if they are a directory, descend and handle it, then find an appropriate file name and rename the thing whether it is a directory or not.
Things used:
local counter=0 # declares a function-local variable counter and initializes it
# to 0. crucially, that it is local means that every invocation
# of handle_directory has its own counter.
[ -d "$i" ] # tests if "$i" is a directory
[ -h "$i" ] # tests if "$i" is a symlink
! [ ... ] # negates the test. ! [ -h "$i" ] is true if "$i" is NOT a symlink
"${i##*.}" # a bashism that cuts off the longest prefix that matches the pattern
[ -e "$dest" ] # tests if "$dest" exists
$1 # the first parameter with which the script is called
[ -z "$1" ] # tests if "$1" is an empty string
! cd "$1" # descends into the directory "$1". true if that failed.
Manpages to read to further understanding:
man test
man bash # that's the big one.

Is it a single level directory? Or do you have subdirectories?
Something like this?
Before:
.
├── folderE
│   ├── conf.properties
│   ├── omg.avi
│   └── test-diff.diff
├── folder_a
│   ├── first.png
│   ├── main.jpg
│   └── second.gif
├── folder_d
│   ├── another.zip
│   └── one.mpeg
└── with space
└── file with spaces too.mov
Command:
countDir=1
for dir in *; do
cd "$dir";
countFile=1
for file in *; do
mv "$file" $countFile.${file#*.}
((countFile++))
done
cd ..
mv "$dir" $countDir
((countDir++))
done
Or, the same in one line:
countDir=1; for dir in *; do cd "$dir"; countFile=1; for file in *; do mv "$file" $countFile.${file#*.}; ((countFile++)); done; cd ..; mv "$dir" $countDir; ((countDir++)); done
After:
.
├── 1
│   ├── 1.properties
│   ├── 2.avi
│   └── 3.diff
├── 2
│   ├── 1.png
│   ├── 2.jpg
│   └── 3.gif
├── 3
│   ├── 1.zip
│   └── 2.mpeg
└── 4
└── 1.mov
Important: keep in mind that this is just a quick and dirty solution and there's no checking for files/directoris already named "1" "2" etc.

Related

Create Folder for Each File in Recursive Directory, Placing File in Folder

Create Folder for Each File in Recursive Directory, Placing File in Folder
On MacOS, so far I have...
for file in $(ls -R); do
if [[ -f "$file" ]]; then mkdir "${file%.*}"; mv "$file" "${file%.*}"; fi;
done
This operates correctly on the top level of the nested folder, but does nothing with lower levels.
To isolate the error, I tried this instead, operating on rtf files . .
for i in $(ls -R);do
if [ $i = '*.rtf' ];then
echo "I do something with the file $i"
fi
done
This hangs, so I simplified to . .
for i in $(ls -R); do echo "file is $i" done
That hangs also, so I tried . .
for i in $(ls -R); do echo hello
That hangs also.
ls -R works to provide a recursive list of all files.
Suggestions appreciated !!
First of all don't use ls in scripts. It is meant to show output interactively. Although the newish GNU ls version has some features/options for shell parsing, not sure about on a Mac though.
Now using find with the sh shell.
find . -type f -name '*.rtf' -execdir sh -c '
for f; do mkdir -p -- "${f%.*}" && mv -v -- "$f" "${f%.*}"; done' _ {} +
For whatever reason -execdir is not available, one can use -exec
find . -type f -name '*.rtf' -exec sh -c '
for f; do mkdir -p -- "${f%.*}" && mv -v -- "$f" "${f%.*}" ; done' _ {} +
See Understanding-the-exec-option-of-find
Given this file structure:
$ tree .
.
├── 123
│   └── test_2.rtf
├── bar
│   ├── 456
│   │   └── test_1.rtf
│   └── 789
└── foo
There are two common ways to find all the .rtf files in that tree. The first (and most common) is to use find:
while IFS= read -r path; do
echo "$path"
done < <(find . -type f -name "*.rtf")
Prints:
./bar/456/test_1.rtf
./123/test_2.rtf
The second common way is to use a recursive glob. This is not a POSIX way and is only found in more recent shells such as Bash, zsh, etc:
shopt -s globstar # In Bash, this enables recursive globs
# In zsh, this is not required
for path in **/*.rtf; do
echo "$path"
done
# same output
Now that you have the loop to find the files, you can modify the files found.
The first issue you will run across is that you cannot have two files with the same name in a single directory; a directory is just a type of file. So you will need to proceed this way:
Find all the files with their paths;
Create a tmp name and create a sub-directory with that temp name;
Move the found file into the temp directory;
Rename the temp directory to the found file name.
Here is a Bash (or zsh that is default on MacOS) script to do that:
shopt -s globstar # remove for zsh
for p in **/*.rtf; do
[ -f "$p" ] || continue # if not a file, loop onward
tmp=$(uuidgen) # easy temp name -- not POSIX however
fn="${p##*/}" # strip the file name from the path
path_to="${p%/*}" # get the path without the file name
mkdir "${path_to}${tmp}" # use temp name for the directory
mv "$p" "${path_to}$tmp" # move the file to that directory
mv "${path_to}$tmp" "$p" # rename the directory to the path
done
And the result:
.
├── 123
│   └── test_2.rtf
│   └── test_2.rtf
├── bar
│   ├── 456
│   │   └── test_1.rtf
│   │   └── test_1.rtf
│   └── 789
└── foo

Bash script to swap file locations, files with same names

I have a bunch of .mov video in a folder on my mac:
Documents/MyMovs
├── SubFolder01
│   ├── SubSubFolder01
│   │   ├── MyFile1.mov
│   │   ├── MyPdf.pdf
│   │   └── MyTxt.txt
│   └── SubSubFolder02
│   ├── MyFile2.mov
│   ├── MyPdf.pdf
│   └── MyTxt.txt
etc.
Using HandBrake I converted all of them to .mp4 but HandBrake put them all in Movies/
Movies
├── MyFile1.mp4
├── MyFile2.mp4
etc.
What I want to do is swap the files and get the exact reverse situation where all my mp4 are stored (at the correct location) in Documents/MyMovs and all mov files are in Movies/ where I will be able to delete them eventually.
at the end I want:
Documents/MyMovs
├── SubFolder01
│   ├── SubSubFolder01
│   │   ├── MyFile1.mp4
│   │   ├── MyPdf.pdf
│   │   └── MyTxt.txt
│   └── SubSubFolder02
│   ├── MyFile2.mp4
│   ├── MyPdf.pdf
│   └── MyTxt.txt
etc.
and
Movies
├── MyFile1.mov
├── MyFile2.mov
etc.
I've designed the following bash script to help me (although I have the time to do it by hand due to the coronavirus crisis):
#!/usr/local/bin/bash
MOV_DIR=$1
MP4_DIR=$2
declare -A FILES=( )
find $MOV_DIR -maxdepth 3 -name '*.mov' -print0 |
while IFS= read -r -d '' line; do
filename=$(basename "$line")
extension="${filename##*.}"
filename="${filename%.*}"
path_to_file=$(dirname "$line")
if [[ ! -v FILES[$filename] ]]; then
FILES[$filename]="$path_to_file"
#echo $FILES[$filename] prints something!
fi
done
#for k in "${!FILES[#]}"; do echo "$k - ${FILES[$k]}"; done ## prints nothing!
find $MP4_DIR -maxdepth 3 -name '*.mp4' -print0 |
while IFS= read -r -d '' line; do
filename=$(basename "$line")
extension="${filename##*.}"
filename="${filename%.*}"
path_to_file=$(dirname "$line")
if [[ ! -z ${FILES["$filename"]} ]]; then
mv "$path_to_file/$filename.mp4" "${FILES[$filename]}"
mv "${FILES[$filename]}/filename.mov" "$path_to_file"
#I would like to remove $filename from FILES as well but I don't know how
fi
done
Let's assume all filenames are unique and distinct from each other for simplicity.
I put all my mov files in a map filename -> path_to_filename; for example MyFile1 -> Documents/MyMovs/SubFolder01/SubSubFolder01 . This is the first find block.
Secondly I find all my mp4 files and retrieve the correct location using the map.
The code does not work it seems the map is local to the first find command and does not exist after. Would anybody know why? I use bash version 5.0.16(1)-release. Thanks everybody!
PS: I you have a completely different but better solution feel free to share
If you have one file, then find the other
find Documents -maxdepth 4 -mindepth 3 -name '*.mov' -print0 |
while IFS= read -r -d '' mov; do
# extract the filename without extension
name=$(basename "$mov" .mov)
# find exactly the same name in Movies but with mp4
mp4="Movies/$name.mp4"
if [[ ! -f "$mp4" ]]; then
echo "ERROR: File $mp4 does not exists" >&2
exit 2
fi
# switch mp4 with mov
movdir=$(dirname "$mov")
mp4dir=$(dirname "$mp4")
echo mv -v "$mp4" "$movdir"
echo mv -v "$mov" "$mp4dir"
done
With the following directory/file recreation:
# find -type f
./Documents/MyMovs/Subfolder01/SubSubFolder01/MyFile1.mov
./Documents/MyMovs/Subfolder01/SubSubFolder02/MyFile2.mov
./Movies/MyFile1.mp4
./Movies/MyFile2.mp4
outputs:
mv -v Movies/MyFile1.mp4 Documents/MyMovs/Subfolder01/SubSubFolder01
mv -v Documents/MyMovs/Subfolder01/SubSubFolder01/MyFile1.mov Movies
mv -v Movies/MyFile2.mp4 Documents/MyMovs/Subfolder01/SubSubFolder02
mv -v Documents/MyMovs/Subfolder01/SubSubFolder02/MyFile2.mov Movies
does not exist after. Would anybody know why?
Because the right side of | is run in a subshell. Parent shell doesn't know anything about child shell environment.
$ echo | { a=1; echo a=$a; }
a=1
$ echo a=$a
a=

How can I put files into subdirectories based on the name of the file using bash?

I am trying to take a directory filled with an unknown number of files and put each file into a subdirectory based on the name of the file. A file's name could have multiple subdirectories in it all separated by the underscore character so it needs to be able to recursively extract subdirectory names.
Example Files:
2020_Documents_Bills_Water Bill.pdf
2020_Documents_Taxes_W2.pdf
2020_Documents_Receipts_Store Name_Groceries.pdf
2020_Pictures_Family Trip_California_Disney Land_Family Pic.jpg
So the 2020_Documents_Bills_Water Bill.pdf file would end up as 2020/Documents/Bills/Water Bill.pdf.
I would like to limit the tools used to bash, sed, grep, mkdir, and mv if possible.
I had some thoughts on how I believe the script should flow, but I don't know how to make it recursively get subdirectories without a lot of yucky if statements. I was thinking this code could probably get the first subdir and put it in an array and then remove that text and the underscore that follows it from the name of the file and then iterate again until it runs out of underscores.
#!/bin/bash
# cd to directory where files are located
cd /directory/with/files
# iterate over files in directory
for file in *; do
subDirs=() # empty array for subdirs
filePath="" # empty string to build filepath
# ------------------------------------------------------------
# code to extract subdir names and add to subDirs array
# ------------------------------------------------------------
# build filepath using string of all subdirs
for i in ${!subDirs[#]}; do
filepath="${filePath}/${subDirs[$i]}"
done
# set filename to text after last underscore
filename=${file##*_}
# make filepath based on subdirs
mkdir -p "${filepath}"
# move file into filepath without subdirs in name
mv ${file} "${filepath}/${filename}"
done
You can do it simpler because mkdir -p path/to/yours works just with one invocation. You do not have to recursively create subdirectories one by one.
Would you please try:
cd /directory/with/files # cd to directory where files are located
for file in *; do
[[ -f $file ]] || continue # skip non-file entries (just in case)
dir=${file%_*}
base=${file##*_}
dir=${dir//_/\/} # replace "_"s with "/"s
mkdir -p "$dir"
mv -- "$file" "$dir/$base"
done
[Strict Version]
The script below performs the validation of the filenames (with a help of jhnc).
for file in *; do
[[ -f $file ]] || continue # skip non-file entries (just in case)
dir=${file%_*}
base=${file##*_}
dir=${dir//_//} # replace "_"s with "/"s
# validate filenames
case "/$dir/" in
*/../* | */./* | //*) # $dir contains extra dot(s)
echo "skipping invalid filename: $file"
continue
;;
esac
if [[ -z $base ]]; then # the filename ends with "_"
echo "skipping invalid filename: $file"
continue
fi
mkdir -p "$dir"
mv -- "$file" "$dir/$base"
done
Result:
/directory/
└── with
└── files
└── 2020
├── Documents
│   ├── Bills
│   │   └── Water Bill.pdf
│   ├── Receipts
│   │   └── Store Name
│   │   └── Groceries.pdf
│   └── Taxes
│   └── W2.pdf
└── Pictures
└── Family Trip
└── California
└── Disney Land
└── Family Pic.jpg
Just one note to add cd /directory/with/files
Without the exit . Assuming /directory/with/files does not exists.
#!/bin/bash
# cd to directory where files are located
cd /directory/with/files
printf '%s\n' 'rm this' 'rm that' 'mv this' 'mv that'
the output is
myscript: line 4: cd: /directory/with/files: No such file or directory
rm this
rm that
mv this
mv that
all the code after the cd is/was still executed!
With the exit and assuming the /directory/with/files does not exists.
#!/bin/bash
# cd to directory where files are located
cd /directory/with/files || exit
printf '%s\n' 'rm this' 'rm that' 'mv this' 'mv that'
The output.
myscript: line 4: cd: /directory/with/files: No such file or directory
The script did exit and it did not execute the rest of the code.

deleting intermediary folders

Maybe one of you guys has something like this at hand already? I tried to use robocopy on windows but to no avail. I also tried to write a bash script in linux with find etc... but gave up on that one also ^^ Google search brought no solution also unfortunately. I need this for my private photo library.
Solution could be linux or windows based, both are fine. Any ideas?
I would like to get rid of hundreds of 'intermediary folders'.
I define an 'intermediary folder' as a folder that contains nothing else than exactly one sub-folder. Example
folder 1
file in folder 1
folder 2 <-- 'intermediary folder: contains exactly one sub-folder, nothing else'
folder 3
file in folder 3
What I would like to end up with is:
folder 1
file in folder 1
folder 3
file in folder 3
I do not need the script to be recursive (removing several layers of intermediary folders at once), I'll just run it several times.
Even cooler would be if the script could rename folder 3 in the above example to 'folder 2 - folder 3', but I can live without this feature I guess.
I guess one of you linux experts has a one liner handy for that? ^^
Thank you very much!
Take a look at this code:
#!/usr/bin/env bash
shopt -s nullglob
while IFS= read -rd '' dir; do
f=("$dir"/*)
if ((${#f[#]}==1)) && [[ -d $f ]]; then
mv -t "${dir%/*}" "$f" || continue
rm -r "$dir"
fi
done < <(find folder1 -depth -mindepth 1 -type d -print0)
Explanation:
shopt -s nullglob: allows filename patterns which match no files to expand to a null string
find ... -depth: makes find traverse the file system in a depth-first order
find ... -mindepth 1: processes all directories except the starting-point
find ... -type d: finds only directories
find ... -print0: prints the directories separated by a null character \0 (to correctly handle possible newlines in filenames)
while IFS= read ...: loops over all the directories (the output of find)
f=("$dir"/*): creates an array with all files in the currently processed directory
((${#f[#]}==1)) && [[ -d $f ]]: true if there is only one file and it is a directory
mv -t "${dir%/*}" "$f": moves the only subdirectory one directory above
mv ... || continue: mv can fail if the subdirectory already exists in the directory above. || continue ignores such subdirectory
rm -r "$dir": removes the processed directory
Test run:
$ tree folder1
folder1
├── file1
├── folder2
│   └── folder3
│   └── file3
├── folder4
│   ├── file4a
│   ├── file4b
│   └── file4c
└── folder5
└── folder6
├── file6
└── folder7
└── folder8
└── folder9
├── dir9
└── file9
$ ./script
$ tree folder1
folder1
├── file1
├── folder3
│   └── file3
├── folder4
│   ├── file4a
│   ├── file4b
│   └── file4c
└── folder6
├── file6
└── folder9
├── dir9
└── file9

How to get duplicate files numbered as file1.txt_1, file1.txt_2, file.txt_3 when copying in BASH?

In a script, I need to copy files from several directories, into one directory, e.g.:
$ dir
dir1 dir2 dir3 output
$ cp */* output/
Sometimes, there are files with the same name though:
$ ls dir1/*
file1.txt
$ ls dir2/*
file1.txt
$ ls dir3/*
file1.txt
How can I copy all of the files from the different directories, into a single directory, without with the filenames changed whenever two files with identical names are placed in the same directory? Sample result:
$ cp */* output/
$ ls output/
file1.txt_1
file1.txt_2
file1.txt_3
A pretty simple way to do this is use the --backup flag to cp. It will back up files which are replaced rather than leaving them, but that might be sufficient. This approach is simple enough it would be easy to wrap an alias around.
source
├── dir1
│   └── file1.txt
├── dir2
│   └── file2.txt
├── dir3
│   └── file1.txt
└── dir4
└── file10.txt
cp --backup=numbered source/dir*/* dest
$ tree dest
dest
├── file1.txt
├── file1.txt.~1~
├── file10.txt
└── file2.txt
I'm having a little trouble with the _number, but if you're ok with the syntax as file.txt file.txt.1 file.txt.2 file.txt.3 then this should work
#!/bin/bash
treedir="/path/to/tree"
cd $treedir
for folder in * ; do
if [ -d "$folder" ] && [ "$folder" != "output" ] ; then
cd "$folder"
for file in * ; do
if [ -e "$treedir/output/$file" ] ; then
fcount=$(grep -s "" "$treedir/$file.count")
if [ -n "$fcount" ] ; then
fcount=$(( fcount + 1 ))
echo $fcount > "$treedir/$file.count"
else
fcount=1
echo $fcount > "$treedir/$file.count"
fi
cp "$file" "$treedir/output/$file.$fcount"
else
cp "$file" "$treedir/output/$file"
fi
done
cd ..
fi
done
rm *.count
exit 0
This creates temporary *.count files to keep track of how many files of the same name there are and then it deletes them in the end.
Just replace "/path/to/tree/" with the path to the directory containing your dir1 dir2 dir3 output etc. and you should be good to go
declare -A indexes
for file in */*
do
name=$(basename "$file")
index=${indexes[$name]}
if [ -z "$index" ]
then
cp "$file" output/
indexes[$name]=1
else
cp "$file" output/"$name"_$index
indexes[$name]=$((index+1))
fi
done
unset indexes
There is another, perhaps shorter way to insure a unique copy name when consolidating files from multiple directories. The following will copy all files (one level deep) from multiple directories into a single output_dir and add the minimum file_# index to insure unique filenames (if more depth is needed -- use find):
#!/bin/bash
arr=( path/to/*/* ) # fill array with files to be copied
for file in ${arr[#]}; do # for each file
idx=0 # set the copy index to 0
dfn=${file##*/} # destination file name (dfn) w/path stripped
while [ -f output_dir/"$dfn" ] ; do # test if $dfn exist in output_dir
dfn=${file##*/}_$((idx++)) # if so, add copy index "_#" (increment until unique)
done
cp "$file" ../c/"$dfn" # copy file with unique file name
done
input dir: tmp/*/*
tmp/a/a.j
tmp/a/a.k
tmp/a/a.l
tmp/a/a.txt
tmp/b/a.j
tmp/b/a.k
tmp/b/a.l
tmp/c/a.j
tmp/c/a.k
tmp/c/a.l
tmp/c/a.txt
tmp/d/a.j
tmp/d/a.k
tmp/d/a.l
tmp/d/a.txt
output_dir: output
a.j
a.j_0
a.j_1
a.j_2
a.k
a.k_0
a.k_1
a.k_2
a.l
a.l_0
a.l_1
a.l_2
a.txt
a.txt_0
a.txt_1
Another approach if you don't have a modern GNU cp available would be to use the source directory as the suffix. This is simpler than some approaches as you don't have to do any counting and you don't have to use an array.
SOURCE=$1
DEST=$2
for i in $(find $SOURCE -type f) ; do
if [[ -r $DEST/$(basename $i) ]] ; then
NEW_NAME=$(basename $i)_$(basename $(dirname $i))
cp $i $DEST/$NEW_NAME
else
cp $i $DEST/
fi
done
$ tree source
source
├── dest
├── dir1
│   └── file1.txt
├── dir2
│   └── file2.txt
├── dir3
│   └── file1.txt
└── dir4
└── file10.txt
$ bash ./copy.sh source dest
$ tree dest
dest
├── file1.txt
├── file1.txt_dir3
├── file10.txt
└── file2.txt

Resources