Delete/move files which in all respects have same file name except one set has a specific prefix added - bash

I have a large set of working files of the form a.mp4 b.txt c.avi d.doc etc (the extension is irrelevant to the question). I also have a set of files which include the same named files except with a common specific prefix "broken_" eg broken_a.mp4, broken_b.txt
If I have a.mp4 and broken_a.mp4, I want to move the broken_a.mp4 to a holding directory. If I have broken_d.mp4 but no matching d.mp4, then leave it alone.
I have some code successfully used to identify and move files with the same extension which I'd like to modify
This is the form of working example code for same extension files (kudos to the original author) which I'd like to modify if possible to do the job
#!/bin/bash
# Name of source directory
SOURCE_DIR=.
# Name of destination directory
DEST_DIR=already_converted_m4v
# Create the destination directory for the moved files, if it doesn't already exist.
[ ! -d $DEST_DIR ] && mkdir -p $DEST_DIR
find $SOURCE_DIR -maxdepth 1 -type f -iname "*.avi" | while read fin
do
#echo "m4v doing avi"
fm4v=${fin/.avi/.m4v}
[ -f "$fm4v" ] && gmv -v --backup=numbered "$fin" $DEST_DIR/
done
My garbage first attempt which clearly doesnt work looks horribly like:
#!/bin/bash
# Name of source directory
SOURCE_DIR=.
# Name of destination directory
DEST_DIR=Already_broken
# Create the destination directory for the moved files, if it doesn't already exist.
[ ! -d $DEST_DIR ] && mkdir -p $DEST_DIR
find $SOURCE_DIR -maxdepth 1 -type f -iname "*" | while read fin
do
#echo "working to find existing broken and unbroken files"
filetest_basename=$(basename "$fin" )
filetest_extension=$(extension "$fin" )
echo $filetest_basename
echo $filetest_extension
fileok=${filetest_basename/!broken_/broken_}
[ -f "$fileok" ] && gmv -v --backup=numbered "$fin" $DEST_DIR/
done
Grateful for help

find is irrelevant here, a simple shell loop would suffice:
SRCDIR='.'
DSTDIR='Already_broken'
if ! [ -d "$DSTDIR" ]; then
mkdir -p -- "$DSTDIR"
fi
for broken in "$SRCDIR"/broken_*; do
if [ -f "${broken%"${broken##*/}"}${broken##*/broken_}" ]; then
echo gmv -v --backup=numbered "$broken" -- "$DSTDIR"
fi
done
If its output looks good, remove echo.
Some notes:
We can't use ${broken/broken_} here for SRCDIR might contain broken_ in the future.
Nested PE (${broken##*/}) needs to be quoted for its result might contain metacharacters and that would bring about undesired results.

Related

Find operator can't go up in directory?

I'm trying to list all files of a directory that not exist in another.
And the script tells me:
find: ‘ (...) /nanorc/../original/*’: No such file or directory
Where is the problem? Thanks!
Code:
# todo: list all files that isn't in original folder
# get a file in nanorc folder
# try to find it in the ../original folder
# if there isn't, list it.
cd nanorc/
for file in *; do
[ -e "$file" ] || continue
# if [ -z $(find "$(pwd)/../original/*" -name "$file") ]; then
if [ -z $(find $(pwd)/../original/* -name "$file") ]; then
lite=`printf "%s %s" "$lite" "$file"`
fi
done
cd ..
echo "$lite
Notes:
The script must be POSIX complaint.
I tried various variations of find, like: "../original", "../original/*",...
I don't want to create files; the script should populate one variable after the checks.
Solution:
Just put out the "".
To check if a file exists use test -e.
if [ ! -e "../original/$file" ]; then
echo "File ../original/$file does not exists"
fi
Path are by default searched in current working directory, there is no need to use $(pwd).
The error in your script came from "$(pwd)/../original/*" - the * is inside " so it does not expand. You could change it to "../original/"*, but because find works recursively, just find ../original.

Bash: how to copy multiple files with same name to multiple folders

I am working on Linux machine.
I have a lot of files named the same, with a directory structure like this:
P45_input_foo/result.dat
P45_input_bar/result.dat
P45_input_tar/result.dat
P45_input_cool/result.dat ...
It is difficult to copy them one by one. I want to copy them into another folder named as data with similar folder names and file names:
/data/foo/result.dat
/data/bar/result.dat
/data/tar/result.dat
/data/cool/result.dat ...
In stead of copy them one by one what I should do?
Using a for loop in bash :
# we list every files following the pattern : ./<somedirname>/<any file>
# if you want to specify a format for the folders, you could change it here
# i.e. for your case you could write 'for f in P45*/*' to only match folders starting by P45
for f in */*
do
# we strip the path of the file from its filename
# i.e. 'P45_input_foo/result.dat' will become 'P45_input_foo'
newpath="${f%/*}"
# mkdir -p /data/${newpath##*_} will create our new data structure
# - /data/${newpath##*_} extract the last chain of character after a _, in our example, 'foo'
# - mkdir -p will recursively create our structure
# - cp "$f" "$_" will copy the file to our new directory. It will not launch if mkdir returns an error
mkdir -p /data/${newpath##*_} && cp "$f" "$_"
done
the ${newpath##*_} and ${f%/*} usage are part of Bash string manipulation methods. You can read more about it here.
You will need to extract the 3rd item after "_" :
P45_input_foo --> foo
create the directory (if needed) and copy the file to it. Something like this (not tested, might need editing):
STARTING_DIR="/"
cd "$STARTING_DIR"
VAR=$(ls -1)
while read DIR; do
TARGET_DIR=$(echo "$DIR" | cut -d'_' -f3)
NEW_DIR="/data/$DIR"
if [ ! -d "$NEW_DIR" ]; then
mkdir "$NEW_DIR"
fi
cp "$DIR/result.dat" "$NEW_DIR/result.dat"
if [ $? -ne 0 ];
echo "ERROR: encountered an error while copying"
fi
done <<<"$VAR"
Explanation: assuming all the paths you've mentioned are under root / (if not change STARTING_PATH accordingly). With ls you get the list of the directories, store the output in VAR. Pass the content of VAR to the while loop.
A bit of find and with a few bash tricks, the below script could do the trick for you. Remember to run the script without the mv and see if "/data/"$folder"/" is the actual path that you want to move the file(s).
#!/bin/bash
while IFS= read -r -d '' file
do
fileNew="${file%/*}" # Everything before the last '\'
fileNew="${fileNew#*/}" # Everything after the last '\'
IFS="_" read _ _ folder <<<"$fileNew"
mv -v "$file" "/data/"$folder"/"
done < <(find . -type f -name "result.dat" -print0)

How can I copy files recursively without overwriting duplicate file names?

Let's say I have files nested in subdirectories with potentially duplicate file names. I want to copy all files to a new directory but prevent overwriting AND preserve filenames (mostly).
The following does not work because it overwrites duplicate filenames:
find /SourceDir/. -type f -exec cp -pv \{\} /DestDir/ \;
Adding noclobber (cp -n) doesn't help either because duplicates are just skipped.
Current File Structure:
SourceDir
--SubdirA
----File1.gif
---- ...
----File1000.jpg
--SubdirB
----File1.gif
---- ...
----File1000.png
...
--SubdirZ
----SubdirAA
------File1.sh
------ ...
------File1000.jpg
Desired File Structure:
DestDir
--File1.gif
--File1_1.gif <-- original name was `File1.gif` but this already existed
--File2.jpg
--File2.gif <-- `File2.jpg` already exists, but not `File2.gif`
--File3.gif
--File3_1.gif
--File4.jpg
--File4_1.jpg
--File4_2.jpg <-- original name was `File4.jpg`, but `File4_1.jpg` already existed too.
-- ...
--File1000.png
I do not want to rename every file. And I don't want to give arbitrary hashes to those I need to duplicate. What do you recommend?
I'm on a Mac, so Linux commands are all fair game.
Here is one solution.
#!/bin/bash
SourceDir=".";
DestDir="../dest";
cd ${SourceDir}
find . -type f |
while read x
do
bn=`basename $x`;
if [ -f "${DestDir}/$bn" ]
then
for i in {1..9999}
do
if [ ! -f "${DestDir}/${bn%.*}_${i}.${bn##*.}" ]
then
echo "Next free file extension is no $i";
bn="${DestDir}/${bn%.*}_${i}.${bn##*.}"
break;
fi
done
fi
echo "copy file $x to ${DestDir}/$bn";
cp -p "$x" "${DestDir}/$bn";
done
Please let me know if that works for you.

bash scripting copying all files in folder

I'm writing a shell script as follows:
for file in `ls`
do
mkdir "$file"_folder
cp $file "$file"_folder
done
What I want to do is to make a folder for each file in the current directory with its name and then underscore folder as the name and then copy that file into it. My problem is that the file names contain spaces in them. How do I escape them?
There are many resources explaining how to do this for variables but none of them can be applied to this situation where I use a for loop to get the names.
Don't use ls there, use shell globbing. (In general, do not parse the output of ls.)
for file in *
do
# only consider files, not directories
if [ -f "$file" ] ; then
new_dir="$file"_folder
# create the directory
if [ ! -d "$new_dir" ] ; then
mkdir "$new_dir"
if [ $? -ne 0 ] ; then
# handle directory creation eror
fi
fi
# possibly check for the copied file existence here
# and deal with that appropriately (i.e. skip/error/copy anyway)
cp "$file" "$new_dir"
fi
done
How about
find . -type f -exec mkdir {}_folder \; -exec cp {} {}_folder \;
It finds all regular files in the current directory, creates the folder (first -exec), and copies the file into the new folder (second -exec).
You do not parse
ls for exactly this reason
for file in *
do
mkdir "${file}_folder"
cp "$file" "${file}_folder"
done

How to copy and rename files in shell script

I have a folder "test" in it there is 20 other folder with different names like A,B ....(actually they are name of people not A, B...) I want to write a shell script that go to each folder like test/A and rename all the .c files with A[1,2..] and copy them to "test" folder. I started like this but I have no idea how to complete it!
#!/bin/sh
for file in `find test/* -name '*.c'`; do mv $file $*; done
Can you help me please?
This code should get you close. I tried to document exactly what I was doing.
It does rely on BASH and the GNU version of find to handle spaces in file names. I tested it on a directory fill of .DOC files, so you'll want to change the extension as well.
#!/bin/bash
V=1
SRC="."
DEST="/tmp"
#The last path we saw -- make it garbage, but not blank. (Or it will break the '[' test command
LPATH="/////"
#Let us find the files we want
find $SRC -iname "*.doc" -print0 | while read -d $'\0' i
do
echo "We found the file name... $i";
#Now, we rip off the off just the file name.
FNAME=$(basename "$i" .doc)
echo "And the basename is $FNAME";
#Now we get the last chunk of the directory
ZPATH=$(dirname "$i" | awk -F'/' '{ print $NF}' )
echo "And the last chunk of the path is... $ZPATH"
# If we are down a new path, then reset our counter.
if [ $LPATH == $ZPATH ]; then
V=1
fi;
LPATH=$ZPATH
# Eat the error message
mkdir $DEST/$ZPATH 2> /dev/null
echo cp \"$i\" \"$DEST/${ZPATH}/${FNAME}${V}\"
cp "$i" "$DEST/${ZPATH}/${FNAME}${V}"
done
#!/bin/bash
## Find folders under test. This assumes you are already where test exists OR give PATH before "test"
folders="$(find test -maxdepth 1 -type d)"
## Look into each folder in $folders and find folder[0-9]*.c file n move them to test folder, right?
for folder in $folders;
do
##Find folder-named-.c files.
leaf_folder="${folder##*/}"
folder_named_c_files="$(find $folder -type f -name "*.c" | grep "${leaf_folder}[0-9]")"
## Move these folder_named_c_files to test folder. basename will hold just the file name.
## Don't know as you didn't mention what name the file to rename to, so tweak mv command acc..
for file in $folder_named_c_files; do basename=$file; mv $file test/$basename; done
done

Resources