I am using the following code but the final 'echo $dirname' is giving empty output on console
for folderpath in find /u01/app/SrcFiles/commercial/ngdw/* -name "IQ*";
do
folder_count=ls -d $folderpath/* | wc -l
echo -e "Total Date folder created : $folder_count in $folderpath \n"
if [ $folder_count -ne 0 ];
then
for dirpath in `find $folderpath/* -name "2*" `;
do
dirname=${dirpath##*/}
(( dirname <= 20210106 )) || continue
echo $dirname
done
fi
done
First I would calculate the date it was 3 months ago with the date command:
# with GNU date (for example on Linux)
mindate=$(date -d -3month +%Y%m%d)
# with BSD date (for example on macOS)
mindate=$(date -v -3m +%Y%m%d)
Then I would use a shell arithmetic comparison for determining the directories to remove:
# for dirpath in "$directory"/*
for dirpath in "$directory"/{20220310,20220304,20220210,20220203,20210403,20210405}
do
dirname=${dirpath##*/}
(( dirname <= mindate )) || continue
echo "$dirpath"
# rm -rf "$dirpath"
done
== doesn't do wildcard matching. You should do that in the for statement itself.
There's also no need to put * at the beginning of the wildcard, since the year is at the beginning of the directory name, not in the middle.
for i in "$directory"/202104*; do
if [ -d "$i" ]; then
echo "$i"
rm -rf "$i"
fi
done
The if statement serves two purposes:
If there are no matching directories, the wildcard expands to itself (unless you set the nullglob option), and you'll try to remove this nonexistent directory.
In case there are matching files rather than directories, they're skipped.
Suggesting to find which are the directories that created before 90 days ago or older, with find command.
find . -type d -ctime +90
If you want to find which directories created 90 --> 100 days ago.
find . -type d -ctime -100 -ctime +90
Once you have the correct folders list. Feed it to the rm command.
rm -rf $(find . -type d -ctime +90)
Related
I have an audio sample library with thousands of files. I would like to shuffle/randomize the order of these files. Can someone provide me with a bash script/line that would prepend a single random character to all files in a folder (including files in sub-folders). I do not want to prepend a random character to any of the folder names though.
Example:
Kickdrum73.wav
Kickdrum SUB.wav
Kick808.mp3
Renamed to:
f_Kickdrum73.wav
!_Kickdrum SUB.wav
4_Kick808.mp3
If possible, I would like to be able to run this script more than once, but on subsequent runs, it just changes the randomly prepended character instead of prepending a new one.
Some of my attempts:
find ~/Desktop/test -type f -print0 | xargs -0 -n1 bash -c 'mv "$0" "a${0}"'
find ~/Desktop/test/ -type f -exec mv -v {} $(cat a {}) \;
find ~/Desktop/test/ -type f -exec echo -e "Z\n$(cat !)" > !Hat 15.wav
for file in *; do
mv -v "$file" $RANDOM_"$file"
done
Note: I am running on macOS.
Latest attempt using code from mr. fixit:
find . -type f -maxdepth 999 -not -name ".*" |
cut -c 3- - |
while read F; do
randomCharacter="${F:2:1}"
if [ $randomCharacter == '_' ]; then
new="${F:1}"
else
new="_$F"
fi
fileName="`basename $new`"
newFilename="`jot -r -c $fileName 1 A Z`"
filePath="`dirname $new`"
newFilePath="$filePath$newFilename"
mv -v "$F" "$newFilePath"
done
Here's my first answer, enhanced to do sub-directories.
Put the following in file randomize
if [[ $# != 1 || ! -d "$1" ]]; then
echo "usage: $0 <path>"
else
find $1 -type f -not -name ".*" |
while read F; do
FDIR=`dirname "$F"`
FNAME=`basename "$F"`
char2="${FNAME:1:1}"
if [ $char2 == '_' ]; then
new="${FNAME:1}"
else
new="_$FNAME"
fi
new=`jot -r -w "%c$new" 1 A Z`
echo mv "$F" "${FDIR}/${new}"
done
fi
Set the permissions with chmod a+x randomize.
Then call it with randomize your/path.
It'll echo the commands required to rename everything, so you can examine them to ensure they'll work for you. If they look right, you can remove the echo from the 3rd to last line and rerun the script.
cd ~/Desktop/test, then
find . -type f -maxdepth 1 -not -name ".*" |
cut -c 3- - |
while read F; do
char2="${F:2:1}"
if [ $char2 == '_' ]; then
new="${F:1}"
else
new="_$F"
fi
new=`jot -r -w "%c$new" 1 A Z`
mv "$F" "$new"
done
find . -type f -maxdepth 1 -not -name ".*" will get all the files in the current directory, but not the hidden files (names starting with '.')
cut -c 3- - will strip the first 2 chars from the name. find outputs paths, and the ./ gets in the way of processing prefixes.
while read VAR; do <stuff>; done is a way to deal with one line at a time
char2="${VAR:2:1} sets a variable char2 to the 2nd character of the variable VAR.
if - then - else sets new to the filename, either preceded by _ or with the previous random character stripped off.
jot -r -w "%c$new" 1 A Z tacks random 1 character from A-Z onto the beginning of new
mv old new renames the file
You can also do it all in bash and there are several ways to approach it. The first is simply creating an array of letters containing whatever letters you want to use as a prefix and then generating a random number to use to choose the element of the array, e.g.
#!/bin/bash
letters=({0..9} {A..Z} {a..z}) ## array with [0-9] [A-Z] [a-z]
for i in *; do
num=$(($RANDOM % 63)) ## generate number
## remove echo to actually move file
echo "mv \"$i\" \"${letters[num]}_$i\"" ## move file
done
Example Use/Output
Current the script outputs the changes it would make, you must remove the echo "..." surrounding the mv command and fix the escaped quotes to actually have it apply changes:
$ bash ../randprefix.sh
mv "Kick808.mp3" "4_Kick808.mp3"
mv "Kickdrum SUB.wav" "h_Kickdrum SUB.wav"
mv "Kickdrum73.wav" "l_Kickdrum73.wav"
You can also do it by generating a random number representing the ASCII character between 48 (character '0') through 126 (character '~'), excluding 'backtick'), and then converting the random number to an ASCII character and prefix the filename with it, e.g.
#!/bin/bash
for i in *; do
num=$((($RANDOM % 78) + 48)) ## generate number for '0' - '~'
letter=$(printf "\\$(printf '%03o' "$num")") ## letter from number
while [ "$letter" = '`' ]; do ## exclude '`'
num=$((($RANDOM % 78) + 48)) ## generate number
letter=$(printf "\\$(printf '%03o' "$num")")
done
## remove echo to actually move file
echo "mv \"$i\" \"${letter}_$i\"" ## move file
done
(similar output, all punctuation other than backtick is possible)
In each case you will want to place the script in your path or call it from within the directory you want to move the file in (you split split dirname and basename and join them back together to make the script callable passing the directory to search as an argument -- that is left to you)
I'm trying to use the newer command as follows:
touch $HOME/mark.start -d "$d1"
touch $HOME/mark.end -d "$d2"
SF=$HOME/mark.start
EF=$HOME/mark.end
find . -newer $SF ! -newer $EF
But this gives me an output like this:
.
./File5
and counts it as 2 files, however that directory only has 1 file i.e., File5. Why is this happening and how to solve it?
UPDATE:
I'm actually trying to run the following script:
#!/bin/bash
check_dir () {
d1=$2
d2=$((d1+1))
f1=`mktemp`
f2=`mktemp`
touch -d $d1 $f1
touch -d $d2 $f2
n=$(find $1 \( -name "*$d1*" \) -o \( -newer $f1 ! -newer $f2 \) | wc -l)
if [ $n != $3 ]; then echo $1 "=" $n ; fi
rm -f $f1 $f2
}
That checks if the directory has file that either has a particular date in the format YYYMMDD or if its last modification time was last 1 day.
check_dir ./dir1 20151215 4
check_dir ./dir2 20151215 3
where in dir1 there should be 4 such files and if it is not true then it will print the actual number of files that is there.
So, when the directory only has file with dates in their name, then it checks them fine, but when it checks with newer, it always gives 1 file extra (which is not even there in the directory). Why is this happening???
The question asks why there's an extra . in the results from find, even when no file or directory by that name exists. The answer is simple: . always exists, even when it's hidden. Use ls -a to show hidden contents, and you'll see that it's present.
Your existing find command doesn't exempt the target directory itself -- . -- from being a legitimate result, which is why you're getting more results than you expect.
Add the following filter:
-mindepth 1 # only include content **under** the file or directory specified
...or, if you only want to count files, use...
-type f # only include regular files
Assuming GNU find, by the way, this all can be made far more efficient:
check_dir() {
local d1 d2 # otherwise these variables leak into global scope
d1=$2
d2=$(gdate -d "+ 1 day $d1" '+%Y%m%d') # assuming GNU date is installed as gdate
n=$(find "$1" -mindepth 1 \
-name "*${d1}*" -o \
'(' -newermt "$d1" '!' -newermt "$d2" ')' \
-printf '\n' | wc -l)
if (( n != $3 )); then
echo "$1 = $n"
fi
}
I have a Base Directory that has 4 directories : Dir1 Dir2 Dir3 Dir4. Each of these directories have files in the format: "Sometext_YYYMMMDD". I'm writing a UNIX script to search through the files in all these directories that have a particular string say "20151215", and then printing it on the console.
find . -name "*20151215" -print
Example of files: File1_20151215 (this will be printed);
File2_20151214 (this will not be printed)
I want to write a script that runs through these directories and checks if Dir1 contains 4 files with string "20151215", Dir2 contains 3 files with string "20151215" and Dir3 & Dir4 contains 4 files with string "20151215". If the directories don't contain that number of files with that string, then I want to print those directories.
How do I do that? Please help!
UPDATE: I have an addition to this: There are also some files that are not in the format "Sometext_YYYMMMDD" So, for those I used something like:
find . -name "FILENAME*" -mtime -1 -exec ls -ltr '{}' \;
to extract the timestamp when that file was created. But, I want to know how do I add it to the script so that if the timestamp is 15 Dec 2015, then this file should also be counted in the search?
You got the find part, but now you need to count how many files match the pattern. Since find prints one line per match to the output, you can use "wc -l" to count how many lines there are. Assign that to a variable, that you can use in a comparison, and you're 90% of the way there. E.g.
d1=$(find ./dir1 -name '*20151215*' | wc -l)
if [ $d1 != 4 ]; then echo "dir1" ; fi
For extra credit, you can imagine turning this into a function with inputs of
Directory to search
Filename pattern to match on
How many matches to expect
Which would look like:
check_dir () {
d1=$(find $1 -name "*$2*" | wc -l)
if [ $d1 != $3 ]; then echo $1 ; fi
}
check_dir ./dir1 20151215 4
check_dir ./dir2 20151215 3
Update: with the new requirement to find files based either on the name of the file or the last modification (creation isn't possible), here's two approaches:
The first uses a fairly modern feature of find that isn't available in all versions, newermt:
check_dir () {
d1=$2
d2=$((d1+1))
n=$(find $1 \( -name "*$d1*" \) -o \( -newermt $d1 ! -newermt $d2 \) | wc -l)
if [ $n != $3 ]; then echo $1 ; fi
}
check_dir ./dir1 20151215 4
check_dir ./dir2 20151215 3
Which looks a little confusing, but break it down into small steps and it makes sense:
d1=$2 # So d1=20151215
d2=$((d1+1)) # d2=20151216 (lucky you're specifying the date format this way!)
The find command now has two predicates, to match based on the filename or the modification time:
\( -name "*$2*" \) # Matches filenames that contain 20151215
-o # Or
\( -newermt $d1 ! -newermt $d2 \)
The modification time is greater than midnight on the first day, and not greater than midnight on the next day
The second approach uses a couple of temp files, and sets the timestamps on them using the -d option of the touch command
#!/bin/bash
check_dir () {
d1=$2
d2=$((d1+1))
f1=`mktemp`
f2=`mktemp`
touch -d $d1 $f1
touch -d $d2 $f2
n=$(find $1 \( -name "*$d1*" \) -o \( -newer $f1 ! -newer $f2 \) | wc -l)
if [ $n != $3 ]; then echo $1 "=" $n ; fi
rm -f $f1 $f2
}
Again, it's lucky that the date is in YYYYMMDD since that works with the -d option of the touch command. If not, you would need to do some string manipulation to get the date into the correct format for "touch -t".
I need to loop through a directory recursively and remove all files with extension .pdf and .doc. I'm managing to loop through a directory recursively but not managing to filter the files with the above mentioned file extensions.
My code so far
#/bin/sh
SEARCH_FOLDER="/tmp/*"
for f in $SEARCH_FOLDER
do
if [ -d "$f" ]
then
for ff in $f/*
do
echo "Processing $ff"
done
else
echo "Processing file $f"
fi
done
I need help to complete the code, since I'm not getting anywhere.
As a followup to mouviciel's answer, you could also do this as a for loop, instead of using xargs. I often find xargs cumbersome, especially if I need to do something more complicated in each iteration.
for f in $(find /tmp -name '*.pdf' -or -name '*.doc'); do rm $f; done
As a number of people have commented, this will fail if there are spaces in filenames. You can work around this by temporarily setting the IFS (internal field seperator) to the newline character. This also fails if there are wildcard characters \[?* in the file names. You can work around that by temporarily disabling wildcard expansion (globbing).
IFS=$'\n'; set -f
for f in $(find /tmp -name '*.pdf' -or -name '*.doc'); do rm "$f"; done
unset IFS; set +f
If you have newlines in your filenames, then that won't work either. You're better off with an xargs based solution:
find /tmp \( -name '*.pdf' -or -name '*.doc' \) -print0 | xargs -0 rm
(The escaped brackets are required here to have the -print0 apply to both or clauses.)
GNU and *BSD find also has a -delete action, which would look like this:
find /tmp \( -name '*.pdf' -or -name '*.doc' \) -delete
find is just made for that.
find /tmp -name '*.pdf' -or -name '*.doc' | xargs rm
Without find:
for f in /tmp/* tmp/**/* ; do
...
done;
/tmp/* are files in dir and /tmp/**/* are files in subfolders. It is possible that you have to enable globstar option (shopt -s globstar).
So for the question the code should look like this:
shopt -s globstar
for f in /tmp/*.pdf /tmp/*.doc tmp/**/*.pdf tmp/**/*.doc ; do
rm "$f"
done
Note that this requires bash ≥4.0 (or zsh without shopt -s globstar, or ksh with set -o globstar instead of shopt -s globstar). Furthermore, in bash <4.3, this traverses symbolic links to directories as well as directories, which is usually not desirable.
If you want to do something recursively, I suggest you use recursion (yes, you can do it using stacks and so on, but hey).
recursiverm() {
for d in *; do
if [ -d "$d" ]; then
(cd -- "$d" && recursiverm)
fi
rm -f *.pdf
rm -f *.doc
done
}
(cd /tmp; recursiverm)
That said, find is probably a better choice as has already been suggested.
Here is an example using shell (bash):
#!/bin/bash
# loop & print a folder recusively,
print_folder_recurse() {
for i in "$1"/*;do
if [ -d "$i" ];then
echo "dir: $i"
print_folder_recurse "$i"
elif [ -f "$i" ]; then
echo "file: $i"
fi
done
}
# try get path from param
path=""
if [ -d "$1" ]; then
path=$1;
else
path="/tmp"
fi
echo "base path: $path"
print_folder_recurse $path
This doesn't answer your question directly, but you can solve your problem with a one-liner:
find /tmp \( -name "*.pdf" -o -name "*.doc" \) -type f -exec rm {} +
Some versions of find (GNU, BSD) have a -delete action which you can use instead of calling rm:
find /tmp \( -name "*.pdf" -o -name "*.doc" \) -type f -delete
For bash (since version 4.0):
shopt -s globstar nullglob dotglob
echo **/*".ext"
That's all.
The trailing extension ".ext" there to select files (or dirs) with that extension.
Option globstar activates the ** (search recursivelly).
Option nullglob removes an * when it matches no file/dir.
Option dotglob includes files that start wit a dot (hidden files).
Beware that before bash 4.3, **/ also traverses symbolic links to directories which is not desirable.
This method handles spaces well.
files="$(find -L "$dir" -type f)"
echo "Count: $(echo -n "$files" | wc -l)"
echo "$files" | while read file; do
echo "$file"
done
Edit, fixes off-by-one
function count() {
files="$(find -L "$1" -type f)";
if [[ "$files" == "" ]]; then
echo "No files";
return 0;
fi
file_count=$(echo "$files" | wc -l)
echo "Count: $file_count"
echo "$files" | while read file; do
echo "$file"
done
}
This is the simplest way I know to do this:
rm **/#(*.doc|*.pdf)
** makes this work recursively
#(*.doc|*.pdf) looks for a file ending in pdf OR doc
Easy to safely test by replacing rm with ls
The following function would recursively iterate through all the directories in the \home\ubuntu directory( whole directory structure under ubuntu ) and apply the necessary checks in else block.
function check {
for file in $1/*
do
if [ -d "$file" ]
then
check $file
else
##check for the file
if [ $(head -c 4 "$file") = "%PDF" ]; then
rm -r $file
fi
fi
done
}
domain=/home/ubuntu
check $domain
There is no reason to pipe the output of find into another utility. find has a -delete flag built into it.
find /tmp -name '*.pdf' -or -name '*.doc' -delete
The other answers provided will not include files or directories that start with a . the following worked for me:
#/bin/sh
getAll()
{
local fl1="$1"/*;
local fl2="$1"/.[!.]*;
local fl3="$1"/..?*;
for inpath in "$1"/* "$1"/.[!.]* "$1"/..?*; do
if [ "$inpath" != "$fl1" -a "$inpath" != "$fl2" -a "$inpath" != "$fl3" ]; then
stat --printf="%F\0%n\0\n" -- "$inpath";
if [ -d "$inpath" ]; then
getAll "$inpath"
#elif [ -f $inpath ]; then
fi;
fi;
done;
}
I think the most straightforward solution is to use recursion, in the following example, I have printed all the file names in the directory and its subdirectories.
You can modify it according to your needs.
#!/bin/bash
printAll() {
for i in "$1"/*;do # for all in the root
if [ -f "$i" ]; then # if a file exists
echo "$i" # print the file name
elif [ -d "$i" ];then # if a directroy exists
printAll "$i" # call printAll inside it (recursion)
fi
done
}
printAll $1 # e.g.: ./printAll.sh .
OUTPUT:
> ./printAll.sh .
./demoDir/4
./demoDir/mo st/1
./demoDir/m2/1557/5
./demoDir/Me/nna/7
./TEST
It works fine with spaces as well!
Note:
You can use echo $(basename "$i") # print the file name to print the file name without its path.
OR: Use echo ${i%/##*/}; # print the file name which runs extremely faster, without having to call the external basename.
Just do
find . -name '*.pdf'|xargs rm
If you can change the shell used to run the command, you can use ZSH to do the job.
#!/usr/bin/zsh
for file in /tmp/**/*
do
echo $file
done
This will recursively loop through all files/folders.
The following will loop through the given directory recursively and list all the contents :
for d in /home/ubuntu/*;
do
echo "listing contents of dir: $d";
ls -l $d/;
done
I'm writing a script to remove some build artifacts older than 1 week.
The files have names in the form artifact-1.1-200810391018.exe.
How do I go about removing only the files that are greater than 1 week old, excluding the time in hours and minutes at the end of the date-time-stamp?
Currently it is removing all of the files in the directory.
#!/bin/sh
NIGHTLY_LOCATIONS=( "/foo" "/bar" )
ARTIFACT_PREFIX="artifact-*-"
NUM_TO_KEEP=7
for home in $(seq 0 $((${#NIGHTLY_LOCATIONS[#]} - 1))); do
echo "Removing artifacts for" ${NIGHTLY_LOCATIONS[$location]}
for file in `find ${NIGHTLY_LOCATIONS[$location]} -name "$ARTIFACT_PREFIX*"`; do
keep=true
for day in $(seq 0 $((${NUM_TO_KEEP} - 1))); do
date=`date --date="$day days ago" +%Y%m%d`
echo $(basename $file ".exe") " = " $ARTIFACT_PREFIX$date
if [ "$(basename $file ".exe")" != "$ARTIFACT_PREFIX$date" ]; then
keep=false
fi
done
if [ !$keep ]; then
echo "Removing file"
rm -f $file
fi
done done
You mean, something along the line of:
find /path/to/files -name "artifact*" -type f -mtime +7 -exec rm {} \;
?
If you trust the mtime of the file, you can do it in a simple sweep with find:
find "${NIGHTLY_LOCATIONS}" -name $ARTIFACT_PREFIX -type f -mtime +7 -delete