I've build this little script for finding files which was modified in a specified month.
My script worked fine for this last four years, but from a recent update, now with bash V4, this won't work anymore and I don't understand why.
There is the script (fmonth.sh):
#!/bin/sh
month=$1
shift
printf "now\n%s-01\n%s-01 00:00 +1 month\n" $month $month |
date -f - +%s |
xargs printf "n=%s/60-.5;s=%s/60;e=%s/60;n-s;if (n-e <0) 0 else n-e\n" |
bc -l |
xargs printf "find $# -mmin -%.0f -mmin +%.0f -print0\n" |
sh |
xargs -0 /bin/ls -ltrd |
less -S
And this could by used as following:
fmonth.sh 2012-09 /etc /home/user /root /var/lib
Ok, I found the answer: $# could be replaced by $*:
#!/bin/sh
month=$1
shift
printf "now\n%s-01\n%s-01 00:00 +1 month\n" $month $month |
date -f - +%s |
xargs printf "n=%s/60-.5;s=%s/60;e=%s/60;n-s;if (n-e <0) 0 else n-e\n" |
bc -l |
xargs printf "find $* -mmin -%.0f -mmin +%.0f -print0\n" |
sh |
xargs -0 /bin/ls -ltrd |
less -S
This work fine now!
I'm not sure to really understand why... for now... I would search for the reason later...
Related
I would like to add a progress bar to this command line:
find . \( -iname "*.bz" -o -iname "*.zip" -o -iname "*.gz" -o -iname "*.rar" \) -print0 | while read -d '' file; do echo "$file"; lzgrep -a stringtosearch\.anything "$file"; done
The progress file should be calculated on the total of compressed size files (not on the single file).
Of course, it can be a script too.
I would also like to add other progress bars, if possible:
The total number of files processed (example 3 out of 21)
The percentage of progress of the single file
Can anybody help me please?
Here some example of it should look alike (example from here):
tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '{print $1}') | gzip > big-files.tar.gz
Multiple progress bars (example from here):
pv -cN orig < foo.tar.bz2 | bzcat | pv -cN bzcat | gzip -9 | pv -cN gzip > foo.tar.gz
Thanks,
This is the first time I've ever heard of pv and it's not on any machine I have access to but assuming it needs to know a total at startup and then a number on each iteration of a command, you could do something like this to get a progress bar per file processed:
IFS= readarray -d '' files < <(find . -whatever -print0)
printf '%s\n' "${files[#]}" | pv -s "${#files[#]}" | command
The first line gives you an array of files so you can then use "${#files[#]}" to provide pv it's initial total value (looks like you use -s value for that?) and then do whatever you normally do to get progress as each file is processed.
I don't see any way to tell pv that the pipe it's reading from is NUL-terminated rather than newline-terminated so if your files can have newlines in their names then you'd have to figure out how to solve that problem.
To additionally get progress on a single file you might need something like:
IFS= readarray -d '' files < <(find . -whatever -print0)
printf '%s\n' "${files[#]}" |
pv -s "${#files[#]}" |
xargs -n 1 -I {} sh -c 'pv {} | command'
I don't have pv so all of the above is untested so check the syntax, especially since I've never heard of pv :-).
Thanks to Max C., I found a solution for the main question:
find ./ -type f -iname *\.gz -o -iname *\.bz | (tot=0;while read fname; do s=$(stat -c%s "$fname"); if [ ! -z "$s" ] ; then echo "$fname"; tot=$(($tot+$s)); fi; done; echo $tot) | tac | (read size; xargs -i{} cat "{}" | pv -s $size | lzgrep -a something -)
But this work only for gz and bz files, now I have to develop to use different tool according to extension.
I'm gonna to try the Ed solution too.
Thanks to ED and Max C., here the verision 0.2
This version work with zgrep, but not with lzgrep. :-\
#!/bin/bash
echo -n "collecting dump... "
IFS= readarray -d '' files < <(find . \( -iname "*.bz" -o -iname "*.gz" \) -print0)
echo done
echo "Calculating archives size..."
tot=0
for line in "${files[#]}"; do
s=$(stat -c\%s "$line")
if [ ! -z "$s" ]
then
tot=$(($tot+$s))
fi
done
(for line in "${files[#]}"; do
s=$(stat -c\%s "$line")
if [ ! -z "$s" ]
then
echo "$line"
fi
done
) | xargs -i{} sh -c 'echo Processing file: "{}" 1>&2 ; cat "{}"' | pv -s $tot | zgrep -a anything -
I have a bash script that counts compressed files by file extension and prints the count.
#!/bin/bash
FIND_COMPRESSED=$(find . -type f | sed -e 's/.*\.//' | sort | uniq -c | sort -rn | grep -Ei '(deb|tgz|tar|gz|zip)$')
COUNT_LINES=$($FIND_COMPRESSED | wc -l)
if [[ $COUNT_LINES -eq 0 ]]; then
echo "No archived files found!"
else
echo "$FIND_COMPRESSED"
fi
However, the script works only if there are NO files with .deb .tar .gz .tgz .zip.
If there are some, say test.zip and test.tar in the current folder, I get this error:
./arch.sh: line 5: 1: command not found
Yet, if I copy the contents of the FIND_COMPRESSED variable into the COUNT_LINES, all works fine.
#!/bin/bash
FIND_COMPRESSED=$(find . -type f | sed -e 's/.*\.//' | sort | uniq -c | sort -rn | grep -Ei '(deb|tgz|tar|gz|zip)$')
COUNT_LINES=$(find . -type f | sed -e 's/.*\.//' | sort | uniq -c | sort -rn | grep -Ei '(deb|tgz|tar|gz|zip)$'| wc -l)
if [[ $COUNT_LINES -eq 0 ]]; then
echo "No archived files found!"
else
echo "$FIND_COMPRESSED"
fi
What am I missing here?
So when you do that variable like that, it tries to execute it like a command, which is why it fails when it has contents. When it's empty, wc simply returns 0 and it marches on.
Thus, you need to change that line to this:
COUNT_LINES=$(echo $FIND_COMPRESSED | wc -l)
But, while we're at it, you can also simplify the other line with something like this:
FIND_COMPRESSED=$(find . -type f -iname "*deb" -or -iname "*tgz" -or -iname "*tar*") #etc
you can do
mapfile FIND_COMPRESSED < <(find . -type f -regextype posix-extended -regex ".*(deb|tgz|tar|gz|zip)$" -exec bash -c '[[ "$(file {})" =~ compressed ]] && echo {}' \;)
COUNT_LINES=${#FIND_COMPRESSED[#]}
I would like to check if my last file is older than 24 hours or not. (project in django)
I have many zip packages in directory so I have to 'filter' the last one with this part of code: ls -1 | sort -n | tail -n1.
My code in .sh file:
#!/bin/bash
file="$HOME/path_directory ls -1 | sort -n | tail -n1"
current=`date +%s`;
last_modified=`stat -c "%Y" $file`;
if [ $(($current-$last_modified)) -gt 86400 ]; then
echo "File is older that 24 hours" | mailx noreply#address -s "Older than 24 hours" me#mailmail.com
else
echo "File is up to date.";
fi;
Here is an error, that I got:
stat: invalid option -- '1'
Try 'stat --help' for more information.
/path_directory/imported_file.sh: line 9: 1538734802-: syntax error: operand expected (error token is "-")
If somebody made something similar, please some hint.
You can obtain the list of files in the directory that have been modified earlier than 1440 minutes (86400 seconds), you can use find for this:
find -maxdepth 1 -mmin +1440
It will thus select all files in this directory (no subdirectories), with a change time in minutes older than 1440.
The + in +1440 is important, since otherwise you will obtain files that are exactly 1440 minutes unmodified.
You can also use -mtime to specify the number in days:
find -maxdepth 1 -mtime +1
If you want all files (in this directory and subdirectories), you can remove the -maxdepth 1.
You can add -type f if you only want to include files, etc. For more flags, and (filtering) options, please read the manpage of find.
I'd advise you to try this:
if test "`find file -mtime +1`"
but if you insist you can fix it by changing it to this:
#!/bin/bash
file="$HOME/path_directory ls -1 | sort -n | tail -n1"
current=$(date +%s);
last_modified=$(stat -c "%Y" $file);
if [ $((current - last_modified)) -gt 86400 ]; then
echo "File is older that 24 hours" | mailx noreply#address -s "Older than 24 hours" me#mailmail.com
else
echo "File is up to date.";
fi;
The file variable are not well formed I belive that you want something like:
file=`find $HOME/path_directory | sort -n | tail -n1`
or
file=$( find $HOME/path_directory | sort -n | tail -n1)
If you like the moderm way
I need to delete the oldest folders (including their contents) from a certain path. E.g. if there are more than 10 directories, delete the oldest ones until you are below 8 directories. The log would show count of directories before/after + the filesystem before/after and what dirs were deleted.
Thank you in advance!
You should test this first on your backup directory,
#!/bin/bash
DIRCOUNT="$(find . -type d -printf x | wc -c)"
if [ "$DIRCOUNT" -gt 10 ]; then
ls -A1td */ | tail -n -8 | xargs rm -r
fi
if i do not misunderstanding your intentions, below is your answer
#! /usr/bin/env bash
DIRCOUNT="$(find . -maxdepth 1 -type d -printf x | wc -c)"
echo "Now you have $DIRCOUNT dirs"
[[ "$DIRCOUNT" -gt 10 ]] && ls -A1td */ | tail -n $((DIRCOUNT-8)) | xargs rm -r && echo "Now you have 8 dirs"
Hi Guys pls help on this...
[root#uenbe1 ~]# cat test.sh
#!/bin/bash
cd /vol/cdr/MCA
no='106'
value='55'
size=`df -kh | grep '/vol/cdr/MCA' | awk '{print $5}'| sed 's/%//g'`
if [ "$size" -gt "$value" ] ;
then
delete=$(($size-$value))
echo $delete
count=$(($no*$delete))
`ls -lrth | head -n $count | xargs rm -rf`
fi
output:
+ cd /vol/cdr/MCA
+ no=106
+ value=55
++ df -kh
++ grep /vol/cdr/MCA
++ awk '{print $5}'
++ sed s/%//g
+ size=63
+ '[' 63 -gt 55 ']'
+ delete=8
+ echo 8
8
+ count=848
++ ls -lrth
++ head -n 848
++ xargs rm -rf
rm: invalid option -- 'w'
Try `rm --help' for more information.``
i want to delete these files which in $count.
The command ls -lrth prints lines like:
-rw-r--r-- 1 bize bize 0 may 22 19:54 text.txt
-rw-r--r-- 1 bize bize 0 may 22 19:54 manual.pdf
that text given to the command rm will be interpreted as options
$ rm -rw-r text.txt
rm: invalid option -- 'w'
List only the name of files. That is: remove the long -l option from ls (and the -h option since it works only with -l):
$ ls -1rt | head -n "$count" | xargs
But Please: do not make a rm -rf automatic, that is a sure path to future problems.
Maybe?:
$ ls -1rt | head -n "$count" | xargs -I{} echo rm -rf /vol/cdr/MCA/'{}' \;
why are you passing
ls -l
use just, it will find the list of file greater than given size,
if you get this list in a file you can then take list of files which are to be deleted or whatever
find /vol/cdr/MCA -type f -size +56320c -exec ls '{}' \;
> `ls -lrth | head -n $count | xargs rm -rf`
This line has multiple problems. The backticks are superfluous, and you are passing the directory permission, file size, owner information etc as if that were part of the actual file name.
The minimal fix is to lose the backticks and the -l option to ls (and incidentally, the -r option to rm looks misplaced, too); but really, a proper solution would not use ls here at all.