I have a directory like this:
-rw-r--r-- 1 root root 0 Jan 7 15:04 tmp_FILE2015_123_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 15:04 tmp_FILE2015_133_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1_3125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1__3223125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1_3223125.LOG
I need to remove tmp_ and I can do like this:
for i in *; do s=$(sed -r 's/^(tmp_)(.*.LOG)/\2/' <<< $i); if [[ "$i" != "$s" ]]; then mv "$i" "$s"; fi; done;
But I need to do this for just older than 1 hour (modified time) files:
For example ( now: Jan 7 16:10 ):
-rw-r--r-- 1 root root 0 Jan 7 13:00 FILE2015_123_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 15:04 FILE2015_133_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 15:01 FILE2015_133_1_3125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1__3223125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:10 tmp_FILE2015_133_1_3223125.LOG
How can I do that?
This will operate on all files modified in the last hour:
for orig_file in $(find . -type f -depth 1 -mtime -60m); do
new_file="${orig_file#./tmp_}"
if [[ "$new_file" != "$orig_file" ]]; then
mv "$orig_file" "$new_file";
fi;
done
If you want to operate on files older than one hour use +60m instead
I've changed the use of sed to use some built in bash functionality.
Related
I want to make my writing to the console output in a nice tidy human readable.
here is how it looks now:
====================== Sat Apr 16 12:57:17 EDT 2022 ======================
==========================================================================
====================== Leopard - Download from S3 ======================
==========================================================================
==========================================================================
====================== Leopard - Decompressing ======================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
==========================================================================
====================== NewYorkCity - Download from S3 ======================
==========================================================================
==========================================================================
====================== NewYorkCity - Unloading SSL Example ======================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
I want that all the === line will be in the same length, and the text always in the center with 1 space on each side
Will appreciate assistance here :)
UPDATE / EDIT:
The original script is something like that:
eecho () { echo ==========================================================================; }
echo_stage () {
START=1
END=11
for (( c=$START; c<=$END; c++ ))
do
printf == '-%.0s'
done
echo -n " " $1 " "
for (( c=$START; c<=$END; c++ ))
do
printf == '-%.0s'
done
echo
}
stage() {
eecho
echo_stage "$1" "$2"
eecho
}
print_date () { echo "======================" $(date) "======================"; }
reload_db() {
print_date
rm -rf /var/tmp/db
mkdir -p /var/tmp/db
stage "DB - Download from S3"
aws s3 sync s3://db-backup/latest/ /var/tmp/db --profile=papilon --quiet
stage "DB - Decompressing"
pigz -d /var/tmp/db/*
stage "DB - Restoring Data"
cd /var/tmp/db
stage "DB - Restoring Tables"
for i in `ls -1 *.sql | grep -v "_view.sql"`;do echo $i;mysql db < $i;done
stage "DB - Restoring Views"
for i in `ls -1 *.sql | grep "_view.sql"`;do echo $i;mysql db < $i;done
stage "DB - Clean up"
rm -rf /var/tmp/db
print_date
}
reload_db
This awk filter will size your === padding correctly, and align titles to the center, if you pipe your command output through it:
# cmd |
awk '
BEGIN {a[1] = "="}
NF>1 && $1~/^=+$/ && $NF~/^=+$/ {
sub(/^=+/, "")
sub(/=+$/, "")
title_len=length($0)
pad = ""
for (i=1; i<=(74-title_len)/2; ++i) {
pad=pad"="
}
$0 = pad $0 pad a[title_len%2]
}
1'
This doesn't wrap the ls -l output, only the titles. It assumes 74 is hardcoded as the length of the solid === lines (and doesn't change depending on terminal size). The array a is used to add an extra = when 74 - title_len is an odd number.
Example output:
====================== Sat Apr 16 12:57:17 EDT 2022 ======================
==========================================================================
====================== Leopard - Download from S3 ======================
==========================================================================
==========================================================================
======================= Leopard - Decompressing ========================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
==========================================================================
==================== NewYorkCity - Download from S3 ====================
==========================================================================
==========================================================================
================= NewYorkCity - Unloading SSL Example ==================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
Because you have now posted your script, I will add a new answer for pure bash.
title() {
local text pad
(( ${#1} > 70 )) && { echo "$1"; return; }
text=${1:+ }$1${1:+ }
pad=$( eval "printf %.1s ={1..$(( ( 74 - ${#text} ) / 2 ))}" )
echo "$pad$text$pad$( (( ${#text} % 2 )) && printf = )"
}
For title 'foo bar', this function prints foo bar (or any string up to 70 characters) with a space either side, centered and padded to 74 columns with =. If the string is longer than 70 characters it's too long to pad, so it's printed as is.
================================ foo bar =================================
With no argument, or an empty argument, it prints a solid line of 74 =:
==========================================================================
You can swap = for any single ASCII character.
You can swap 74 for any even number. Also for an odd number, if you change && printf to || printf in the last line. (also change the 70 to N - 4)
You can call title once for the date, or three times for the larger three line banners (see banner below).
Explanation:
${#text} is bash for "length of $text".
${1:+ } expands to a space, unless $1 is empty or unset. This allows us to add spaces to either end of the string, or exclude them for an empty string.
={1..10} expands to =1 =2 =3 ... =10, and printf %.1s prints the first character of each string. Combining these allows us to repeat a string (=) N times.
But we can't use a variable (or arithmetic) in {1..10} normally. So we need eval.
You probably heard eval is bad, and a security risk. That's often true, but here we are not passing any unknown data to eval (such as user input), and it's safe from code injection. (${#text} always expands to a single number)
So we make two bars, of length (74 - text-length) / 2, adding another = to the second bar if the text length is an odd number.
I also made a few changes to your script which you might consider. Apart from the date and titles, these have nothing to do with the padding. The title and banner functions will work on your old script.
title() {
local text pad
(( ${#1} > 70 )) && { echo "$1"; return; }
text="${1:+ }$1${1:+ }"
pad=$( eval "printf %.1s ={1..$(( ( 74 - ${#text} ) / 2 ))}" )
echo "$pad$text$pad$( (( ${#text} % 2 )) && printf = )"
}
banner() {
title
title "$1"
title
}
reload_db() {
local i
title "$(date)"
rm -rf /var/tmp/db || exit 1
mkdir -p /var/tmp/db || exit 1
banner 'DB - Download from S3'
aws s3 sync s3://db-backup/latest/ /var/tmp/db --profile=papilon --quiet
banner 'DB - Decompressing'
pigz -d /var/tmp/db/*
banner 'DB - Restoring Data'
cd /var/tmp/db || exit 1
banner 'DB - Restoring Tables'
GLOBIGNORE='*_view.sql'
for i in *.sql; do
echo "$i"
mysql db < "$i"
done
GLOBIGNORE=
banner 'DB - Restoring Views'
for i in *_view.sql; do
echo "$i"
mysql db < "$i"
done
banner 'DB - Clean up'
rm -rf /var/tmp/db
title "$(date)"
}
At the very least, you should exit early if cd, mkdir, or the first rm fail. Also, looping over an unquoted ls command sub is a bad idea. Instead you can use glob expansion (or find).
You could also concatenate the SQL scripts, but this only works if all the commands end with a semicolon (see Run multiple sql files in mysql batch):
big_title 'DB - Restoring Tables'
GLOBIGNORE='*_view.sql'
printf '%s\n' *.sql
cat *.sql | mysql db
GLOBIGNORE=
big_title 'DB - Restoring Views'
printf '%s\n' *_view.sql
cat *_view.sql | mysql db
I am trying to grep some pattern in a file set under a folder like below
Where on the output I have to perform remaining operation.
The output main.log is coming so huge almost 50k lines ,as the files starting with server02.log are almost 30 to 40 in number . The script based on this output is taking forever to complete.
Is there a way that I can only take files name starting with server02.log. and generated between time
20:00:00 and 21:00:00
ls -lrth server02.log.*
-rw-r--r-- 1 user user 1.9M Apr 15 20:20 server02.log.2020
-rw-r--r-- 1 user user 1.7M Apr 15 20:30 server02.log.2030
-rw-r--r-- 1 user user 1.6M Apr 15 20:41 server02.log.2041
-rw-r--r-- 1 user user 1.9M Apr 15 20:50 server02.log.2050
-rw-r--r-- 1 user user 2.1M Apr 15 21:00 server02.log.2100
-rw-r--r-- 1 user user 1.4M Apr 15 21:10 server02.log.2110
-rw-r--r-- 1 user user 1.9M Apr 15 21:20 server02.log.2120
-rw-r--r-- 1 user user 656K Apr 15 21:29 server02.log.2129
-rw-r--r-- 1 user user 4.6M Apr 15 21:40 server02.log.2140
-rw-r--r-- 1 user user 1.9M Apr 15 21:50 server02.log.2150
-rw-r--r-- 1 user user 1.7M Apr 15 21:59 server02.log.2159
-rw-r--r-- 1 user user 724K Apr 15 22:09 server02.log.2209
-rw-r--r-- 1 user user 1.3M Apr 15 22:20 server02.log.2220
-rw-r--r-- 1 user user 1.1M Apr 15 22:29 server02.log.2229
-rw-r--r-- 1 user user 1.7M Apr 15 22:41 server02.log.2241
-rw-r--r-- 1 user user 1.5M Apr 15 22:49 server02.log.2249
-rw-r--r-- 1 user user 2.4M Apr 15 23:01 server02.log.2301
-rw-r--r-- 1 user user 1.4M Apr 15 23:10 server02.log.2310
-rw-r--r-- 1 user user 585K Apr 15 23:19 server02.log.2319
-rw-r--r-- 1 user user 858K Apr 15 23:30 server02.log.2330
-rw-r--r-- 1 user user 892K Apr 15 23:40 server02.log.2340
-rw-r--r-- 1 user user 698K Apr 15 23:49 server02.log.2349
grep -E "###Update |###Initiate |###Re-Initiate " server02.log.* >> main.log
from the comments I made the change to my code as below
#!/bin/bash
DIR="."
d=$(date +%Y-%m-%d);
log_dir="logs/$d"
PREFIX="$log_dir/srv_02.log"
#PREFIX="srv_02.log"
echo "prefix value is $PREFIX"
START_HOUR="06"
for F in "$( find "$DIR" -name "${PREFIX}*" -printf '%Tc %p\n' | grep "\ ${START_HOUR}:" )"; do
echo "F value is $F"
grep -E "###Update |###Initiate |###Re-Initiate" "$F" >> main.log
done
error:
prefix value is logs/2021-04-16/srv_02.log
find: warning: Unix filenames usually don't contain slashes (though pathnames do). That means that '-name `logs/2021-04-16/srv_02.log*'' will probably evaluate to false all the time on this system. You might find the '-wholename' test more useful, or perhaps '-samefile'. Alternatively, if you are using GNU grep, you could use 'find ... -print0 | grep -FzZ `logs/2021-04-16/osbpd_srv_02.log*''.
F value is
grep: : No such file or directory
This solution looks for files in the given directory, created during the specified hour with names matching the given prefix.
#!/bin/bash
d=$(date +%Y-%m-%d)
DIR="logs/$d/$log_dir"
PREFIX="srv_02.log"
#PREFIX=server02.log
echo "prefix value is $PREFIX"
START_HOUR="06"
for F in "$( find "$DIR" -name "${PREFIX}*" -printf '%TY-%Tm-%Td\n' | grep "\ ${START_HOUR}:" )"; do
echo "$F"
# grep -E "###Update |###Initiate |###Re-Initiate Assignment Milestone|###Complete Assignment Milestone|###Cancel Assignment Milestone|###Suspend Assignment Milestone|###Resume Assignment Milestone" "$F" >> main.log
done
I am trying to create files with different date created:
$ touch -t 20{11..15}01120000 file_{1..5}.txt
$ ls -al
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 201201120000
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 201301120000
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 201401120000
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 201501120000
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 file_1.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 file_2.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 file_3.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 file_4.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 file_5.txt
As you can see all file's dates created are 2011 Jan 12 0.
How can I create files with a different year?
# this is what I want
-rw-r--r-- 1 shinokada staff 0 Jan 12 2011 file_1.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2012 file_2.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2013 file_3.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2014 file_4.txt
-rw-r--r-- 1 shinokada staff 0 Jan 12 2015 file_5.txt
What is the best way?
touch command allows you to enter one one timestamp using -t option.
A traditional for-loop would be better:
for i in {1..5}; do touch -t 201${i}01120000 file_$i.txt; done
Shortly, but keeping filenumbers and years as separated variables
I think this is more readable, but...
Care about timezone!
filenum=1
for year in {2011..2015};do
TZ=UTC touch -t ${year}12312345 file-$((filenum++))
done
Then if you look about this, using a different timezone:
(Note that created date was Dec 31, 23h45')
TZ=UTC-1 ls -ltr
-rw-r--r-- 1 user user 0 jan 1 2012 file-1
-rw-r--r-- 1 user user 0 jan 1 2013 file-2
-rw-r--r-- 1 user user 0 jan 1 2014 file-3
-rw-r--r-- 1 user user 0 jan 1 2015 file-4
-rw-r--r-- 1 user user 0 jan 1 2016 file-5
Could you please try following, I would go with following approach with a for loop. Where I am providing year, number of files which needed, output file's initial value and same time value for all output files so that we can manage it in for loop.
cat script.bash
year=2011
numberoffiles="10"
time="01120000"
outputfileInitials="file"
nameSequence="1"
for ((i = 1 ; i <= numberoffiles ; i++ ));
do
touch -t $year$time "${outputfileInitials}_$nameSequence.txt"
(( nameSequence = nameSequence + 1 ))
(( year = year + 1 ))
done
[root#amp logs]# ls -l
total 0
-rw-r--r-- 1 root root 0 Nov 23 17:51 lb-quarzcenter.log
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-01
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-02
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-03
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-04
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-05
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-06
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-07
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-08
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-10
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-11
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-12
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-13
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-14
-rw-r--r-- 1 root root 0 Nov 23 17:51 lb-quarzcenter.log.2019-02-15
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-09
How do I match a string with year-month-date and delete 10 days ago file according to lb-quarzcenter.log.*?
now=$(date +%s) # express current time as seconds since 1970-01-01
(( ten_days_ago = now - 60*60*24*10 )) # subtract 864000 seconds (10 days) from that
date_minus_ten=$(date +%F --date="#$ten_days_ago") # express that number as a YYYY-MM-DD
for filename in lb-quartzcenter.log.* ; do # loop over all matching files
filedate="${filename/lb-quartzcenter.log./}" # remove the filename part before the timestamp
if [[ $filedate < $date_minus_ten ]] ; then # if filename remainder is lexicographically lower...
rm -f "$filename" # ... remove the file
fi
done
I'm using a binary to convert a file type to another:
./rcc -binary themes/redTheme.qrc -o redTheme.rcc
I have several .qrc files and would like to convert them as well, but in the same command. How to regroup them rename and their type? I'm struggling to find a way to keep the file name.
I was thinking about binding my rcc binary to a pipe that would execute a loop converting the .qrc files.
I hope you can bind this with your binary:
ls *.qrc | xargs -I {} sh -c 'mv $1 `basename $1 .html`.rcc' - {}
You could use something like this:
$ for i in *.qrc ; do echo mv -v ${i} ${i%.qrc}.new ; done
See here an example where I have a couple of files with qrc ending and I mv them to .new
$ l
total 0
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 1.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 10.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 2.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 3.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 4.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 5.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 6.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 7.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 8.qrc
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 9.qrc
$ for i in *.qrc ; do mv -v ${i} ${i%.qrc}.new ; done
1.qrc -> 1.new
10.qrc -> 10.new
2.qrc -> 2.new
3.qrc -> 3.new
4.qrc -> 4.new
5.qrc -> 5.new
6.qrc -> 6.new
7.qrc -> 7.new
8.qrc -> 8.new
9.qrc -> 9.new
$ l
total 0
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 1.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 10.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 2.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 3.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 4.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 5.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 6.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 7.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 8.new
-rw-r--r-- 1 chl wheel 0 Mar 9 16:22 9.new
$
Of course you have to use your rcc tool instead of mv and use your ending instead of .new