bash console output print layout - bash

I want to make my writing to the console output in a nice tidy human readable.
here is how it looks now:
====================== Sat Apr 16 12:57:17 EDT 2022 ======================
==========================================================================
====================== Leopard - Download from S3 ======================
==========================================================================
==========================================================================
====================== Leopard - Decompressing ======================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
==========================================================================
====================== NewYorkCity - Download from S3 ======================
==========================================================================
==========================================================================
====================== NewYorkCity - Unloading SSL Example ======================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
I want that all the === line will be in the same length, and the text always in the center with 1 space on each side
Will appreciate assistance here :)
UPDATE / EDIT:
The original script is something like that:
eecho () { echo ==========================================================================; }
echo_stage () {
START=1
END=11
for (( c=$START; c<=$END; c++ ))
do
printf == '-%.0s'
done
echo -n " " $1 " "
for (( c=$START; c<=$END; c++ ))
do
printf == '-%.0s'
done
echo
}
stage() {
eecho
echo_stage "$1" "$2"
eecho
}
print_date () { echo "======================" $(date) "======================"; }
reload_db() {
print_date
rm -rf /var/tmp/db
mkdir -p /var/tmp/db
stage "DB - Download from S3"
aws s3 sync s3://db-backup/latest/ /var/tmp/db --profile=papilon --quiet
stage "DB - Decompressing"
pigz -d /var/tmp/db/*
stage "DB - Restoring Data"
cd /var/tmp/db
stage "DB - Restoring Tables"
for i in `ls -1 *.sql | grep -v "_view.sql"`;do echo $i;mysql db < $i;done
stage "DB - Restoring Views"
for i in `ls -1 *.sql | grep "_view.sql"`;do echo $i;mysql db < $i;done
stage "DB - Clean up"
rm -rf /var/tmp/db
print_date
}
reload_db

This awk filter will size your === padding correctly, and align titles to the center, if you pipe your command output through it:
# cmd |
awk '
BEGIN {a[1] = "="}
NF>1 && $1~/^=+$/ && $NF~/^=+$/ {
sub(/^=+/, "")
sub(/=+$/, "")
title_len=length($0)
pad = ""
for (i=1; i<=(74-title_len)/2; ++i) {
pad=pad"="
}
$0 = pad $0 pad a[title_len%2]
}
1'
This doesn't wrap the ls -l output, only the titles. It assumes 74 is hardcoded as the length of the solid === lines (and doesn't change depending on terminal size). The array a is used to add an extra = when 74 - title_len is an odd number.
Example output:
====================== Sat Apr 16 12:57:17 EDT 2022 ======================
==========================================================================
====================== Leopard - Download from S3 ======================
==========================================================================
==========================================================================
======================= Leopard - Decompressing ========================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz
==========================================================================
==================== NewYorkCity - Download from S3 ====================
==========================================================================
==========================================================================
================= NewYorkCity - Unloading SSL Example ==================
==========================================================================
total 1349872
drwxr-xr-x 2 root root 12288 Apr 16 12:57 .
drwxrwxrwt. 4 root root 102 Apr 16 12:57 ..
-rw-r--r-- 1 root root 185070885 Apr 16 12:03 asdasdasd.sql.gz
-rw-r--r-- 1 root root 40344632 Apr 16 12:03 asdasdas.sql.gz
-rw-r--r-- 1 root root 26631 Apr 16 12:03 asdad.sql.gz
-rw-r--r-- 1 root root 1679 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1237 Apr 16 12:03 asd.sql.gz
-rw-r--r-- 1 root root 5241900 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1144 Apr 16 12:03 asdasasd.sql.gz
-rw-r--r-- 1 root root 489312 Apr 16 12:03 asdasd.sql.gz
-rw-r--r-- 1 root root 1138 Apr 16 12:03 asdasdasd.sql.gz

Because you have now posted your script, I will add a new answer for pure bash.
title() {
local text pad
(( ${#1} > 70 )) && { echo "$1"; return; }
text=${1:+ }$1${1:+ }
pad=$( eval "printf %.1s ={1..$(( ( 74 - ${#text} ) / 2 ))}" )
echo "$pad$text$pad$( (( ${#text} % 2 )) && printf = )"
}
For title 'foo bar', this function prints foo bar (or any string up to 70 characters) with a space either side, centered and padded to 74 columns with =. If the string is longer than 70 characters it's too long to pad, so it's printed as is.
================================ foo bar =================================
With no argument, or an empty argument, it prints a solid line of 74 =:
==========================================================================
You can swap = for any single ASCII character.
You can swap 74 for any even number. Also for an odd number, if you change && printf to || printf in the last line. (also change the 70 to N - 4)
You can call title once for the date, or three times for the larger three line banners (see banner below).
Explanation:
${#text} is bash for "length of $text".
${1:+ } expands to a space, unless $1 is empty or unset. This allows us to add spaces to either end of the string, or exclude them for an empty string.
={1..10} expands to =1 =2 =3 ... =10, and printf %.1s prints the first character of each string. Combining these allows us to repeat a string (=) N times.
But we can't use a variable (or arithmetic) in {1..10} normally. So we need eval.
You probably heard eval is bad, and a security risk. That's often true, but here we are not passing any unknown data to eval (such as user input), and it's safe from code injection. (${#text} always expands to a single number)
So we make two bars, of length (74 - text-length) / 2, adding another = to the second bar if the text length is an odd number.
I also made a few changes to your script which you might consider. Apart from the date and titles, these have nothing to do with the padding. The title and banner functions will work on your old script.
title() {
local text pad
(( ${#1} > 70 )) && { echo "$1"; return; }
text="${1:+ }$1${1:+ }"
pad=$( eval "printf %.1s ={1..$(( ( 74 - ${#text} ) / 2 ))}" )
echo "$pad$text$pad$( (( ${#text} % 2 )) && printf = )"
}
banner() {
title
title "$1"
title
}
reload_db() {
local i
title "$(date)"
rm -rf /var/tmp/db || exit 1
mkdir -p /var/tmp/db || exit 1
banner 'DB - Download from S3'
aws s3 sync s3://db-backup/latest/ /var/tmp/db --profile=papilon --quiet
banner 'DB - Decompressing'
pigz -d /var/tmp/db/*
banner 'DB - Restoring Data'
cd /var/tmp/db || exit 1
banner 'DB - Restoring Tables'
GLOBIGNORE='*_view.sql'
for i in *.sql; do
echo "$i"
mysql db < "$i"
done
GLOBIGNORE=
banner 'DB - Restoring Views'
for i in *_view.sql; do
echo "$i"
mysql db < "$i"
done
banner 'DB - Clean up'
rm -rf /var/tmp/db
title "$(date)"
}
At the very least, you should exit early if cd, mkdir, or the first rm fail. Also, looping over an unquoted ls command sub is a bad idea. Instead you can use glob expansion (or find).
You could also concatenate the SQL scripts, but this only works if all the commands end with a semicolon (see Run multiple sql files in mysql batch):
big_title 'DB - Restoring Tables'
GLOBIGNORE='*_view.sql'
printf '%s\n' *.sql
cat *.sql | mysql db
GLOBIGNORE=
big_title 'DB - Restoring Views'
printf '%s\n' *_view.sql
cat *_view.sql | mysql db

Related

How to display number of items in a folder - Bash

When I execute the ls -l -h command, I get an output as show by the image below.
How can the number of the items in a folder be included in the output?
Update
The current output looks like this
total 41M
-rw-r--r-- 1 root root 41M Dec 20 09:56 completed_projects.bson
-rw-r--r-- 1 root root 213 Dec 20 09:57 completed_projects.metadata.json
drwxrwxr-x 2 adipster adipster 4.0K Jun 16 13:22 contents
-rw-rw-r-- 1 adipster adipster 13 Jun 16 13:20 file.py
drwxrwxr-x 4 adipster adipster 4.0K Jun 16 13:22 folder
drwxrwxr-x 2 adipster adipster 4.0K Jun 16 13:21 items
But I'll like to have another column indicating the number of items in a folder like this
total 41M
-rw-r--r-- 1 root root 41M Dec 20 09:56 completed_projects.bson
-rw-r--r-- 1 root root 213 Dec 20 09:57 completed_projects.metadata.json
drwxrwxr-x 2 adipster adipster 4.0K Jun 16 13:22 contents 235
-rw-rw-r-- 1 adipster adipster 13 Jun 16 13:20 file.py
drwxrwxr-x 4 adipster adipster 4.0K Jun 16 13:22 folder 19
drwxrwxr-x 2 adipster adipster 4.0K Jun 16 13:21 items 5
where the numbers at the extreme right represents the number of items in a folder
You can do something like this:
echo -n "Number of files in folder is: " && ls | wc -l && ls -l
ouptut should be something like this:
umber of files in folder is: 3
Total 279K
-rwxr-xr-x 1 user users 19K Jun 16 00:17 a
-rwxr-xr-x 1 user users 5K Jun 16 00:17 b
-rwxr-xr-x 1 user users 255K Jun 16 00:17 c
You can omit echo statement, just as a note -n is no new line flag.
sed has an option to execute the constructed replacement with /e.
We only count subdirs, looking at the first character
ls -l | sed -r 's/d(.*) ([^ ]*)/printf "d%s %-20s%s\n" "\1" \2 $(ls \2| wc -l)/e'
EDIT: Solution for directories with spaces in their name.
Parsing ls should be avoided. When you try to fix above cmmand for directory names with spaces, you might try
# Don't do this
ls -l | sed -r 's/d(.{,48}) (.*)/printf "d%s %-20s%s\n" "\1" "\2" $(ls "\2"| wc -l)/e'
It is time to write a script. Perhaps with find or something like
#/bin/bash
for i in *; do
printf "%-70s %s\n" "$(/bin/ls -ld "$i")" "$(/bin/ls -d "$i"/* 2>/dev/null| wc -l)"
done
The wc in the subdir will count wrong when filenames have newlines.
ls() { command ls "$#" | tee >(echo "$(wc -l) items"); }
That uses an output process substitution to run the little "echo" script on its stdin while also displaying stdin (thanks to tee). This way, you don't have to run ls twice.
Usual caveat: output will be incorrect when there's a file with a newline in the name.

Clean ten days and older logs using Shell

[root#amp logs]# ls -l
total 0
-rw-r--r-- 1 root root 0 Nov 23 17:51 lb-quarzcenter.log
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-01
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-02
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-03
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-04
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-05
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-06
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-07
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-08
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-10
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-11
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-12
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-13
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-14
-rw-r--r-- 1 root root 0 Nov 23 17:51 lb-quarzcenter.log.2019-02-15
-rw-r--r-- 1 root root 0 Feb 27 17:26 lb-quarzcenter.log.2019-02-09
How do I match a string with year-month-date and delete 10 days ago file according to lb-quarzcenter.log.*?
now=$(date +%s) # express current time as seconds since 1970-01-01
(( ten_days_ago = now - 60*60*24*10 )) # subtract 864000 seconds (10 days) from that
date_minus_ten=$(date +%F --date="#$ten_days_ago") # express that number as a YYYY-MM-DD
for filename in lb-quartzcenter.log.* ; do # loop over all matching files
filedate="${filename/lb-quartzcenter.log./}" # remove the filename part before the timestamp
if [[ $filedate < $date_minus_ten ]] ; then # if filename remainder is lexicographically lower...
rm -f "$filename" # ... remove the file
fi
done

MacOS Command Line - Create folder with files at once

Is it possible to write a command that will create a new directory with name passed as argument 'MyFolder' (for example) and will create four files with the same name (as part):
MyFolder.js
MyFolder.css
MyFolder.test.js
README.md
(using mkdir / touch / echo ...)
Main problem - one line command
This one-liner function should do the work:
$ function mkdir_and_files() { mkdir "${1}"; touch ${1}/${1}.js; touch ${1}/${1}.css; touch ${1}/${1}.test.js; touch ${1}/README.md; }; mkdir_and_files "MyFolder" ;
$ ls -latrh MyFolder/
total 0
drwxrwxrwt 15 root wheel 480B Aug 19 18:58 ..
-rw-r--r-- 1 user wheel 0B Aug 19 18:58 MyFolder.js
-rw-r--r-- 1 user wheel 0B Aug 19 18:58 MyFolder.css
-rw-r--r-- 1 user wheel 0B Aug 19 18:58 MyFolder.test.js
-rw-r--r-- 1 user wheel 0B Aug 19 18:58 README.md
drwxr-xr-x 6 user wheel 192B Aug 19 18:58 .
Try this:
populate_dir() { mkdir "$1"; touch "$1/$1".{js,css,test.js} "$1/README.md"; }
populate_dir MyFolder

How can I rename of old files?

I have a directory like this:
-rw-r--r-- 1 root root 0 Jan 7 15:04 tmp_FILE2015_123_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 15:04 tmp_FILE2015_133_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1_3125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1__3223125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1_3223125.LOG
I need to remove tmp_ and I can do like this:
for i in *; do s=$(sed -r 's/^(tmp_)(.*.LOG)/\2/' <<< $i); if [[ "$i" != "$s" ]]; then mv "$i" "$s"; fi; done;
But I need to do this for just older than 1 hour (modified time) files:
For example ( now: Jan 7 16:10 ):
-rw-r--r-- 1 root root 0 Jan 7 13:00 FILE2015_123_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 15:04 FILE2015_133_1_3123.LOG
-rw-r--r-- 1 root root 0 Jan 7 15:01 FILE2015_133_1_3125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:04 tmp_FILE2015_133_1__3223125.LOG
-rw-r--r-- 1 root root 0 Jan 7 16:10 tmp_FILE2015_133_1_3223125.LOG
How can I do that?
This will operate on all files modified in the last hour:
for orig_file in $(find . -type f -depth 1 -mtime -60m); do
new_file="${orig_file#./tmp_}"
if [[ "$new_file" != "$orig_file" ]]; then
mv "$orig_file" "$new_file";
fi;
done
If you want to operate on files older than one hour use +60m instead
I've changed the use of sed to use some built in bash functionality.

How do I batch renumber files in sequence?

I have a sequence of image files that look like this:
image-149454.jpg
image-149455.jpg
I have some other images that I want to append to the end of the sequence where it left off but currently they are numbered from 0 (i.e. image-000000 to image-010000).
What's a script I could use to rename those new images starting from a certain number and going on, in this case 149456 and onwards?
I would try something like this. You just have to adjust the offset and add a string in front of the name, just as you need it.
x=1
for i in *.jpg; do
temp=$(printf "%08d.jpg" ${x}) #padding since you seem to want it
mv ${i} ${temp}
let x=x+1
done
Here's an untested solution in Python. This should change all of the files in a directory to have sequentially higher numbers than the other.
To use it enter: script.py dir1 dir2
Assuming the original files have the higher number (149455) are in dir1 and the new files start from 000000 are in dir2:
import os, sys, re
max_image = 0
# check if current (with higher numbers) and other (with lower) directory is given
if len(sys.argv) == 3:
for files in os.listdir(sys.argv[1]): # first dir is current
if files.endswith(".jpg"):
#for all jpgs get the max
m = re.search("\-(\d+)", files)
number = int(m.group(1))
if max_image < number:
max_image = number
for files in os.listdir(sys.argv[2]): # second dir is other
if files.endswith(".jpg"):
#get current start point
m = re.search("\-(\d+)", files)
number = int(m.group(1))
os.rename(files, "image-" + str(number + max_image)+".jpg") # add the max from current folder
Here's a pure bash solution which uses the last image sequence as your starting point (per your request):
#!/bin/bash
last_seq=$(ls image-* | tail -1 | sort -n | cut -c7-) # grab the sequence number
last_seq=${last_seq%.jpg} # remove the trailing .jpg
if [ -z $last_seq ] ; then
echo "Unable to obtain the last image sequence number"
exit 1
fi
for image in image-0*.jpg ; do
[ -f ${image} ] || break # In case no files match image-0*.jpg
let last_seq=last_seq+1
mv -v ${image} image-$(printf "%06d.jpg" ${last_seq})
done
Here's how it worked locally:
before the run:
~/tmp › ls -l
total 80
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-00000.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-01000.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14908.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14909.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14910.jpg
-rw-r--r-- 1 dyoung staff 418 Dec 12 10:24 testh.sh
during the run:
~/tmp › sh ./testh.sh
image-00000.jpg -> image-14911.jpg
image-01000.jpg -> image-14912.jpg
after the run:
~/tmp › ls -l
total 80
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14908.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14909.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:05 image-14910.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-14911.jpg
-rw-r--r-- 1 dyoung staff 0 Dec 12 10:20 image-14912.jpg
-rw-r--r-- 1 dyoung staff 429 Dec 12 12:25 testh.sh

Resources