Awk skip file if it doesn't exist - shell

I've spent some time trying to figure this out with various internet searches and digging through stackoverflow. I'll try to explain this as best as a noob can.
I have a script that searches a config repository directory that is populated with directories for every Juniper and Cisco router and switch we have deployed. Each device directory has a file or two that I'm interested in, "show.version" and "show.chassis.hardware", except when they don't. The second file, "show.chassis.hardware" is not a command that Cisco boxes has, so that file does not exist in Cisco device directories. There also isn't a naming scheme that can easily tell me if the device is Juniper or Cisco, which is also part of the reason why my script exists.
To make things more fun, different models and even software versions output the show.version file in different formats (for both Cisco and Juniper), so my awk is full of all the different fields we will encounter.
Script:
#!/usr/local/bin/zsh
svn="$HOME/svn/nw_config_data/"
case "$1" in
("-a")
hosts=""
;;
("-b")
hosts=".bb.domain.net"
;;
("-c")
hosts=".cpe.domain.net"
;;
("-e")
hosts=".etech.domain.net"
;;
("-k")
hosts=".core.domain.net"
;;
("-m")
hosts=".maint.domain.net"
;;
("-o")
hosts=".ohgov.domain.net"
;;
esac
dirs=($(ls -d $svn*$hosts*))
for hostdir in $dirs
do host=$(echo $hostdir | grep -Eo "(\w|\.|-)*$")
awk -v h=$host '/^Model/{m=$2} /^Model number/{m=$4} /^\*0/{m=$2}
/^JUNOS Base OS boot/{v=$5} /^Junos:/{v="["$2"]"} /^BOOTLDR:/{v=$7}
/^JUNOS EX Software Suite/{v=$5} /^ROM:/{v=$5} /^JUNOS Software
Release/{v=$4} /^Chassis/{s=$2} /^Motherboard serial number/{s=$3}
END {if(m!="number") {printf("%s %s %s %s\n",h,m,v,s)}}'
"$hostdir/show.version" "$hostdir/show.chassis.hardware"
done
What it looks like when I run the script:
% cver -b
device1-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
device2-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
awk: can't open file /home/clmbn eng2/a/rwalker/svn/nw_config_data/device3-e1.bb.domain.net/show.chassis.hardware
input record number 55, file /home/clmbn-eng2/a/rwalker/svn/nw_config_data/device3-e1.bb.domain.net/show.chassis.hardware
source line number 1
device4-r0.bb.domain.net m7i [13.3R6.5] Serial#
...
What I want it to look like
% cver -b
device1-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
device2-e0.bb.domain.net ex4300-24t [14.1X53-D25.2] Serial#
device3-e1.bb.domain.net C3750 12.1(14r)EA1a, Serial#
device4-r0.bb.domain.net m7i [13.3R6.5] Serial#
...
I have 13 directories that do not have the "show.chassis.hardware" file but do have the "show.version" file which does have all the information I need from it. I have one directory that has no files, but it doesn't matter because that device is going to be replaced.
From what I've been reading, awk might not be able to do this, but I have faith that someone out there knows a way to make it work. If my approach (shell & awk scripting) just isn't going to work and I need to do it in something else (Perl or Python for example) I'll be completely stuck until I can learn those enough to convert my script to one of those languages.
Also, we don't have bash installed on this server and I don't know when we will since I'm not the admin.

You need
-f file
true if file exists and is a regular file.
if [[ -f "$hostdir/show.version" && -f "$hostdir/show.chassis.hardware" ]]; then
# your awk command goes here...
awk '{ }' "$hostdir/show.version" "$hostdir/show.chassis.hardware"
else
echo "not enough files found"
fi
You may refer : http://zsh.sourceforge.net/Doc/Release/Conditional-Expressions.html
--edit--
That's cool that this bit of script skips the directories that doesn't
have the file, but it doesn't pull the information from the
"show.version" file and print that information anyway. So the output
shows device1, device2, device 4...
Here is code snippet
function myfunc(){
# replace with your awk
awk '{ print }' "$#"
}
if [[ -f "$hostdir/show.version" && -f "$hostdir/show.chassis.hardware" ]]; then
# your awk command goes here...
myfunc "$hostdir/show.version" "$hostdir/show.chassis.hardware"
else
echo "not enough files found"
# pass only one file, version file
myfunc "$hostdir/show.version"
fi

You can use zsh's (...|...) globbing:
$ mkdir foo bar; touch {foo,bar}/show.version foo/show.chassis.hardware
$ echo foo/show.(version|chassis.hardware)
foo/show.chassis.hardware foo/show.version
$ echo bar/show.(version|chassis.hardware)
bar/show.version
Since this is globbing, it will only expand to existing files. So your awk command would look like:
awk -v h=$host '/^Model/{m=$2} ... END {...}' "$hostdir"/show.(version|chassis.hardware)
(omitting the awk script for readability)
I'd also simplify your script a bit using associative arrays instead of cases:
#!/usr/local/bin/zsh
usage () {
echo "Help!" # echo your help message here
exit $1
}
svn="$HOME/svn/nw_config_data/"
declare -A hosts # make associative array
hosts["-a"]=""
hosts["-b"]=".bb.domain.net"
hosts["-c"]=".cpe.domain.net"
hosts["-e"]=".etech.domain.net"
hosts["-k"]=".core.domain.net"
hosts["-m"]=".maint.domain.net"
hosts["-o"]=".ohgov.domain.net"
if [[ $1 == -h ]]
then
usage
elif (( ${+hosts["$1"]} )) # check if $1 is a key in hosts
then
echo "Invalid option: $1"
usage 1 # exit with status 1 to indicate error
fi
dirs=( $svn*$hosts["$1"]* ) # no need for ls here
for hostdir in $dirs
do
host=$(echo $hostdir | grep -Eo "(\w|\.|-)*$")
awk -v h=$host '
/^Model/{m=$2}
/^Model number/{m=$4}
/^\*0/{m=$2}
/^JUNOS Base OS boot/{v=$5}
/^Junos:/{v="["$2"]"}
/^BOOTLDR:/{v=$7}
/^JUNOS EX Software Suite/{v=$5}
/^ROM:/{v=$5}
/^JUNOS Software Release/{v=$4}
/^Chassis/{s=$2}
/^Motherboard serial number/{s=$3}
END {if(m!="number") {printf("%s %s %s %s\n",h,m,v,s)}}' \
"$hostdir"/show.(version|chassis.hardware)
done
Alternately, you can use a concise case with the array:
declare -A hosts # make associative array
hosts["-a"]=""
hosts["-b"]=".bb.domain.net"
hosts["-c"]=".cpe.domain.net"
hosts["-e"]=".etech.domain.net"
hosts["-k"]=".core.domain.net"
hosts["-m"]=".maint.domain.net"
hosts["-o"]=".ohgov.domain.net"
case $1 in
-[abcekmno]) dirs=( $svn*${hosts["$1"]}* )
;;
-h|help) usage
;;
*) echo "Invalid option: $1"
usage 1 # exit with status 1 to indicate error
;;
esac

Related

(Ubuntu bash script) Setting rights from a config txt

I am a beginner and trying to write a script that takes a config file (example below) and sets the rights for the users, if that user or group doesn´t exist, they get added.
For every line in the file, I am cutting out the user or the group and check if they exist.
Right now I only check for users.
#!/bin/bash
function SetRights()
{
if [[ $# -eq 1 && -f $1 ]]
then
for line in $1
do
var1=$(cut -d: -f2 $line)
var2=$(cat /etc/passwd | grep $var1 | wc -l)
if [[ $var2 -eq 0 ]]
then
sudo useradd $var1
else
setfacl -m $line
fi
done
else
echo Enter the correct path of the configuration file.
fi
}
SetRights $1
The config file looks like this:
u:TestUser:- /home/temp
g:TestGroup:rw /home/temp/testFolder
u:TestUser2:r /home/temp/1234.txt
The output:
grep: TestGroup: No such file or directory
grep: TestUser: No such file or directory
"The useradd help menu"
If you could give me a hint what I should look for in my research, I would be very grateful.
Is it possible to reset var1 and var2? Using unset didn´t work for me and I couldn´t find variables could only be set once.
It's not clear how you are looping over the contents of the file -- if $1 contains the file name, you should not be seeing the errors you report.
But anyway, here is a refactored version which hopefully avoids your problems.
# Avoid Bash-only syntax for function definition
SetRights() {
# Indent function body
# Properly quote "$1"
if [[ $# -eq 1 && -f "$1" ]]
then
# Read lines in file
while read -r acl file
do
# Parse out user
user=${acl#*:}
user=${user%:*}
# Avoid useless use of cat
# Anchor regex correctly
if ! grep -q "^$user:" /etc/passwd
then
# Quote user
sudo useradd "$user"
else
setfacl -m "$acl" "$file"
fi
done <"$1"
else
# Error message to stderr
echo Enter the correct path of the configuration file. >&2
# Signal failure to the caller
return 1
fi
}
# Properly quote argument
SetRights "$1"

Cygwin save package selections for later reinstall

I was wondering if there is a way to save the current package selections for cygwin for a later reinstall or porting on a different system.
It would be really great to:
run a command to export a list of installed packages on an existing system
pass the list to the installer on another system in a way such as setup-x86_64.exe --list list.txt
I don't think the setup has such a switch, so even any type of script or batch working in this direction would be just fine.
Since the number of needed packages is very high, it should be unattended in order to consider it as a good solution!
What would be the best way to accomplish a quick reinstall like this?
The list of installed packages is available with cygcheck. Setup does not accept a list option but you can specific the list with -P
The following code, when used with -A option will create
a crafted cyg-reinstall-${Arch}.bat batch file to install all
packages existing in a system.
#!/bin/bash
# Create a batch file to reinstall using setup-{ARCH}.exe
# all packages reported as incomplete
print_error=1
if [ $# -eq 1 ]
then
if [ $1 == "-I" ]
then
lista=$(mktemp)
cygcheck -c | grep "Incomplete" > $lista
print_error=0
fi
if [ $1 == "-A" ]
then
lista=$(mktemp)
cygcheck -cd | sed -e "1,2d" > $lista
print_error=0
fi
fi
if [ $# -eq 2 ]
then
if [ $1 == "-f" ]
then
lista=$2
print_error=0
fi
fi
# error message if options are incorrect.
if [ $print_error -eq 1 ]
then
echo -n "Usage : " $(basename $0)
echo " [ -A | -I | -f filelist ]"
echo " create cyg-reinstall-{ARC}.bat from"
echo " options"
echo " -A : All packages as reported by cygcheck"
echo " -I : incomplete packages as reported by cygcheck"
echo " -f : packages in filelist (one per raw)"
exit 1
fi
if [ $(arch) == "x86_64" ]
then
A="x86_64"
else
A="x86"
fi
# writing header
echo -n -e "setup-${A}.exe " > cyg-reinstall-${A}.bat
# option -x remove and -P install
# for re-install packages we need both
if [ $1 == "-I" ]
then
awk 'BEGIN{printf(" -x ")} NR==1{printf $1}{printf ",%s", $1}' ${lista} >> cyg-reinstall-${A}.bat
fi
awk 'BEGIN{printf(" -P ")} NR==1{printf $1}{printf ",%s", $1} END { printf "\r\n pause "}' ${lista} >> cyg-reinstall-${A}.bat
# execution permission for the script
chmod +x cyg-reinstall-${A}.bat
I recognize that this question is several years old, but I've often found useful information on here from even longer ago, so this might still help someone someday.
The script above did not work for me; I suspect the list was too long, or something of that nature. So I kept trying things, and I eventually arrived at a shell one-liner that worked correctly by trimming the list to only those items that I had explicitly requested. The key came from #Andrey's comment above: /etc/setup/installed.db!
Here's the command I used:
(ORIG_PKGS="/path/to/other-cygwin64/etc/setup/installed.db" ; PKGS=$(awk '/ 1$/ {print $1}' "${ORIG_PKGS}") ; PLIST=$(tr '\n' ',' <<< "${PKGS}") ; /setup-x86_64 -q -P "${PLIST%%,}")
For readability, here it is split up into multiple lines:
ORIG_PKGS="/path/to/other-cygwin64/etc/setup/installed.db"
PKGS=$(awk '/ 1$/ {print $1}' "${ORIG_PKGS}")
PLIST=$(tr '\n' ',' <<< "${PKGS}")
/setup-x86_64 -q -P "${PLIST%%,}"
All you should need is the /etc/setup/installed.db from the previous Cygwin installation; just alter the value of ORIG_PKGS with the correct path to that file, and the rest should Just Work®!

Bash Loop with If Statements not Looping Properly

Quick bash script for awstats. Noting beautiful, but having an issue with loop not ocurring. Output only shows that it has performed one loop for name1, it never gets to name2, name3 with the printf's.
#!/bin/bash
awstats_command='perl /var/www/html/awstats/wwwroot/cgi-bin/awstats.pl'
html_path="/var/www/html/awstats/wwwroot"
activelogpath="/logs/web/active"
archivelogpath="/logs/web/archive"
day='date +%Y-%m-%d.%H'
# List of web servers we are processing stats for:
for i in name1 name2 name3
do
if [[ $i = "name1" ]]
then
# Custom reports for name1 contains subdirectory statistics
printf "\nProcessing log files for $i...\n"
/usr/bin/perl /var/www/html/awstats/wwwroot/cgi-bin/awstats.pl -config=name1 -update
printf "done.\n"
printf "\nGenerating .html files for $i...\n"
/var/www/html/awstats/wwwroot/cgi-bin/do.reports $i
$awstats_command -config=$i -output=urldetail:/about/ -staticlinks > $html_path/$i/awstats.$i.about.html
printf "done.\n"
else
printf "\nProcessing log files for $i...\n"
# Will do something when working $i
printf "done.\n"
printf "\nGenerating .html files for $i...\n"
# Will do something when working $i
printf "done.\n"
fi
printf "\nCompressing and archiving log files...\n"
exec /usr/bin/gzip -cv "$activelogpath"/"$i"/*.log > "$archivelogpath"/"$i"/"$(date +%Y%m%d_%H%M%S)".gz
# rm -f $activelogpath/$i/*.log
printf "\nCompleted!\n"
done
exec /usr/bin/gzip -cv "$activelogpath"/"$i"/*.log > "$archivelogpath"/"$i"/"$(date +%Y%m%d_%H%M%S)".gz
exec replaces the current process with the named program. Execution does not continue past an exec statement. There's no need for it here. Just call gzip without it.
gzip -cv "$activelogpath/$i"/*.log > "$archivelogpath/$i/$(date +%Y%m%d_%H%M%S).gz"
There's also no need to write /usr/bin/, or to leave and re-enter quotes so often.

How do I detect a failed state on multiple files when running md5sum on multiple files

I use:
md5sum * > checklist.chk # Generates a list of checksums and files.
and use:
md5sum -c checklist.chk # runs through the list to check them
How can I automate a PASS or FAIL state? I basically want to get a notification if something on my app changes. Whether by hacker or unauthorized change by a developer. I want to write a script that will notify of any changes to my code.
I found a few scripts online but they only appear to work for single files, I have been unable to adapt the script to work for multiple files with pass or fail states.
if [ "$(md5sum < File.name)" = "24f4ce42e0bc39ddf7b7e879a -" ]
then
echo Pass
else
echo Fail
fi
Reference:
Shell scripts and the md5/md5sum command: need to decide when to use which one
https://unix.stackexchange.com/questions/290240/md5sum-check-no-file
I would do something like:
for f in $(awk '{printf"%s ", $2}' checklist.chk); do
md5sum=$(grep $f checklist.chk | awk '{print $1}')
if [[ "$(md5sum < $f)" = "$md5sum -" ]]; then
echo Pass
else
echo Fail
fi
done
Store your checksums directly in the script. Then just run the md5sum -c.
Something like:
#!/bin/bash
get_stored_checksums() {
grep -P '^[0-9a-f]{32} .' <<-'EOF'
#########################################################
# Stored checksums in the script itself
# the output from the md5sum for the files you want guard
5e3f61b243679426d7f74c22b863438b Workbook1.xls
777a161c82fe0c810e00560411fb076e Workbook1.xlsx
# empty lines and comments - theyre simply ignored
d41d8cd98f00b204e9800998ecf8427e abc def.xxx
# this very important file
809f911bcde79d6d0f6dc8801d367bb5 jj.xxx
#########################################################
EOF
}
#MAIN
cd /where/the/files/are
#run the md5sum in the check-mode
result=$( md5sum -c <(get_stored_checksums) )
if (( $? ))
then
#found some problems
echo "$result"
#mail -s "PROBLEM" security.manager#example.com <<<"$result"
#else
# echo "all OK"
fi
If something is wrong, you will see something like:
Workbook1.xls: OK
Workbook1.xlsx: OK
abc def.xxx: FAILED
jj.xxx: OK
md5sum: WARNING: 1 of 4 computed checksums did NOT match
Of course, you can change the get_stored_checksums function to anything other, like:
get_stored_checksums() {
curl -s 'http://integrityserver.example.com/mytoken'
}
and you will fetch the guarded checksums from the remote server...

How to run my bash functions in terminal using a parent name?

* the wording of the question is terrible, sorry!
I have some bash functions I create
test() {echo "hello wold"}
test2() {echo "hello wold"}
Then in my .bashrc I source the file that has the above function . ~/my_bash_scripts/testFile
In the terminal I can run test and get hello world.
is there a way for me to add parent variable that holds all my functions together. For example personal test, personal test2.
Similar to every other gem out there, I downloaded a tweeter one. All it's methods are followed by the letter t, as in t status to write a status, instead of just status
You are asking about writing a command-line program. Just a simple one here:
#!/usr/bin/env bash
if [[ $# -eq 0 ]]; then
echo "no command specified"
exit
elif [[ $# -gt 1 ]]; then
echo "only one argument expected"
exit
fi
case "$1" in
test)
echo "hello, this is test1"
;;
test2)
echo "hello, this is test2"
;;
*)
echo "unknown command: $1"
;;
esac
Then save it and make it an executable by run chmod +x script.sh, and in your .bashrc file, add alias personal="/fullpath/to/the/script.sh".
This is just very basic and simple example using bash and of course you can use any language you like, e.g. Python, Ruby, Node e.t.c.
Use arguments to determine final outputs.
You can use "$#" for number of arguments.
For example,
if [ $# -ne 2 ]; then
# TODO: print usage
exit 1
fi
Above code exits if arguments not euqal to 2.
So below bash program
echo $#
with
thatscript foo bar baz quux
will output 4.
Finally you can combine words to determine what to put stdout.
If you want to flag some functions as your personal functions; no, there is no explicit way to do that, and essentially, all shell functions belong to yourself (although some may be defined by your distro maintainer or system administrator as system-wide defaults).
What you could do is collect the output from declare -F at the very top of your personal shell startup file; any function not in that list is your personal function.
SYSFNS=$(declare -F | awk '{ a[++i] = $3 }
END { for (n=1; n<=i; n++) printf "%s%s", (n>1? ":" : ""), a[n] }')
This generates a variable SYSFNS which contains a colon-separated list of system-declared functions.
With that defined, you can check out which functions are yours:
myfns () {
local fun
declare -F |
while read -r _ _ fun; do
case :$SYSFNS: in *:"$fun":*) continue;; esac
echo "$fun"
done
}

Resources