I have written the following bash script which is designed to create a file if the UFW rules get changed on a server. The existence of this file will then be checked with Zabbix.
#!/bin/bash
file="/tmp/ports.txt"
file_open="/tmp/open_ports.txt"
md5_file=$(md5sum /tmp/ports.txt | awk '{ print $1 }')
md5_file_open=$(md5sum /tmp/open_ports.txt | awk '{ print $1 }')
file_diff="/tmp/ports_diff"
if [[ ! -f $file ]]; then
touch $file && sudo ufw status|grep ALLOW > $file
fi
if [ -f $file_diff ];then
rm $file_diff
fi
sudo ufw status|grep ALLOW > $file_open
if [ $md5_file != $md5_file_open ];then
touch $file_diff
fi
What i'm finding is that sometimes the file doesnt get created or deleted when it should but if I run the command the 2nd or 3rd time without anything further changing, it does.
Please help.
Thanks
During the first call the files "/tmp/ports.txt" and "/tmp/open_ports.txt" don't exist. After the first run both files should be created and the diff is empty.
Afer this, the ufw status changes.
In the next run, the statements
md5_file=$(md5sum /tmp/ports.txt | awk '{ print $1 }')
md5_file_open=$(md5sum /tmp/open_ports.txt | awk '{ print $1 }')
will process the original files (without the changed status), so they will be operating on the files that are equal. No diff will be found.
During this run $file_open will be filled with the new value, but the md5sum function is not called after this change. The diff wile will nogt be made.
The next run will start with the changed $file_open, and the difference will be found.
When the status is changed back to the original value, the first run the md5 on the old files, claining to see a difference and not deleting the diff file. This run will also write the $file_open with the new values, that will be detected the next run.
Solution:
Move the 2 md5sum lines until after the last sudo.
Related
"get a domain from a url" is quite a common question here on this site and the answer I have used for a long time is from this question:
How to extract domain name from url?
The most popular answer has a comment from user "sakumatto" which also handles sub-domains too, it is this:
echo http://www.test.example.com:3030/index.php | sed -e "s/[^/]*\/\/\([^#]*#\)\?\([^:/]*\).*/\2/" | awk -F. '{print $(NF-1) "." $NF}'
How would I further extend this command to exclude ".com" or ".co.uk" etc???
Insight:
I am writing a bash script for an amazing feature that Termux (Terminal emulator for Android) has, "termux-url-opener" that allows one to write a script that is launched when you use the native Android "share" feature, lets say i'm in the browser, github wants me to login, I press "share", then select "Termux", Termux opens and runs the script, echos the password to my clipboard and closes, now im automatically back in the browser with my password ready to paste!
Its very simple and uses pass (password-store) with pass-clip extension, gnupg and pinentry here is what I have so far which works fine, but currently its dumb (it would need me to continue writing if/elif statements for every password I have in pass) so I would like to automate things, all I need is to cut ".com" or ".co.uk" etc.
Here is my script so far:
#!/data/data/com.termux/files/usr/bin/bash
URL="$1"
WEBSITE=$(echo "$URL" | sed -e "s/[^/]*\/\/\([^#]*#\)\?\([^:/]*\).*/\2/" | awk -F. '{print $(NF-1) "." $NF}')
if [[ $WEBSITE =~ "github" ]]; then
# website contains "github"
pass -c github
elif [[ $WEBSITE =~ "codeberg" ]]; then
# website contains "codeberg"
pass -c codeberg
else
# is another app or website, so list all passwords entries.
pass clip --fzf
fi
As my pass password entries are just website names e.g "github" or "codeberg" if I could cut the ".com" or ".co.uk" from the end then I could add something like:
PASSWORDS=$(pass ls)
Now I can check if "$1" (my shared URL) is a listed within pass ls and this stops having to write:
elif [[ $WEBSITE =~ "codeberg" ]]; then
For every single entry in pass.
Thank you! its really appreciated!
i might be missing something, but why don't you just strip the offending TLDs from the hostname?
as in:
sed \
-e "s|[^/]*//\([^#]*#\)\?\([^:/]*\).*|\2|" \
-e 's|\.$||' \
-e 's|\.com$||' \
-e 's|\.co\.[a-zA-Z]*$||' \
-e 's|.*\.\([^.]*\.[^.]*\)|\1|'
"s|[^/]*//\([^#]*#\)\?\([^:/]*\).*|\2|" - this is your original regex, but using | as delimiter rather than / (gives you less quoting)
's|\.$||' - drop any accidently trailing dot (example.com. is a valid hostname!)
's|\.com$||' - remove trailing .com
's|\.co\.[a-zA-Z]*$||' - remove trailing .co.uk, .co.nl,...
's|.*\.\([^.]*\.[^.]*\)|\1|' - remove all components from the hostname except for the last two (this is basically your awk-script)
How about doing it entirely within bash:
if [[ $WEBSITE =~ ^(.*)([.]co)[.][a-z]+$ || $WEBSITE =~ ^(.*)[.][a-z]+$ ]]
then
pass=${BASH_REMATCH[1]}
else
echo WARNING: Unexpected value for WEBSITE: $WEBSITE
pass=$WEBSITE # Fallback
fi
I used two clauses (for the .co case and for the other cases), because bash a regexp does not understand non-greedy matching (i.e. .*?).
I propose you to work around a very simple modification like this grep command add:
WEBSITE=$(echo $1 | grep -vE ".com|.uk" | sed -e "s/[^/]*\/\/\([^#]*#\)\?\([^:/]*\).*/\2/" | awk -F. '{print $(NF-1) "." $NF}')
test -z $WEBSITE && exit 1 # if empty (.com or .uk generates an empty variable)
$ cat > toto
WEBSITE=$(echo $1 | grep -vE ".com|.uk" | sed -e "s/[^/]*\/\/\([^#]*#\)\?\([^:/]*\).*/\2/" | awk -F. '{print $(NF-1) "." $NF}')
test -z $WEBSITE && exit 1
echo $WEBSITE
With an example:
$ bash toto http://www.google.fr
google.fr
$ bash toto http://www.google.com
$ bash toto http://www.google.uk
$ bash toto http://www.google.gertrude
google.gertrude
$ rm toto
$
I used .uk in my example so do not just copy/paste the line.
I have a bash-script that moves backup-files to the remote location. On few occasions the temporary HDDs on the remote server had no space left, so I added a md5 check to compare local and remote files.
The remote ssh breaks however the while-loop (i.e. it runs only for first item listed in dir_list file).
# populate /tmp/dir_list
(while read dirName
do
# create archive files for sub-directories
# populate listA variable with archive-file names
...
for fileName in $listA; do
scp /PoolZ/__Prepared/${dirName}/$fileName me#server:/archiv/${dirName}/
md5_local=`md5sum /PoolZ/__Prepared/${dirName}/${fileName} | awk '{ print $1 }'`
tmpRemoteName=`printf "%q\n" "$fileName"` # some file-names have strange characters
md5_remote=`ssh me#server 'md5sum /archiv/'${dirName}'/'$tmpRemoteName | awk '{ print $1 }'`
if [[ $md5_local == $md5_remote ]]; then
echo "Checksum of ${fileName}: on local ${md5_local}, on remote ${md5_remote}."
mv -f /PoolZ/__Prepared/${dirName}/$fileName /PoolZ/__Backuped/${dirName}/
else
echo "Checksum of ${fileName}: on local ${md5_local}, on remote ${md5_remote}."
# write eMail
fi
done
done) < /tmp/dir_list
When started the script gives the same md5-sums for the first directory listed in dir_list. The files are also copied both local and remote to expected directories and then script quits.
If I remove the line:
md5_remote=`ssh me#server 'md5sum /archiv/'${dirName}'/'$tmpRemoteName | awk '{ print $1 }'`
then apparently the md5-comaprison is not working but the whole script goes through whole list from dir_list.
I also tried to use double-quotes:
md5_remote=`ssh me#server "md5sum /archiv/${dirName}/${tmpRemoteName}" | awk '{ print $1 }'`
but there was no difference (broken dirName-loop).
I went so far, that I replaced the md5_remote... line with a remote ls-command without any shell-variables, and eventually I even tried a line without setting value to the md5_remote variable, i.e.:
ssh me#server "ls /dir/dir/dir/ | head -n 1"
Every solution that has a ssh-command breaks the while-loop. I have no idea why ssh should break bash-loop. Any suggestion are welcomed.
I'm plainly stupid. I found just answer on — what a surprise — stackoverflow.com.
ssh breaks out of while-loop in bash
As suggested I added a pipe to /dev/null and it works now:
md5_remote=`ssh me#server 'md5sum /archiv/'${dirName}'/'$tmpRemoteName < /dev/null | awk '{ print $1 }'`
There are a few answers on this topic already, but pretty much all of them say that it's bad to parse the output of ls -l, and therefore suggest other methods.
However, I'm using ncftpls -l, and so I can't use things like shell globs or find – I think I have a genuine need to actually parse the ls -l output. Don't worry if you're not familiar with ncftpls, the output returns in exactly the same format as if you were just using ls -l.
There is a list of files at a public remote ftp directory, and I don't want to burden the remote server by re-downloading each of the desired files every time my cronjob fires. I want to check, for each one of a subset of files within the ftp directory, whether the file exists locally; if not, download it.
That's easy enough, I just use
tdy=`date -u '+%Y%m%d'`_
# Today's files
for i in $(ncftpls 'ftp://theftpserver/path/to/files' | grep ${tdy}); do
if [ ! -f $i ]; then
ncftpget "ftp://theftpserver/path/to/files/${i}"
fi
done
But I came upon the issue that sometimes the cron job will download a file that hasn't finished uploading, and so when it fires next, it skips the partially downloaded file.
So I wanted to add a check to make sure that for each file that I already have, the local file size matches the size of the same file on the remote server.
I was thinking along the lines of parsing the output of ncftpls -l and using awk, something like
for i in $(ncftpls -l 'ftp://theftpserver/path/to/files' | awk '{print $9, $5}'); do
...
x=filesize # somehow get the file size and the filename
y=filename # from $i on each iteration and store in variables
...
done
but I can't seem to get both the filename and the filesize from the server into local variables on the same iteration of the loop; $i alternates between $9 and $5 in the awk string with each iteration.
If I could manage to get the filename and filesize into separate variables with each iteration, I could simply use stat -c "%s" $i to get the local size and compare it with the remote size. Then its a simple ncftpget on each remote file that I don't already have. I tinkered with syncing programs like lftp too, but didn't have much luck and would rather do it this way.
Any help is appreciated!
for loop splits when it sees any whitespace like space, tab, or newline. So, IFS is needed before loop, (there are a lot of questions about ...)
IFS=$'\n' && for i in $(ncftpls -l 'ftp://theftpserver/path/to/files' | awk '{print $9, $5}'); do
echo $i | awk '{print $NF}' # filesize
echo $i | awk '{NF--; print}' # filename
# you may have spaces in filenames, so is better to use last column for awk
done
The better way I think is to use while not for, so
ls -l | while read i
do
echo $i | awk '{print $9, $5}'
#split them if you want
x=echo $i | awk '{print $5}'
y=echo $i | awk '{print $9}'
done
I have the following simple snippet:
#!/bin/bash
tail -f "data/top.right.log" | while read val1
do
val2=$(tail -n 1 "data/top.left.log")
echo $(echo "$val1 - $val2" | bc)
done
top.left.log and top.right.log are files to which some other processes continually write. The bash script simply subtracts the last lines of both files and show a result.
I would like to make the script more efficient. In pseudo-code I would like to do this:
#!/bin/bash
magiccommand "data/top.right.log" "data/top.left.log" | while read val1 val2
do
echo $(echo "$val1 - $val2" | bc)
done
so that whenever top.left.log OR top.right.log changes the echo command is called.
I have already tried various snippets from StackOverflow but often they rely on the fact that the files do not change or that both files contain the same amount of lines which is not my case.
If you have inotify-tools you can use following command:
inotifywait -q -e modify file1 file2
Description:
inotifywait efficiently waits for changes to files using Linux's inotify(7) interface.
It is suitable for waiting for changes to files from shell scripts.
It can either exit once an event occurs, or continually execute and output events as they occur.
An example:
while : ;
do
inotifywait -q -e modify file1 file2
echo `tail -n1 file1`
echo `tail -n1 file2`
done
Create a temporary file that you touch each time the files are processed. If any of the files is newer than the temporary file, process the files again.
#!/bin/bash
log1=top.left.log
log2=top.right.log
tmp=last_change
last_change=0
touch "$tmp"
while : ; do
if [[ $log1 -nt $tmp || $log2 -nt $tmp ]] ; then
touch "$tmp"
x=$(tail -n1 "$log1")
y=$(tail -n1 "$log2")
echo $(( x - y ))
fi
done
You might need to remove the temporary file once the script is killed.
If the files are changing fast, you might miss some lines. Otherwise, adding sleep 1 somewhere would decrease the CPU usage.
Instead of calling tail every time, you can open file descriptors once and read line after line. This makes use of the fact that the files are kept open, and read will always read from the next line of a file.
First, open the files in bash, assigning them file descriptors 3 and 4
exec 3<file1 4<file2
Now, you can read from these files using read -u <fd>. In combination with inotifywait of Dawid's answer, this gives you an efficient way to read files line by line:
while :; do
# TODO: add some break condition
# wait until one of the files has changed
inotifywait -q -e modify file1 file2
# read the next line of file1 into val1_new
# if file1 has not changed and there is no new line, read will return with failure
read -u 3 val1_new && val1="$val1_new"
# same for file2
read -u 4 val2_new && val2="$val2_new"
done
You may extend this by reading until you have reached the last line, or parsing inotifywait's output to detect which file has changed.
A possible way is to parse the output of tail -f and display the difference in value whenever the ==> <== pattern is found.
I came up with this script:
$ cat test.awk
$0 ~ /==>.*right.*<==/ {var=1}
$0 ~ /==>.*left.*<==/ {var=2}
$1~/[0-9]+/ && var==1 { val1=$1 }
$1~/[0-9]+/ && var==2 { val2=$1 }
val1 != "" && val2 != "" && $1~/[0-9]+/{
print val1-val2
}
The script assume the values are integer [0-9]+ in both file.
You can use it like this:
tail -f top.right.log top.left.log | awk -f test.awk
Whenever a value is appended in any of the file, the difference between the last value of each file is displayed.
I have a text file holding numerous records formatted as follows:
Ford:Mondeo:1997:Blue:5
There are around 100 of them that I'm trying to sort via a bash script, and I'm looking to extract all cars made between 1994 and 1999. Here's what I have so far:
awk -F: '$3=="1994"' | awk -F: '$3<="1999"' $CARFILE > output/1994-1999.txt
The output file is containing all of the correct information, no duplicates etc but it freezes and doesn't echo out the confirmation afterwards. I have to ctrl + D my way out of the script.
Here's the full file for reference:
#CS101 Assignment BASH script
#--help option
#case $1 in
# --help | carslist.txt)
# cat <<-____HALP
# Script name: ${0##*/} [ --help | carslist.txt ]
# This script will organise the given text file and save various #sections to new files.
# No continuation checks are used but when each part is finished, a #confirmation message will print before the script continues.
#____HALP
# exit 0;;
#esac
CARFILE=$1
while [ ! -f "$CARFILE" ]
do
echo "We cannot detect a car file to load, please enter the new filename and press [ENTER]"
read CARFILE
done
echo "We have detected that you're using $CARFILE as your cars file, please continue."
if [ -f output ]
then
echo "Sorry, a file called 'output' exists in the working directory. The script will now exist."
elif [ -d output ]
then
echo "The directory 'output' has been detected, instead of creating a new one we'll be working in there instead."
else
mkdir output
echo "We couldn't find an existing file or directory named 'output' so we've made one for you. Aren't we generous?"
fi
grep 'Vauxhall' $CARFILE > output/Vauxhall_Cars.txt
echo "We've saved all Vauxhall information in the 'Vauxhall_Cars.txt' file. The script will now continue."
grep '2001' $CARFILE > output/Manufactured_2001.txt
echo "We've saved all cared manufactured in 2001 in the 'Manufactured_2001.txt' file. The script will now continue."
awk -F: '$1=="Volkswagen" && $4=="Blue"' $CARFILE > output/Blue_Volkswagen.txt
echo "We've saved all Blue Volkswagens cars in Blue_Volkswagen.txt. The script will now continue"
grep 'V' $CARFILE > output/Makes_V.txt
echo "All cars with the make starting with 'V' have been saved in Makes_V.txt. The script will now continue."
awk -F: '$3=="1994"' | awk -F: '$3<="1999"' $CARFILE > output/1994-1999.txt
echo "Cars made between 1994 and 1999 have been saved in 1994-1999.txt. The script will now continue."
With the run command being bash myScript.sh carslist.txt
Can anyone tell me why it's freezing after outputting correctly?
Just noticed that a record of 1993 has slipped through the cracks, is there a way of formatting the dates in the line above so it's only between 1994-1999?
Thanks in advance.
The expression:
awk -F: '$3=="1994"' | awk -F: '$3<="1999"' $CARFILE > output/1994-1999.txt
Means: run awk on "something" and then pipe to another awk. But you are not providing any "something", so awk is waiting for it.
This is like saying:
awk '{print}' | awk 'BEGIN {print 1}'
It indeed prints the 1 but waits for some kind of input to come.
Just join the conditions:
awk -F: '$3=="1994" && $3<="1999"' $CARFILE > output/1994-1999.txt
Regarding the rest of the script: note you are not using many double quotes. They are a good practise to prevent problems when you have names with spaces, etc. So for example you can say grep 'Vauxhall' "$CARFILE" and allow $CARFILE to contains things like "my car".
You can find out these kind of errors by pasting your script in ShellCheck.