Bash script? Manipulating & graphing collected data from a csv file - bash

I'm trying to write a script that can detect my presence at home. So far I've written a script that outputs data from hcitool lescan into a csv file in the following format:
TIMESTAMP MAC_ADDRESS_1 MAC_ADDRESS_2 AD_INFINITUM
2018-09-22.11:48:34 FF:FF:FF:FF:FF:FF FF:FF:FF:FF:FF:FF FF:FF:FF:FF:FF:FF
I'm trying to figure out how to write a script to convert the data into a graphable format - is gnuplot the program to be used for this? I guess that this would require a bash? script that imports the csv file keeping all timestamps, then adding a new column into the array for each unique MAC address then populating the entries with a 1 or 0 depending if the Mac address is detected per line. Are there any built in commands that can do/help with this or would I have to script it myself?
The code I used to generate the .csv is here. Sorry, its probably not the prettiest as I've just only started with bash scripting.
cd /home/pi/projects/bluetooth_control;
while true
do
echo 'reset hci0';
sudo hciconfig hci0 down;
sudo hciconfig hci0 up;
echo 'timestamp';
echo `date +%Y-%m-%d.%H:%M:%S` &> test1.csv;
echo 'running scan';
(sudo timeout 20 stdbuf -oL hcitool lescan | grep -Eo '(([A-Z]|[0-9]){2}:){5}([A-Z]|[0-9]){2}') &> test.csv;
echo 'removing duplicates to test.csv';
(sort test.csv | uniq) >> test1.csv;
(paste -s test1.csv) >> data.csv;
echo 'sleep for 60s';
sleep 60;
done

I've had time to play around and in the interest of completing the answer here is the solution I came up with. I'm not sure how efficient it is to run it in Bash vs. Python but here goes:
#!/bin/bash
cd /home/pi/projects/bluetooth_control;
while true
do
echo 'reset hci0';
sudo hciconfig hci0 down;
sudo hciconfig hci0 up;
echo 'timestamp';
# Create necessary temp files
echo "temp" &> test1.csv;
echo `date +%Y-%m-%d.%H:%M:%S` &> test2.csv;
echo 'running scan';
# Filter out only MAC addresses
(sudo timeout 20 stdbuf -oL hcitool lescan | grep -Eo '(([A-Z]|[0-9]){2}:){5}([A-Z]|[0-9]){2}') &> /home/pi/projects/bluetooth_control/test.csv;
echo 'removing duplicates to test.csv';
# Append each unique value to test1.csv
(sort test.csv | uniq) >> test1.csv;
# For each line in test1.csv, add text to mac_database if it doesn't exist
while read line
do
grep -q -F $line mac_database || echo $line >> mac_database
done <test1.csv
# For each line in mac_database, run an if loop
while read line
do
# If $line from mac_database exists in test1.csv, then
if grep -Fxq "$line" test1.csv
then
echo '1' >> test2.csv
else
echo '0' >> test2.csv
fi
done <mac_database
# Convert file to csv format, and append to data.csv
(paste -s test2.csv) >> data.csv;
echo 'sleep for 60s';
sleep 60;
done
Hopefully this helps whoever might choose to do this in the future.

Related

Redirect stderr/-out of running application to my own bash script

I am running an application called "hd-idle". It is spinning down disks after a specific time of inactivity.
The output looks like this:
user#linux:~$ sudo /usr/sbin/hd-idle -i 10800
symlinkPolicy=0, defaultIdle=10800, defaultCommand=scsi, defaultPowerCondition=0, debug=false, logFile=, devices=
sda spindown
sdd spindown
sde spindown
sda spinup
sdd spinup
sdd spindown
[...]
I want to save this output to a logfile (while the application in running), add timestamps and change sd[a-z] to corresponding model/serial of the hard drive.
I wrote a small bash script that does what I want:
user#linux:~$ cat hd_idle_logger.sh
#!/bin/bash
DATUM=$(date '+%Y-%m-%d %H:%M:%S')
INPUT=$(cat)
REGEX='(sd[a-z])\s(spin(down|up))'
[[ $INPUT =~ $REGEX ]]
if [ -n ${BASH_REMATCH[1]} ]
then
MODEL=$(lsblk /dev/${BASH_REMATCH[1]} -n -o MODEL)
SERIAL=$(lsblk /dev/${BASH_REMATCH[1]} -n -o SERIAL)
fi
echo -e "$DATUM\t${MODEL}_$SERIAL (${BASH_REMATCH[1]})\t${BASH_REMATCH[2]}" >> /home/linux/hd_idle_logger.log
I can verify that it works:
user#linux:~$ echo "sdd spindown" |& ./hd_idle_logger.sh
user#linux:~$ cat hd_idle_logger.log
2023-02-12 12:14:54 WDC_WD120EMAZ-10BLFA6_1PAEL2ES (sdd) spindown
But running the application and passing the output to my script doesn't work, the logfile doesn't produce any content and I don't see the output on console anymore:
user#linux:~$ sudo /usr/sbin/hd-idle -i 10800 |& /home/user/hd_idle_logger.sh
So what I am doing wrong?
As long as hd-idle is running, your script will be stuck at INPUT=$(cat). Because $(cat) has to capture ALL output, it can online terminate once hd-idle terminated.
You need a script/program that processes hd-idle's output on the fly; e.g. line by line, while hd-idle is still running. You could do this with a while read loop:
#! /bin/bash
regex='(sd[a-z])\s(spin(down|up))'
while IFS= read -r line; do
[[ $line =~ $regex ]] || continue
model=$(lsblk /dev/"${BASH_REMATCH[1]}" -n -o MODEL)
serial=$(lsblk /dev/"${BASH_REMATCH[1]}" -n -o SERIAL)
printf '%(%Y-%m-%d %H:%M:%S)T\t%s_%s (%s)\t%s\n' \
"$model" "$serial" "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}"
done >> /home/linux/hd_idle_logger.log
However, it would be more efficient to switch to utils like sed or awk and pre-compute the list of serial numbers or look for the required information in the /sys/block file system, so that you don't have to execute lsblk for each line.

Need to print current CPU usage and Memory usage in file continuously

I have prepared the below script, but it's not adding any data to the output file.
My intention is to get the current CPU usage and Memory usage and print them on log file.
What is wrong with my below script? I will run this script file in CentOS machine.
#!/usr/bin/bash
HOSTNAME=$(hostname)
mkdir -p /root/scripts
LOGFILE=/root/scripts/xcpuusagehistory.log
touch $LOGFILE
a=0;
b=1;
while [ "$a" -lt "$b" ]
do
CPULOAD=`top -d10 | grep "Cpu(s)"`
echo "$CPULOAD on Host $HOSTNAME" >> $LOGFILE
done
while true
do
cpu_load="$(top -b -n1 -d10 | grep "Cpu(s)")"
echo "$cpu_load on Host $HOSTNAME" >> "$log_file"
sleep 1
done
See top batch mode (-b) in the man page.

the bash script only reboot the router without echoing whether it is up or down

#!/bin/bash
ip route add 10.105.8.100 via 192.168.1.100
date
cat /home/xxx/Documents/list.txt | while read output
do
ping="ping -c 3 -w 3 -q 'output'"
if $ping | grep -E "min/avg/max/mdev" > /dev/null; then
echo 'connection is ok'
else
echo "router $output is down"
then
cat /home/xxx/Documents/roots.txt | while read outputs
do
cd /home/xxx/Documents/routers
php rebootRouter.php "outputs" admin admin
done
fi
done
The other documents are:
lists.txt
10.105.8.100
roots.txt
192.168.1.100
when i run the script, the result is a reboot of the router am trying to ping. It doesn't ping.
Is there a problem with the bash script.??
If your files only contain a single line, there's no need for the while-loop, just use read:
read -r router_addr < /home/xxx/Documents/list.txt
# the grep is unnecessary, the return-code of the ping will be non-zero if the host is down
if ping -c 3 -w 3 -q "$router_addr" &> /dev/null; then
echo "connection to $router_addr is ok"
else
echo "router $router_addr is down"
read -r outputs < /home/xxx/Documents/roots.txt
cd /home/xxx/Documents/routers
php rebootRouter.php "$outputs" admin admin
fi
If your files contain multiple lines, you should redirect the file from the right-side of the while-loop:
while read -r output; do
...
done < /foo/bar/baz
Also make sure your files contain a newline at the end, or use the following pattern in your while-loops:
while read -r output || [[ -n $output ]]; do
...
done < /foo/bar/baz
where || [[ -n $output ]] is true even if the file doesn't end in a newline.
Note that the way you're checking for your routers status is somewhat brittle as even a single missed ping will force it to reboot (for example the checking computer returns from a sleep-state just as the script is running, the ping fails as the network is still down but the admin script succeeds as the network just comes up at that time).

script does not stop when arguments are passed

I have the following which works perfectly.
#!/bin/bash
killall java
#program USB
make iris install.1 mib510,/dev/ttyUSB0
#listen serial port and write to file
java net.tinyos.tools.PrintfClient -comm serial#/dev/ttyUSB1:iris > foo.txt &
sleep 2
#if "Erase done" is printed to file, stop
if tail -f foo.txt | grep -n "Erase done" -q; then echo "Write ok";fi
killall java
But when I change my script to receive arguments below (sh test.sh USB0 USB1 foo.txt), it does not end. Although it writes the file, the process does not end
#!/bin/bash
killall java
#program USB
make iris install.1 mib510,/dev/tty$1
#listen serial port and write to file
java net.tinyos.tools.PrintfClient -comm serial#/dev/tty$2:iris > $3 &
sleep 2
#if "Erase done" is printed to file, stop
if tail -f $3 | grep -n "Erase done" -q; then echo "Write ok";fi
killall java
Am I doing something wrong?
It appears tail -f will quit when grep quits. So the problem might be with:
if tail -f $3 | grep -n "Erase done" -q; then echo "Write ok";fi
You can replace it with the following:
tail -f $3 | while read LOGLINE
do
[[ "${LOGLINE}" == *"Erase done"* ]] && echo "Write ok" && pkill -P $$ tail
done

FTP File Transfers Using Piping Safely

I have a file forwarding system where a bunch of files are downloaded to a directory, de-multiplexed and copied to individual machines.
The files are forwarded when they are received by the master server. And files normally arrive in bursts. (Auth by ssh keys)
This script creates the sftp session, and uses a pipe to watch the head of a fifo pipe.
HOST=$1
pipe=/tmp/pipes/${HOST%%.*}
ps aux | grep -v grep | grep sftp | grep "user#$HOST" > /dev/null
if [[ $? == 0 ]]; then
echo "FTP is Running on this Server"
exit
else
pid=`ps aux | grep -v grep | grep tail | tr -s ' ' | grep $pipe`
[[ $? == 0 ]] && kill -KILL `echo $pid | cut -f2 -d' '`
fi
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
tail -n +1 -f $pipe | sftp -o 'ServerAliveInterval 60' user#$HOST > /dev/null &
echo cd /tmp/data >>$pipe #Sends Command to Host
echo "Started FTP to $HOST"
Update: I ended up changing the cleanup code to use "ps aux" to see if an ftp session is running, and subsequently if the tail -f is still running. Grep by user#host and the name of the pipe respectively. This is done when the script is called, and the script is called whenever I try to upload a file.
IE:
FILENAME=`basename $1`
function transfer {
echo cd /apps/data >> $2 # For Safety
echo put $1 .$FILENAME >> $2
echo rename .$FILENAME $FILENAME >> $2
echo chmod 0666 $FILENAME >> $2
}
./ftp.sh host
[ -p $pipedir/host ] && transfer $1 $pipedir/host
Files received on the master server are caught by Incron which writes a put command and the available file's location to the fifo pipe, to be sent by sftp (rename is also preformed).
My question is, is this safe? Could this crash on ftp errors/events. Not really worried about login errors.
The goal is to reduce the number of ftp logins. Single Session/Minute(or more) intervals.
And allow files to be forwarded as they're received. Dynamic Commands.
I'd prefer to use standard ubuntu libraries, if possible.
EDIT: After testing and working through some issues the server simply runs with
[[ -p $pipe ]] && echo FTP is Running on this Server
ln -s $pipe $lock &> /dev/null || (echo FTP is Running on this Server && exit)
[[ ! -p $pipe ]] && mkfifo $pipe
( tail -n +1 -F $pipe & echo $! > $pipe.pid ) | tee >
( sed "/tail:/ q" >/dev/null && kill $(cat $pipe.pid) |& rm -f $pipe >/dev/null; )
| sftp -i ~/.ssh/$HOST.rsa -oServerAliveInterval=60 user#$HOST &
rm -f $lock
Its rather simple but works nicely.
you might be intrested in setting up a more simpler(and robust) syncronization infrastructure:
if a given host is not connected when a file arrives...it never recieves it (if i understand correctly your code)
i would do something like
rsync -a -e ssh user#host:/apps/data pathToLocalDataStore
on the client machines either periodically or by event...rsync is intelligently syncronizes the files by their timestamp and size (-a contains -t)
the event would be some process termination like:
client does(configure private key usage in ~/.ssh/config for host):
#!/bin/bash
while :;do
ssh user#host /srv/bin/sleepListener 600
rsync -a -e ssh user#host:/apps/data pathToLocalDataStore
done
on the server
/srv/bin/sleepListener is a symbolic link to /bin/sleep
server after recieving new file:
killall sleepListener
note: every 10 minutes a full check is performed...if nodes go offline/online it doesn't matter...

Resources