Script to Format and Mount all available instance store devices - amazon-ec2

Amazon provides instance store for EC2 instances. If you use your own AMI, these are not formatted or mounted automatically for you. You need to manually format and mount them.
The available devices are listed here and vary based on type of instance. For example an m1.small will have different available instance store devices than c1.xlarge.
I'm looking for a script which
Detects what the instance type is. Perhaps by using
curl -s http://169.254.169.254/latest/meta-data/instance-type
Formats and mounts all devices which are available for that instance type but have not yet been formatted/mounted.
Possible? Done it? Have it?

So, here is what I built for this.
#!/bin/bash
# This script formats and mounts all available Instance Store devices
##### Variables
devices=( )
##### Functions
function add_device
{
devices=( "${devices[#]}" $1 )
}
function check_device
{
if [ -e /dev/$1 ]; then
add_device $1
fi
}
function check_devices
{
check_device sda2
check_device sda3
check_device sdb
check_device sdc
check_device sdd
check_device sde
}
function print_devices
{
for device in "${devices[#]}"
do
echo Found device $device
done
}
function do_mount
{
echo Mounting device $1 on $2
fdisk $1 << EOF
n
p
1
w
EOF
# format!
mkfs -t xfs -f $1
mkdir $2
mount $1 $2
echo "$1 $2 xfs defaults 0 0" >> /etc/fstab
}
function mount_devices
{
for (( i = 0 ; i < ${#devices[#]} ; i++ ))
do
mountTarget=/mnt
if [ $i -gt 0 ]; then
mountTarget=/mnt$(($i+1))
fi
do_mount /dev/${devices[$i]} $mountTarget
done
}
##### Main
check_devices
print_devices
mount_devices

#!/bin/bash
#SETUP RAID0
checkAllDevices()
{
devicemount=/ephemeral
logicalname=/dev/md0
deviceslist=( '/dev/xvdb' '/dev/xvdc' '/dev/xvdd' '/dev/xvde' )
for device in ${deviceslist[#]}; do
if ([ -b $device ]) then
aDevices=( "${aDevices[#]}" $device )
fi
done
if [ "${#aDevices[#]}" -gt '1' ];then
yes | mdadm --create $logicalname --level=0 -c256 --raid-devices=${#aDevices[#]} ${aDevices[#]}
echo \'DEVICE ${aDevices[#]}\' > /etc/mdadm.conf
mdadm --detail --scan >> /etc/mdadm.conf
blockdev --setra 65536 $logicalname
mkfs.xfs -f $logicalname > /dev/null
mkdir -p $devicemount
mount -t xfs -o noatime $logicalname $devicemount
if [ ! -f /etc/fstab.backup ]; then
cp -rP /etc/fstab /etc/fstab.backup
echo "$logicalname $devicemount xfs defaults 0 0" >> /etc/fstab
fi
else
echo "Required more than one devices"
fi
}
#MAIN FUNCTION
aDevices=()
checkAllDevices

Related

using cronic to reduce email notifications in a wrapper script for successful backups

I've been using cronic to silence emails from cron jobs when the job is successful. I'm trying to customize it so when a response code is 0 and the error output matches a string of "mount: /VessRAID/RH: /dev/sde1 already mounted on /VessRAID/RH.", to not send an email. Below is the script, then the contents of the email then my attempt at trying to suppress the email which is not working. Any idea what I may be doing wrong?
#!/bin/bash
# Cronic v3 - cron job report wrapper
# Copyright 2007-2016 Chuck Houpt. No rights reserved, whatsoever.
# Public Domain CC0: http://creativecommons.org/publicdomain/zero/1.0/
set -eu
TMP=$(mktemp -d)
OUT=$TMP/cronic.out
ERR=$TMP/cronic.err
TRACE=$TMP/cronic.trace
set +e
"$#" >$OUT 2>$TRACE
RESULT=$?
set -e
PATTERN="^${PS4:0:1}\\+${PS4:1}"
if grep -aq "$PATTERN" $TRACE
then
! grep -av "$PATTERN" $TRACE > $ERR
else
ERR=$TRACE
fi
if [ $RESULT -ne 0 -o -s "$ERR" ]
then
echo "Cronic detected failure or error output for the command:"
echo "$#"
echo
echo "RESULT CODE: $RESULT"
echo
echo "ERROR OUTPUT:"
cat "$ERR"
echo
echo "STANDARD OUTPUT:"
cat "$OUT"
if [ $TRACE != $ERR ]
then
echo
echo "TRACE-ERROR OUTPUT:"
cat "$TRACE"
fi
fi
rm -rf "$TMP"
Here is what the email notification looks like:
Cronic detected failure or error output for the command:
/usr/local/sbin/reg-backup-cronic.sh daily
RESULT CODE: 0
ERROR OUTPUT:
mount: /VessRAID/RH: /dev/sde1 already mounted on /VessRAID/RH.
STANDARD OUTPUT:
/dev/sde1 on /VessRAID/RH type ext4 (rw,relatime)
Here is my attempt at a wrapper script:
#!/bin/bash
/usr/local/sbin/reg-backup.sh $1
CODE=$?
err=$TRACE
if [[ $CODE -eq 0 && $err = "mount: /VessRAID/RH: /dev/sde1 already mounted on /VessRAID/RH." ]]
then
exit $CODE
fi
Alas the emails continue.
Hat tip to the creator of cronic, Chuck Houpt, for cluing me in to an answer, which was to look at the original script and why the error is happening. Case-sensitivity got the best of me:
if mount | grep Vessraid; then
echo starting $1 backup >> /var/log/vessraid.log
Notice the case in VessRAID should have been:
if mount | grep VessRAID; then
echo starting $1 backup >> /var/log/vessraid.log
Now emails only happen when there really is an error.

Delete first line/chars on Windows Pipe

I'm trying to capture a live tcpdump from a custom designed linux system. The command I'm using so far is:
plink.exe -ssh user#IP -pw PW "shell nstcpdump.sh -s0 -U -w - not port 22 and not host 127.0.0.1" | "C:\Program Files\Wireshark\wireshark" -i -
This will fail since when executing the command on the remote system, this (custom) shell will output "Done" before sending data. I tried to find out a way to remove the "Done" message from the shell but doesn't appear to be any.
So I came up with this (added findstr -V):
plink.exe -ssh user#IP -pw PW "shell nstcpdump.sh -s0 -U -w - not port 22 and not host 127.0.0.1" | findstr -V "Done" | "C:\Program Files\Wireshark\wireshark" -i -
This works more or less fine since I will get some errors and the live capture will stop. I believe it might have something to do with the buffers, but I'm not sure.
Does anyone know of any other method of removing/bypassing the first bytes/chars of the output from plink/remote shell?
[edit]
as asked, nstcpdump.sh is a wrapper for tcpdump. as mentioned before, this system is highly customized. nstcpdump.sh code:
root#hostname# cat /netscaler/nstcpdump.sh
#!/bin/sh
# piping the packet trace to the tcpdump
#
# FILE: $Id: //depot/main/rs_120_56_14_RTM/usr.src/netscaler/scripts/nstcpdump.sh#1 $
# LAST CHECKIN: $Author: build $
# $DateTime: 2017/11/30 02:14:38 $
#
#
# Options: any TCPDUMP options
#
TCPDUMP_PIPE=/var/tmp/tcpdump_pipe
NETSCALER=${NETSCALER:-`cat /var/run/.NETSCALER`}
[ -r ${NETSCALER}/netscaler.conf ] && . ${NETSCALER}/netscaler.conf
TIME=${TIME:-3600}
MODE=${MODE:-6}
STARTCMD="start nstrace -size 0 -traceformat PCAP -merge ONTHEFLY -filetype PIPE -skipLocalSSH ENABLED"
STOPCMD="stop nstrace "
SHOWCMD="show nstrace "
NSCLI_FILE_EXEC=/netscaler/nscli
NSTRACE_OUT_FILE=/tmp/nstrace.out
NS_STARTTRACE_PIDFILE=/tmp/nstcpdump.pid
TRACESTATE=$(nsapimgr -d allvariables | grep tracestate | awk '{ print $2}')
trap nstcpdump_exit 1 2 15
nstcpdump_init()
{
echo "##### WARNING #####"
echo "This command has been deprecated."
echo "Please use 'start nstrace...' command from CLI to capture nstrace."
echo "trace will now start with all default options"
echo "###################"
if [ ! -d $NSTRACE_DIR ]
then
echo "$NSTRACE_DIR directory doesn't exist."
echo "Possible reason: partition is not mounted."
echo "Check partitions using mount program and try again."
exit 1
fi
if [ ! -x $NSCLI_FILE_EXEC ]
then
echo "$NSCLI_FILE_EXEC binary doesn't exist"
exit 1
fi
if [ -e $NSTRACE_OUT_FILE ]
then
rm $NSTRACE_OUT_FILE
echo "" >> $NSTRACE_OUT_FILE
fi
}
nstcpdump_start_petrace()
{
sleep 0.5;
$NSCLI_FILE_EXEC -U %%:.:. $STARTCMD >/tmp/nstcpdump.sh.out
rm -f ${NS_STARTTRACE_PIDFILE}
}
nstcpdump_start()
{
# exit if trace is already running
if [ $TRACESTATE -ne 0 ]
then
echo "Error: one instance of nstrace is already running"
exit 2
fi
nstcpdump_start_petrace &
echo $! > ${NS_STARTTRACE_PIDFILE}
tcpdump -n -r - $TCPDUMPOPTIONS < ${TCPDUMP_PIPE}
nstcpdump_exit
exit 1
}
nstcpdump_exit()
{
if [ -f ${NS_STARTTRACE_PIDFILE} ]
then
kill `cat ${NS_STARTTRACE_PIDFILE}`
rm ${NS_STARTTRACE_PIDFILE}
fi
$NSCLI_FILE_EXEC -U %%:.:. $STOPCMD >> /dev/null
exit 1
}
nstcpdump_usage()
{
echo `basename $0`: utility to view/save/sniff LIVE packet capture on NETSCALER box
tcpdump -h
echo
echo NOTE: tcpdump options -i, -r and -F are NOT SUPPORTED by this utility
exit 0
}
########################################################################
while [ $# -gt 0 ]
do
case "$1" in
-h )
nstcpdump_usage
;;
-i )
nstcpdump_usage
;;
-r )
nstcpdump_usage
;;
-F )
nstcpdump_usage
;;
esac
break;
done
TCPDUMPOPTIONS="$#"
check_ns nstcpdump
#nstcpdump_init
#set -e
if [ ! -e ${TCPDUMP_PIPE} ]
then
mkfifo $TCPDUMP_PIPE
if [ $? -ne 0 ]
then
echo "Failed creating pipe [$TCPDUMP_PIPE]"
exit 1;
fi
fi
nstcpdump_start
Regards

how to run multiple commands on a remote linux server using bash script

I am currently writing the following script that logs into a remote server and runs couple of commands to verify the performance of the server and prints a message based on the output of those commands .But the ssh doesn't work and returns the stats of the server that hosts the script instead .
Script
#!/bin/bash
#######################
#Function to add hosts to the array
#the following function takes the ip addresses provided while the script is run and stores them in an array
#######################
Host_storing_func () {
HOST_array=()
for i in $# ;do
HOST_array+=(${i});
done
#echo ${HOST_array[*]}
}
#######################
#Calling above function
#######################
Host_storing_func "$#"
############################################################
#Collect Stats of Ping,memory,iowait time test function
############################################################
b=`expr ${#HOST_array[*]} - 1 `
for i in `seq 0 $b` ;do
sshpass -f /root/scripts/passwordFile.txt /usr/bin/ssh student35#${HOST_array[${i}]} << HERE
echo `hostname`
iowaittm=`sar 2 2|awk '/^Average/{print $5};'`
if [ $iowaittm > 10 ];then
echo "IO ==> BAD"
else
echo "IO ==> GOOD"
fi
memoryy=`free -m |grep Swap|awk '{if($2 == 0) print 0;else print (($4 / $2 ) * 100)}'`
if [ ${memoryy} < '10' ] ;then
echo "memory ==> good"
elif [[ "${memory}" -ge 0 ]] && [[ "${memory}" -le 10 ]];then
echo "No Swap"
else
echo "memory ==> bad"`enter code here`
fi
ping -w2 -c2 `hostname` | grep "packet loss"|awk -F, '{print $3}'|awk -F% '{print $1}'|sed 's/^ *//'|awk '{if ($1 == 0) print "Yes" ;else print "No"}'
HERE
done
Output : oc5610517603.XXX.com is the name of the source server
[root#oc5610517603 scripts]# ./big_exercise.sh 9.XXX.XXX.XXX 9.XXX.XXX.XXX
Pseudo-terminal will not be allocated because stdin is not a terminal.
oc5610517603.XXX.com
IO ==> GOOD
No Swap
ping: oc5610517603.ibm.com: Name or service not known
Pseudo-terminal will not be allocated because stdin is not a terminal.
oc5610517603.XXX.com
IO ==> GOOD
No Swap
ping: oc5610517603.XXX.com: Name or service not known
thanks for checking the script , I figured out a way to solve the problem
It is the sshpass command that is causing issue , you just have to put the opening HERE in single quotes if you want to use variables with in the HEREdoc but if the variables are calculated before ssh then you don't have to put opening HERE in single quotes
sshpass -f /root/scripts/passwordFile.txt /usr/bin/ssh -T student35#${i} << 'HERE'
after I changed the sshpass command as above my script worked
I have modified your script a bit.
As suggested by #chepner, I am not using the Host_storing_func.
Heredocs for sshpaas are somewhat tricky. You have to escape every back-tick and $ sign in the heredoc.
Notice the - before the heredoc start, it allows you to indent the heredoc body. Also, try to avoid back-ticks when you can. use $(command) instead.
Hope it helps.
#!/bin/bash
#######################
#Function to add hosts to the array
#the following function takes the ip addresses provided while the script is run and stores them in an array
#######################
array=( "$#" )
user="student35"
############################################################
#Collect Stats of Ping,memory,iowait time test function
############################################################
for host in ${array[#]}; do
sshpass -f /root/scripts/passwordFile.txt /usr/bin/ssh -l ${user} ${host} <<-HERE
thishost=\$(hostname)
echo "Current Host -> \$thishost";
iowaittm=\`sar 2 2|awk '/^Average/{print \$5}'\`
if [ \$iowaittm > 10 ]; then
echo "IO ==> BAD"
else
echo "IO ==> GOOD"
fi
memory=\$(free -m | grep Swap | awk '{if(\$2 == 0) print 0;else print ((\$4 / \$2 ) * 100)}')
if [ \${memory} < '10' ] ;then
echo "memory ==> good"
elif [[ "\${memory}" -ge 0 ]] && [[ "\${memory}" -le 10 ]]; then
echo "No Swap"
else
echo "memory ==> bad"\`enter code here\`
fi
ping -w2 -c2 \`hostname\` | grep "packet loss"|awk -F, '{print \$3}'|awk -F% '{print \$1}'|sed 's/^ *//'|awk '{if (\$1 == 0) print "Yes" ;else print "No"}'
HERE
done

notify-send from within a bash script

I'd like to use notify-send from within a bash script that is running in the background to inform the user about the progress of the script. More specifically this is a script that automagically runs when a USB flash drive is inserted and runs a scan with ClamAV.
Specifically at line 30 and line 66. So far, I'm not having any luck. Can someone give me some advice/help? Thanks.
#!/bin/bash
#doOnUSBinsert_0.2.sh
#Author : Totti
# Make it executable by running 'sudo chmod x doOnUSBinsert_0.2.sh'
if ! [ -f /etc/udev/rules.d/80-doOnUSBinsert.rules ]
then # rule not added
cp "$0" /usr/bin/doOnUSBinsert
chmod u x /usr/bin/doOnUSBinsert
# echo 'SUBSYSTEM=="usb", ACTION=="add", RUN ="/path/to/script.sh"' | sudo tee /etc/udev/rules.d/80-clamscan.rules
echo 'SUBSYSTEM=="usb", ACTION=="add", RUN ="/usr/bin/doOnUSBinsert & "' | tee /etc/udev/rules.d/80-doOnUSBinsert.rules
if [ $? -eq 0 ]
then
echo 'Rule Successfully added. See file "/usr/bin/doOnUSBinsert" if you wish to edit the command'
exit 0
else
echo 'ERROR while adding rule'
exit 1
fi
fi
lfile="/tmp/doOnUSBinsert.log" # udev
lfile2="/tmp/clamscanFromUdev.log" # clamscan
lfile3="/tmp/doOnUSBinsert_mount.log" # mount
notify-send "USB SCAN ON INSERT" "Currently scanning with ClamAV"
main ()
{
sleep 12 # let the partitions to mount
#cat /proc/$$/environ | tr '�' 'n' >> /tmp/udevEnvirn.txt
echo "found $ID_SERIAL" >> "$lfile"
cat /etc/mtab | grep "^$part_c" >> "$lfile.3"
if [ "$ID_SERIAL"x = 'x' ]
then
echo "Exiting on empty ID_SERIAL" >> "$lfile"
exit 1
fi
#Eg: ID_SERIAL --> /dev/disk/by-id/usb-sandisk....42343254343543
#i=0
echo 'searching partitions' >> "$lfile"
for partitionPath in $( find /dev/disk/by-id/ -name "*$ID_SERIAL*part*" )
do
echo "current partition = $partitionPath" >> "$lfile"
# part[i ]="$( readlink -f "$partition" )" # Eg Output: /dev/sdb1 , /dev/sdb2
part_c="$( readlink -f $partitionPath )"
mpoint="$( cat /etc/mtab | grep "^$part_c" | awk '{print $2}' )"
echo "partitionPath= $partitionPath, part = $part_c, mountpoint= $mpoint" >> "$lfile"
echo "Scaning --> $mpoint" >> "$lfile.2"
############################################
clamscan -r --bell "$mpoint"/* >> "$lfile.2"
#############################################
done
}
notify-send "USB SCAN ON INSERT" "Finished scanning with ClamAV"
main &
echo ______________________________________ >> "$lfile"
exit 0
I'm pretty new to the linux world, but while looking for a solution for a similar project I found THIS
Tip: An overview on the available icons can be found here. To send
desktop notification from a background script running as root (replace
X_user and X_userid with the user and userid running X respectively):
sudo -u X_user DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/X_userid/bus notify-send 'Hello world!' 'This is an example notification.'
Hope this will help others.
Depending on how you are running the script it may not have access to the display variable. Try running export DISPLAY=:0.0 prior to the command.
If you are running the script as a different user, ie root, then you may also need to run it as su - <logged in user> -c notify-send ... (I usually don't need to do this, but I remember having to at one point - but I cant recall which distro or version I was on at the time.)

Collecting data from a UNIX program with Bash Redirection

I am attempting to write a script that outputs a CSV file (specified by the user), from the temperature value given by the acpi program. Each data entry is appended to the CSV file, every second (for now).
I have two issues that need to be resolved in my script:
To fix the errors in the script (such as the file access error on line 14), and to allow the user to specify the time delay.
Here's the bash script:
#!/bin/bash
# This script monitors the CPU temperature (in degrees C) over time, and
# records the data to a CSV file.
# Written by Ben Cottrell
function start_log()
{
touch "$output_csv"
if hash acpi 2>/dev/null; then
echo -e "time,temperature\n" > "$output_csv"
while (true); do
sleep 1
echo -e "$(date +%T),$(acpi -t | awk '{print $4}')\n" >> "$output_csv"
done
else
echo "Error: acpi is not installed. Please execute \"sudo apt-get install acpi, before running this script.\""
fi
}
if [ "$1" ]; then
local output_csv = $1
start_log
else
echo -e "Error: No filename specified.\n"
fi
echo -e "\n"
Here's a rewrite with the worst errors and idiom violations fixed.
#!/bin/bash
function start_log()
{
local output_csv=$1
local delay=$2
if hash acpi 2>/dev/null; then
echo "time,temperature" > "$output_csv"
while true; do
sleep "$delay"
echo "$(date +%T),$(acpi -t | awk '{print $4}')"
done >>"$output_csv"
else
cat <<________error >&2
$0: Error: acpi is not installed.
Please execute \"sudo apt-get install acpi\" before running this script.
________error
exit 2
fi
}
delay=1 # default to 1 sec delay
while true; do
case $1 in
-t) shift; delay=$1; shift;; # Maybe validate the value of $delay
-*) echo "$0: Unknown option: '$1'" >&2; exit 1;;
*) break;;
esac
done
if [ "$1" ]; then
start_log "$1" "$delay"
else
echo "$0: Error: No filename specified." >&2
exit 1
fi
Including the program name in error messages is useful when it is being invoked from another script, which might get invoked from another script, etc.
Notice how the redirection is only done once, after the loop, instead of repeatedly inside the main loop.
Notice also how the function receives its parameters from the caller, instead of pulling in global variables.
Hand-crafting the option parsing is hardly more complex than doing it "properly" with getopts for simple processing.
Try this.
#!/bin/bash
# This script monitors the CPU temperature (in degrees C) over time, and
# records the data to a CSV file.
# Written by Ben Cottrell
function start_log()
{
touch "$output_csv"
read -p 'Enter the time interval for collecting data in seconds: ' t
if hash acpi 2>/dev/null; then
echo -e "time,temperature\n" > "$output_csv"
while (true); do
sleep $t
echo -e "$(date +%T),$(acpi -t | awk '{print $4}')\n" >> "$output_csv"
done
else
echo "Error: acpi is not installed. Please execute \"sudo apt-get install acpi, before running this script.\""
fi
}
if [ -n "$1" ]; then
local output_csv = $1
start_log
else
echo -e "Error: No filename specified.\n"
fi
echo -e "\n"

Resources