The error that I am getting is with the df command on mac, I would like to specify a value in gigabytes and perhaps allow the user to choose a different drive if / is full.
destination="$HOME/Desktop/sandbox"
if [ $(df -h --output=avail /|tail -n1) -lt 300000 ]; then
echo "There is less than 300GB available..." ;
exit
else
for files in *.tar ; do echo copying "$files" ; cp "$files" "$destination" ; read -n 1 -p "Press any key..." ; done
fi
Not sure if df -h / | tail -1 | awk '{print $4}' | sed 's/..$//' is a good option
destination="$HOME/Desktop/sandbox"
freespace="$(df -h / | tail -1 | awk '{print $4}' | sed 's/..$//')"
if [ "$freespace" -lt 300 ]; then
echo "There is less than 300GB available..." ;
exit
else
for files in *.sh ; do echo copying "$files" ; cp "$files" "$destination" ; read -n 1 -p "Press any key..." ; done
fi
The problem is that you are using dh -h what means "human readable" and always append unit after the number (M, G, k...). There is a lot of options how to cut it out by grep, awk, cut or sed. But perhaps the best way is to use different option e.g. df -m to have an output in megabytes and multiply the required space.
Also I would propose to add a grep $destination (mountpoint visible in df output) to filter the required FS and to not rely that it's always last line in the output.
Try something like this:
destination="$HOME/Desktop/sandbox"
free_space=$(df -h | grep $destination | awk '{print $4}')
if [ $free_space -lt $((300 * 1024)) ]; then
echo "There is less than 300GB available..." ;
exit
else
for files in *.tar; do
echo copying "$files"
cp "$files" "$destination"
read -n 1 -p "Press any key..."
done
fi
Related
I want to write into a file in a bash script but I want to make sure that the file is backed up if it exists and I also want to avoid overwriting any existing backups.
So basically I have $FILE, if this exists, I want to move $FILE to $FILE.bak if it does not already exist, otherwise to $FILE.bak2, $FILE.bak3, etc.
Is there a shell command for this?
Using a function to find the next available name:
#!/usr/bin/env bash
function nextsuffix {
local name="$1.bak"
if [ -e "$name" ]; then
printf "%s" "$name"
else
local -i num=2
while [ -e "$name$num" ]; do
num+=1
done
printf "%s%d" "$name" "$num"
fi
}
mv "$1" "$(nextsuffix "$1")"
If foo.bak already exists, it just loops until a given foo.bakN filename doesn't exist, incrementing N each time.
You can just output to a file with a date.
FILE=~/test
echo "123" >> $FILE.$(date +'%Y%d%m')
If you want the numbers logrotate seems to be most ideal.
cp "$FILE" "$FILE.bak$(( $(grep -Eo '[[:digit:]]+' <(sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1 )) + 1 ))"
Breaking the commands down
sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1
List the last file in the directory which is sorted in numeric form
grep -Eo '[[:digit:]]+' <(sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1 ))
Strip out everything but the digits
(( $(grep -Eo '[[:digit:]]+' <(sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1 )) + 1 ))
Add one to the result
For posterity, my function with changes inspired by #Shawn's answer
backup() {
local file new n=0
local fmt='%s.%(%Y%m%d)T_%02d'
for file; do
while :; do
printf -v new "$fmt" "$file" -1 $((++n))
[[ -e $new ]] || break
done
command cp -vp "$file" "$new"
done
}
I like to cp not mv.
This script should output a warning notification for the utilization of the main disk if over 50%, but it provides no output. My disk is currently sat at 60% so it should in theory work.
I have added an else statement to identify if the loop is not working but the else statement isnt triggered.
I'm provided no error so its hard to identify where i have gone wrong specifically.
#!/bin/bash
df -H | grep /dev/sda2 | awk '{ printf "%d", $5}' > diskOutput.txt
input="diskOutput.txt"
while IFS= read -r line
do
if [ $line -gt 50 ]
then
up="`uptime | cut -b 1-9`"
output="WARNING UTILISATION $line - $up"
echo "$output"
else
echo "no-in"
fi
done < $input
#rm diskOutput.txt
echo "finished"
Try this.
#!/bin/bash
df -H | grep /dev/sda2 | awk '{ printf "%d", $5}' > diskOutput.txt
echo "" >>diskOutput.txt
input="diskOutput.txt"
while IFS= read -r line
do
if [ $line -gt 50 ]
then
up="`uptime | cut -b 1-9`"
output="WARNING UTILISATION $line - $up"
echo "$output"
else
echo "no-in"
fi
done < $input
#rm diskOutput.txt
echo "finished"
You are setting an internal field separator as space here.
while IFS= read -r line
But when creating file, with %d you are removing all char except digits.
i'm trying to write a code which will print all files taking more than min_size (lets say 10G) in a directory. the problem is output off the below code is all files irrespective of the min_size. i will be getting other details like mtime , owner as well later in the code but this part itself doesnt work fine, whats wrong here ?
#!/bin/sh
if (( $# <3 )); then
echo "$0 dirname min_size count"
exit 1
else
dirname="$1";
min_size="$2";
count="$3";
#shift 3
fi
tmpfile=$(mktemp /lawdump/pulkit/files.XXXXXX)
exec 3> "$tmpfile"
find "${dirname}" -type f -print0 2>&1 | grep -v "Permission denied" | xargs -0 -I {} echo "{}" > "$tmpfile"
for i in `cat tmpfile`
do
x="`du -ah $i | awk '{print $1}' | grep G | sort -nr -k 1`"
size=$(echo $x | sed 's/[A-Za-z]*//g')
if [ size > $min_size ];then
echo $size
fi
done
Note : i know this can be done through find or du but i need to write a shell script to have an email sent out regularly with all the details.
I'm new at Bashing and wrote a code to check my photos files but find it very slow and gets a few empty returns checking 17000+ photos. Is there any way to use all 4 cpus running this script and so speed it up
Please help
#!/bin/bash
readarray -t array < ~/Scripts/ourphotos.txt
totalfiles="${#array[#]}"
echo $totalfiles
i=0
ii=0
check1=""
while :
do
check=${array[$i]}
if [[ ! -r $( echo $check ) ]] ; then
if [ $check = $check1 ]; then
echo "empty "$check
else
unset array[$i]
ii=$((ii + 1 ))
fi
fi
if [ $totalfiles = $i ]; then
break
fi
i=$(( i + 1 ))
done
if [ $ii -gt "1" ]; then
notify-send -u critical $ii" files have been deleted or are unreadable"
fi
It's a filesystem operation so multiple cores will hardly help.
Simplification might:
while read file; do
i=$((i+1)); [ -e "$file" ] || ii=$(ii+1));
done < "$HOME/Scripts/ourphotos.txt"
#...
Two points:
you don't need to keep the whole file in memory (no arrays needed)
$( echo $check ) forks a proces. You generally want to avoid forking and execing in loops.
This is an old question, but a common problem lacking an evidence-based solution.
awk '{print "[ -e "$1" ] && echo "$2}' | parallel # 400 files/s
awk '{print "[ -e "$1" ] && echo "$2}' | bash # 6000 files/s
while read file; do [ -e $file ] && echo $file; done # 12000 files/s
xargs find # 200000 files/s
parallel --xargs find # 250000 files/s
xargs -P2 find # 400000 files/s
xargs -P96 find # 800000 files/s
I tried this on a few different systems and the results were not consistent, but xargs -P (parallel execution) was consistently the fastest. I was surprised that xargs -P was faster than GNU parallel (not reported above, but sometimes much faster), and I was surprised that parallel execution helped so much — I thought that file I/O would be the limiting factor and parallel execution wouldn't matter much.
Also noteworthy is that xargs find is about 20x faster than the accepted solution, and much more concise. For example, here is a rewrite of OP's script:
#!/bin/bash
total=$(wc -l ~/Scripts/ourphotos.txt | awk '{print $1}')
# tr '\n' '\0' | xargs -0 handles spaces and other funny characters in filenames
found=$(cat ~//Scripts/ourphotos.txt | tr '\n' '\0' | xargs -0 -P4 find | wc -l)
if [ $total -ne $found ]; then
ii=$(expr $total - $found)
notify-send -u critical $ii" files have been deleted or are unreadable"
fi
I'm trying to learn some scripting however I can't find solution for one functionality.
Basically I would like to ask to evaluate my script as it's probably possible to reduce the complexity and number of lines.
The purpose of this script is to download random, encrypted MySQL backups from Amazon S3, restore the dump and run some random MySQL queries.
I'm not sure how to email the output from printf statements - one is for headers and second one for actual data. I've tried to format the output so it looks like below but I had to exclude the headers from the loop:
Database: Table: Entries:
database1 random_table 0
database2 random_table 0
database3 random_table 0
database4 random_table 0
I would like to include this output in the email and also change the email subject based on the success/failure of the script.
I probably use to much if loops and MySQL queries are probably to complicated.
Script:
#!/usr/bin/env bash
# DB Details:
db_user="user"
db_pass="password"
db_host="localhost"
# Date
date_stamp=$(date +%d%m%Y)
# Initial Setup
data_dir="/tmp/backup"
# Checks
if [ ! -e /usr/bin/s3cmd ]; then
echo "Required package (http://s3tools.org/s3cmd)"
exit 2
fi
if [ -e /usr/bin/gpg ]; then
gpg_key=$(gpg -K | tr -d "{<,>}" | awk '/an#example.com/ { print $4 }')
if [ "$gpg_key" != "an#example.com" ]; then
echo "No GPG key"
exit 2
fi
else
echo "No GPG package"
exit 2
fi
if [ -d $data_dir ]; then
rm -rf $data_dir/* && chmod 700 $data_dir
else
mkdir $data_dir && chmod 700 $data_dir
fi
# S3 buckets
bucket_1=s3://test/
# Download backup
for backup in $(s3cmd ls s3://test/ | awk '{ print $2 }')
do
latest=$(s3cmd ls $backup | awk '{ print $2 }' | sed -n '$p')
random=$(s3cmd ls $latest | shuf | awk '{ print $4 }' | sed -n '1p')
s3cmd get $random $data_dir >/dev/null 2>&1
done
# Decrypting Files
for file in $(ls -A $data_dir)
do
filename=$(echo $file | sed 's/\.e//')
gpg --out $data_dir/$filename --decrypt $data_dir/$file >/dev/null 2>&1 && rm -f $data_dir/$file
if [ $? -eq 0 ]; then
# Decompressing Files
bzip2 -d $data_dir/$filename
if [ $? -ne 0 ]; then
echo "Decompression Failed!"
fi
else
echo "Decryption Failed!"
exit 2
fi
done
# MySQL Restore
printf "%-40s%-30s%-30s\n\n" Database: Table: Entries:
for dump in $(ls -A $data_dir)
do
mysql -h $db_host -u $db_user -p$db_pass < $data_dir/$dump
if [ $? -eq 0 ]; then
# Random DBs query
db=$(echo $dump | sed 's/\.sql//')
random_table=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SHOW TABLES" | grep -v 'Tables' | shuf | sed -n '1p')
db_entries=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SELECT * FROM $random_table" | grep -v 'id' | wc -l)
printf "%-40s%-30s%-30s\n" $db $random_table $db_entries
mysql -h $db_host -u $db_user -p$db_pass -e "DROP DATABASE $db"
else
echo "The system was unable to restore backups!"
rm -rf $data_dir
exit 2
fi
done
#Remove backups
rm -rf $data_dir
You'll get the best answers if you ask specific questions (rather than, "please review my code")...and if you limit each post to a single question. Regarding emailing the output of your printf statements:
You can group statements into a block and then pipe the output of a block into another program. For example:
{
echo "This is a header"
echo
for x in {1..10}; do
echo "This is row $x"
done
} | mail -s "Here is my output" lars#example.com
If you want to make the email subject conditional upon the success or
failure of something elsewhere in the script, you can (a) save your
output to a file, and then (b) email the file after building the
subject line:
{
echo "This is a header"
echo
for x in {1..10}; do
echo "This is row $x"
done
} > output
if is_success; then
subject="SUCCESS: Here is your output"
else
subject="FAILURE: Here are your errors"
fi
mail -s "$subject" lars#example.com < output