How to run 2 loops simultaneously in bash script - bash

I am very new in writing code, so it might be a silly question, but an answer is highly appreciated to enhance my learning. I have written a simple bash script as below. But how can I optimize this code by using loop, array? I can understand if I use two loops, I can make the lines of code shorter. Please help:
#!/bin/bash
zs=10.0.3.10
zb=/usr/local/bin/zabbix_sender
zh=zabbix
# ql1 = queue link
ql1=https://sqs.us-west-2.amazonaws.com/843390035802/testService1
val1=$(aws sqs get-queue-attributes --queue-url $ql1 --attribute-names ApproximateNumberOfMessages --region us-west-2 --output text | awk '{print $2}')
echo "$ql1 count is $val1"
$zb -z $zs -s $zh -k testService1 -o val1 >/dev/null 2>&1
ql2=https://sqs.us-west-2.amazonaws.com/853390078801/testService2
val2=$(aws sqs get-queue-attributes --queue-url $ql2 --attribute-names ApproximateNumberOfMessages --region us-west-2 --output text | awk '{print $2}')
echo "$ql2 count is $val2"
$zb -z $zs -s $zh -k testService2 -o val2 >/dev/null 2>&1
ql3=https://sqs.us-west-2.amazonaws.com/843393305801/testService3
val3=$(aws sqs get-queue-attributes --queue-url $ql3 --attribute-names ApproximateNumberOfMessages --region us-west-2 --output text | awk '{print $2}')
echo "$ql3 count is $val3"
$zb -z $zs -s $zh -k testService3 -o val3 >/dev/null 2>&1
ql4=https://sqs.us-west-2.amazonaws.com/875660005801/testService4
val4=$(aws sqs get-queue-attributes --queue-url $ql4 --attribute-names ApproximateNumberOfMessages --region us-west-2 --output text | awk '{print $2}')
echo "$ql4 count is $val4"
$zb -z $zs -s $zh -k testService4 -o val4 >/dev/null 2>&1
ql5=https://sqs.us-west-2.amazonaws.com/843390635802/testService5
val5=$(aws sqs get-queue-attributes --queue-url $ql5 --attribute-names ApproximateNumberOfMessages --region us-west-2 --output text | awk '{print $2}')
echo "$ql5 count is $val5"
$zb -z $zs -s $zh -k testService2 -o val5 >/dev/null 2>&1
In above code at this step
$zb -z $zs -s $zh -k testService2 -o val5 >/dev/null 2>&1 I used -k as different 5 values. So how can I arrange it and work the code as same as above?

One loop is sufficient to eliminate the code duplication, and we don't need an array - we can read one queue link after the other in the loop. The variable argument to the option -k can be extracted from the queue link by removing the URL part up to to last / with the shell parameter expansion ${parameter##word}.
zs=10.0.3.10
zb=/usr/local/bin/zabbix_sender
zh=zabbix
# ql = queue link
while read ql
do
val=$(aws sqs get-queue-attributes --queue-url $ql --attribute-names ApproximateNumberOfMessages --region us-west-2 --output text | awk '{print $2}')
echo "$ql count is $val"
$zb -z $zs -s $zh -k ${ql##*/} -o $val >/dev/null 2>&1
done <<END
https://sqs.us-west-2.amazonaws.com/843390035802/testService1
https://sqs.us-west-2.amazonaws.com/853390078801/testService2
https://sqs.us-west-2.amazonaws.com/843393305801/testService3
https://sqs.us-west-2.amazonaws.com/875660005801/testService4
https://sqs.us-west-2.amazonaws.com/843390635802/testService5
END

Related

Writing a comparison BATCH file to verify sha256sum to released code

Trying to write a script that takes 2 arguments ($1 and $2) one to represent the $hash and the $file_name.
I am trying to utilize jq to parse the required data to download and compare PASS or FAIL.
I see to be stuck trying to think this out.
Here is my code
#!/usr/bin/env sh
#
# Sifchain shasum check (revised).
#
# $1
hash_url=$( curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | jq '.[] | select(.name=="v0.10.0-rc.4")' | jq '.assets[]' | jq 'select(.name=="sifnoded-v0.10.0-rc.4-linux-amd64.zip.sha256")' | jq '.browser_download_url' | xargs $1 $2 )
echo $hash_url
# $2
hash=$( curl -s -L $hash_url | jq'.$2')
file_name=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | jq '.[] | .name')
#
#
echo $hash | sha256sum
echo $file_name | sha256sum #null why?
echo "\n"
## version of the release $1, and the hash $2
## sha256 <expected_sha_256_sum> <name_of_the_file>
sha256() {
if echo "$1 $2" #| sha256sum -c --quiet
then
echo pass $1 $2
exit 0
else
echo FAIL $1 $2
exit 1
fi
}
# Invoke sha256
sha256 $hash_url $file_name
Ideally this should work for any comparison of hash with correct file, pulling the 2 parameters when the BASH script is invoked.
I can suggest the following corrections/modifications:
#!/bin/bash
#sha file
SHA_URL=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.name |test("\\.sha256$")) | .browser_download_url')
SHA_VALUE=$(curl -s -L $SHA_URL| tr 1 2)
FILENAME=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.content_type =="application/zip") | .name')
#added just for testing, I'm assuming you have the files locally allready
FILEURL=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.content_type =="application/zip") | .browser_download_url')
wget --quiet $FILEURL -O $FILENAME
echo $SHA_VALUE $FILENAME | sha256sum -c --quiet >/dev/null 2>&1
RESULT=$?
if [ $RESULT -eq 0 ]; then
echo -n "PASS "
else
echo -n "FAIL "
fi
echo $SHA_VALUE $FILENAME
exit $RESULT
Notes:
jq
--arg VERSION v0.10.0-rc.4 creates a "variable" to be used in the script
-r - raw output, strings are not quoted
test("\\.sha256$") - regular expresion, used to search for a generic sha256, so you don't have to hardcode the full name
select(.content_type =="application/zip") - I'm assuming that's the file you are searching for
wget is used just for demo purpose, to download the file, I'm assuming you already have the file on your machine
sha256sum -c --quiet >/dev/null 2>&1 - redirecting to /dev/null is necessary because in case of error sha256sum is not quiet

Bash: use nth row as part of the command line

I am trying to use a tool called fastqtl, but it's probably less relevant here. I am interested in assigning each row of the "loc_info.txt" into the options. I wrote the following commands but it bounced back as "Error parsing command line :unrecognised option '-n+1'
Is there a way that I can make the fastQTL read and use that 1 line from "loc_info.txt" each time it runs the function?
Thanks for any suggestions!!
#!/bin/bash
tool="/path/FastQTL-2.165.linux/bin/"
vcf="/path/vcf/"
out="/path/perm_out"
for i in {1..1061}
do
${tool}fastQTL.1.165.linux --vcf ${vcf}GT.vcf.gz --bed pheno_bed.gz --region tail -n+"$i" loc_info.txt --permute 1000 --out "$i"_perm.txt
done
Read the file in a loop:
i=1
while read -r line; do
${tool}fastQTL.1.165.linux --vcf ${vcf}GT.vcf.gz --bed pheno_bed.gz --region "$line" --permute 1000 --out "$i"_perm.txt
((i++))
done < loc_info.txt
You can use a subshell for this, if you want to use the output from one command within another command so something like:
cmd1 -option $(cmd2)
here you're using the cmd2 output as input in cmd. The key here is '$' and the subshell '()'. So the solution might be:
#!/bin/bash
tool="/path/FastQTL-2.165.linux/bin/"
vcf="/path/vcf/"
out="/path/perm_out"
for i in {1..1061}
do
${tool}fastQTL.1.165.linux --vcf ${vcf}GT.vcf.gz --bed pheno_bed.gz --region $(tail -n+"$i" loc_info.txt) --permute 1000 --out "$i"_perm.txt
done
Try replacing tail -n+"$i" loc_info.txt
with $(head -n $i loc_info.txt | tail -n 1)
Example
numOfLines=$(wc -l loc_info.txt | cut -d ' ' -f 1)
for i in $(seq 1 $numOfLines)
do
${tool}fastQTL.1.165.linux --vcf ${vcf}GT.vcf.gz --bed pheno_bed.gz -
-region $(head -n $i loc_info.txt | tail -n 1) --permute 1000 --out "$i"_perm.txt
done

Bash script help/evaluation

I'm trying to learn some scripting however I can't find solution for one functionality.
Basically I would like to ask to evaluate my script as it's probably possible to reduce the complexity and number of lines.
The purpose of this script is to download random, encrypted MySQL backups from Amazon S3, restore the dump and run some random MySQL queries.
I'm not sure how to email the output from printf statements - one is for headers and second one for actual data. I've tried to format the output so it looks like below but I had to exclude the headers from the loop:
Database: Table: Entries:
database1 random_table 0
database2 random_table 0
database3 random_table 0
database4 random_table 0
I would like to include this output in the email and also change the email subject based on the success/failure of the script.
I probably use to much if loops and MySQL queries are probably to complicated.
Script:
#!/usr/bin/env bash
# DB Details:
db_user="user"
db_pass="password"
db_host="localhost"
# Date
date_stamp=$(date +%d%m%Y)
# Initial Setup
data_dir="/tmp/backup"
# Checks
if [ ! -e /usr/bin/s3cmd ]; then
echo "Required package (http://s3tools.org/s3cmd)"
exit 2
fi
if [ -e /usr/bin/gpg ]; then
gpg_key=$(gpg -K | tr -d "{<,>}" | awk '/an#example.com/ { print $4 }')
if [ "$gpg_key" != "an#example.com" ]; then
echo "No GPG key"
exit 2
fi
else
echo "No GPG package"
exit 2
fi
if [ -d $data_dir ]; then
rm -rf $data_dir/* && chmod 700 $data_dir
else
mkdir $data_dir && chmod 700 $data_dir
fi
# S3 buckets
bucket_1=s3://test/
# Download backup
for backup in $(s3cmd ls s3://test/ | awk '{ print $2 }')
do
latest=$(s3cmd ls $backup | awk '{ print $2 }' | sed -n '$p')
random=$(s3cmd ls $latest | shuf | awk '{ print $4 }' | sed -n '1p')
s3cmd get $random $data_dir >/dev/null 2>&1
done
# Decrypting Files
for file in $(ls -A $data_dir)
do
filename=$(echo $file | sed 's/\.e//')
gpg --out $data_dir/$filename --decrypt $data_dir/$file >/dev/null 2>&1 && rm -f $data_dir/$file
if [ $? -eq 0 ]; then
# Decompressing Files
bzip2 -d $data_dir/$filename
if [ $? -ne 0 ]; then
echo "Decompression Failed!"
fi
else
echo "Decryption Failed!"
exit 2
fi
done
# MySQL Restore
printf "%-40s%-30s%-30s\n\n" Database: Table: Entries:
for dump in $(ls -A $data_dir)
do
mysql -h $db_host -u $db_user -p$db_pass < $data_dir/$dump
if [ $? -eq 0 ]; then
# Random DBs query
db=$(echo $dump | sed 's/\.sql//')
random_table=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SHOW TABLES" | grep -v 'Tables' | shuf | sed -n '1p')
db_entries=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SELECT * FROM $random_table" | grep -v 'id' | wc -l)
printf "%-40s%-30s%-30s\n" $db $random_table $db_entries
mysql -h $db_host -u $db_user -p$db_pass -e "DROP DATABASE $db"
else
echo "The system was unable to restore backups!"
rm -rf $data_dir
exit 2
fi
done
#Remove backups
rm -rf $data_dir
You'll get the best answers if you ask specific questions (rather than, "please review my code")...and if you limit each post to a single question. Regarding emailing the output of your printf statements:
You can group statements into a block and then pipe the output of a block into another program. For example:
{
echo "This is a header"
echo
for x in {1..10}; do
echo "This is row $x"
done
} | mail -s "Here is my output" lars#example.com
If you want to make the email subject conditional upon the success or
failure of something elsewhere in the script, you can (a) save your
output to a file, and then (b) email the file after building the
subject line:
{
echo "This is a header"
echo
for x in {1..10}; do
echo "This is row $x"
done
} > output
if is_success; then
subject="SUCCESS: Here is your output"
else
subject="FAILURE: Here are your errors"
fi
mail -s "$subject" lars#example.com < output

Bash script help/evaluation

I'm trying to learn some scripting however I can't find solution for one functionality.
Basically I would like to ask to evaluate my script as it's probably possible to reduce the complexity and number of lines.
The purpose of this script is to download random, encrypted MySQL backups from Amazon S3, restore the dump and run some random MySQL queries.
I'm not sure how to email the output from printf statements - one is for headers and second one for actual data. I've tried to format the output so it looks like below but I had to exclude the headers from the loop:
Database: Table: Entries:
database1 random_table 0
database2 random_table 0
database3 random_table 0
database4 random_table 0
I would like to include this output in the email and also change the email subject based on the success/failure of the script.
I probably use to much if loops and MySQL queries are probably to complicated.
Script:
#!/usr/bin/env bash
# DB Details:
db_user="user"
db_pass="password"
db_host="localhost"
# Date
date_stamp=$(date +%d%m%Y)
# Initial Setup
data_dir="/tmp/backup"
# Checks
if [ ! -e /usr/bin/s3cmd ]; then
echo "Required package (http://s3tools.org/s3cmd)"
exit 2
fi
if [ -e /usr/bin/gpg ]; then
gpg_key=$(gpg -K | tr -d "{<,>}" | awk '/an#example.com/ { print $4 }')
if [ "$gpg_key" != "an#example.com" ]; then
echo "No GPG key"
exit 2
fi
else
echo "No GPG package"
exit 2
fi
if [ -d $data_dir ]; then
rm -rf $data_dir/* && chmod 700 $data_dir
else
mkdir $data_dir && chmod 700 $data_dir
fi
# S3 buckets
bucket_1=s3://test/
# Download backup
for backup in $(s3cmd ls s3://test/ | awk '{ print $2 }')
do
latest=$(s3cmd ls $backup | awk '{ print $2 }' | sed -n '$p')
random=$(s3cmd ls $latest | shuf | awk '{ print $4 }' | sed -n '1p')
s3cmd get $random $data_dir >/dev/null 2>&1
done
# Decrypting Files
for file in $(ls -A $data_dir)
do
filename=$(echo $file | sed 's/\.e//')
gpg --out $data_dir/$filename --decrypt $data_dir/$file >/dev/null 2>&1 && rm -f $data_dir/$file
if [ $? -eq 0 ]; then
# Decompressing Files
bzip2 -d $data_dir/$filename
if [ $? -ne 0 ]; then
echo "Decompression Failed!"
fi
else
echo "Decryption Failed!"
exit 2
fi
done
# MySQL Restore
printf "%-40s%-30s%-30s\n\n" Database: Table: Entries:
for dump in $(ls -A $data_dir)
do
mysql -h $db_host -u $db_user -p$db_pass < $data_dir/$dump
if [ $? -eq 0 ]; then
# Random DBs query
db=$(echo $dump | sed 's/\.sql//')
random_table=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SHOW TABLES" | grep -v 'Tables' | shuf | sed -n '1p')
db_entries=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SELECT * FROM $random_table" | grep -v 'id' | wc -l)
printf "%-40s%-30s%-30s\n" $db $random_table $db_entries
mysql -h $db_host -u $db_user -p$db_pass -e "DROP DATABASE $db"
else
echo "The system was unable to restore backups!"
rm -rf $data_dir
exit 2
fi
done
#Remove backups
rm -rf $data_dir
move out of the loop :
random_tables=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SHOW TABLES" | grep -v 'Tables')
table_nb=$(wc -l <<<"$random_tables")
and in the loop
random_table=$(sed -n $((RANDOM%table_nb+1))p <<<"$random_tables")
A remark $? is the status of latest command executed so after && rm it will not be the status of decrypt

Bulk account creation from text file

I have the following script, it reads from users.txt the first and second fields and uses them to generate the username and password and creates the accounts for each line. the problem is that the script is only creating accounts for the first 2 lines and not for the rest
#!/bin/bash
FILE=/home/knoppix/users.txt
USERSH=/bin/bash
while IFS=":" read GECOS USRGRP ; do
groupadd -f $USRGRP
USERNM="u$(cat /dev/urandom| tr -dc '0-9' | fold -w 6| head -n 1)"
USERPW=$(cat /dev/urandom| tr -dc 'a-zA-Z0-9' | fold -w 6| head -n 1)
useradd $USERNM -p $USERPW -g $USRGRP -c "$GECOS,$USRGRP" -d $HOME/$USERNM -s $USERSH -m
ACCNT=$(grep $USRNM /etc/passwd)
echo "${tgrn}Account creation successful!${tr}"
echo "$ACCNT"
echo "Credentials"
echo "${tred}Username:${tr} $USERNM ${tred}Password:${tr} $USERPW"
echo
done < $FILE
#!/bin/bash
while IFS=: read GECOS USRGRP; do
# your groupadd and useradd commands here
done < /home/knoppix/users.txt
#!/bin/bash
for line in $file; do
# make the account
done
rm $file
touch $file

Resources