Unix Shell Scripting - Add Extra Column In While Loop - bash

I have a file server.txt containing different hostnames like:
hostname1.com
hostname2.com
My shell script servers.sh is written to get the /etc/passwd and /etc/group files from the list in servers.txt file.
I wish to add the hostname from which the entries came from in my final output file. My script looks something like below:
while read HOST ;
do
sshpass -p $password ssh -n $username#$HOST 'cat /etc/passwd'>>users.txt
sshpass -p $password ssh -n $username#$HOST 'cat /etc/group'>>groups.txt
done < servers.txt
echo -e "UserName;UID;GID;HomeDir;Shell" > final_users.csv
cut -d: -f1,3,4,6,7 users.txt | tr ':' ';'>> final_users.csv
echo -e "GroupName;GID;Members" > final_groups.csv
awk -F '[:,]' '{for(i=4;i<=NF;i++)print$1";"$3";"$i}' groups.txt >> final_groups.csv
The goal is to add another column in both final_users.csv and final_groups.csv like hostname so I can know which servers each entry came from.

Try like this:
while read HOST ;
do
sshpass -p $password ssh -n $username#$HOST 'cat /etc/passwd'>>users.txt
sshpass -p $password ssh -n $username#$HOST 'cat /etc/group'>>groups.txt
done < servers.txt
echo -e "UserName;UID;GID;HomeDir;Shell;Hostname" > final_users.csv
echo "`cut -d: -f1,3,4,6,7 users.txt | tr ':' ';'`;$HOST">> final_users.csv
echo -e "GroupName;GID;Members;Hostname" > final_groups.csv
echo "`awk -F '[:,]' '{for(i=4;i<=NF;i++)print$1";"$3";"$i}' groups.txt`;$HOST" >> final_groups.csv

Think I got it:
while read HOST ;
do
sshpass -p $password ssh -n $username#$HOST 'cat /etc/passwd'>>users_1.txt
while IFS= read -r line; do echo "$line:"$HOST; done < users_1.txt >> users_2.txt
sshpass -p $password ssh -n $username#$HOST 'cat /etc/group'>>groups_1.txt
while IFS= read -r line; do echo "$line:"$HOST; done < groups_1.txt >> groups_2.txt
done < servers.txt
echo -e "UserName;UID;GID;HomeDir;Shell;Hostname" > final_users.csv
cut -d: -f1,3,4,6,7,8 users_2.txt | tr ':' ';'>> final_users.csv
echo -e "GroupName;GID;Members;Hostname" > final_groups.csv
awk -F'[:,]' -v OFS=';' '{for(i=4;i<NF;i++) print $1, $3, $i, $NF}' groups_2.txt >> final_groups.csv

Related

Why SNMP or SSH prevents the while to loop from iterating?

I'm writing a code in order to capture information about routers thanks to SSH and SNMP. I work on network with 4 different router configurations. In my script bash, a while loop read line by line the IP address of routers then try to know the configuration and finally capture information about my router.
To do so, I use a while loop but it only run for one IP address...
I assume that the problem comes from SNMP or SSH.
I tried the while loop just with
echo $line
The while loop :
while IFS= read -r line
do
echo
echo "line : $line"
echo
typeRouter=""
version=""
# type multitech V4.1
type=$(snmpget -v3 -u Ai -l authpriv -a SHA -A "pass" -x AES -X "pass" "$line" 1.3.6.1.4.1.995.14.1.1.8.3.0 | cut -d\" -f2 | cut -d. -f1 )
if [[ "$type" == "4" ]]; then
typeRouter=2
version=4
fi
# type sierra
type=$(snmpget -v3 -u Ai -l authpriv -a SHA -A "pass" -x AES -X "pass" "$line" iso.3.6.1.4.1.20542.9.1.1.9.7.0 | cut -d\" -f2 )
if [[ "$type" == "LX40" ]]; then
typeRouter=3
fi
case $typeRouter in
2)
echo "Mutlitech"
# version 4.1 et ultérieur
MultiV4 "$line" "$txtfile" "pass" "pass"
;;
3) # TODO PASSWORD
echo "Sierra"
Sierra "$line" "$txtfile" "pass" "pass"
;;
*)
echo
echo "Unknown router model, IP : $line"
echo
;;
esac
done < "file"
The function called
MultiV4 () {
# "$1" : #IP
# $2 : txtfile
# "$3" : mdp snmp
# $4 : mdp ssh
# Files
txtfile=$2
model="Multitech"
MACtxt="ephemere/mac.txt"
MacAddressRouter=$(snmpget -v3 -u Ai -l authpriv -a SHA -A "$3" -x AES -X "$3" "$1" iso.3.6.1.4.1.995.14.1.1.5.1.0 | cut -d\" -f2)
echo "Router MAC address : $MacAddressRouter"
sshpass -p "$4" ssh -o StrictHostKeyChecking=no admin#"$1" "cat /proc/net/arp" > "$MACtxt"
rssiPacket=$(snmpget -v3 -u Ai -l authpriv -a SHA -A "$3" -x AES -X "$3" "$1" iso.3.6.1.4.1.995.14.1.1.3.5.0 | cut -d: -f2 | cut -d\" -f2 | tr -d dBm)
echo "RSSI packet : $rssiPacket"
firmware=$(snmpget -v3 -u Ai -l authpriv -a SHA -A "$3" -x AES -X "$3" "$1" iso.3.6.1.4.1.995.14.1.1.3.4.0 | cut -d: -f2 | cut -d\" -f2)
echo "Firmware version : $firmware"
echo "$MacAddressRouter", "$MacAddressPlayer", "$rssiPacket", "$model", "$firmware", "$imei", "$SIMStatus", "$operator", "$connectionType", "$IPaddress", "$SINR", "$RSRP", "$RSRQ" >> "$txtfile"
}
My file
10.252.144.138
10.252.144.140
10.252.144.23
10.252.144.15
10.252.144.134
Can you help me why the loop running only one time ?
I solved the problem using
ssh -n
Go see this answer for more explanation : Shell script while read line loop stops after the first line

Storing a line in a variable

Hi I have the following batch script where I submitted each file to a separate processing as follows:
for file in ../Positive/*.txt_rn; do
bsub <<EOF
#BSUB -L /bin/bash
#BSUB -W 150:00
#BSUB -M 10000
#BSUB -n 3
#BSUB -e /somefolder/errors/%J.err
#BSUB -o /somefolder/errors/%J.out
while read line; do
name=`cat \$line | awk '{print $1":"$2"-"$3}'`
four=`cat \$line | awk '{print $4}' | cut -d\: -f4`
fasta=\$name".fa"
op=\$name".rs"
echo \$name | xargs samtools faidx /somefolder/rn4/Rattus_norvegicus/UCSC/rn4/Sequence/WholeGenomeFasta/genome.fa > \$fasta
Process -F \$fasta -M "list_"\$four".txt" -p 0.003 | awk '(\$5 >= 0.67)' > \$op
if [ -s "\$op" ]
then
cat "\$line" >> ../Positive_Strand/$file".cons"
fi
rm \$lne
rm \$op
rm \$fasta
done < $file
EOF
done
I am am somehow unable to store the values of the column from the line (which is in $line variable into the $name and $four variable and hence unable to carry on further processes. Also any suggestions to edit the code for a better version of it would be welcome.
If you change EOF to 'EOF' then you will more properly disable shell interpretation. Your problem is that your back-ticks (`) are not escaped.
I've fixed your indentation and cleaned up some of your code. Note that the syntax highlighting here doesn't understand cat <<'EOF'. If you paste that into vim with highlighting enabled, you'll see that block is all the same color since it's just a string.
bsub_helper() {
cat <<'EOF'
#BSUB -L /bin/bash
#BSUB -W 150:00
#BSUB -M 10000
#BSUB -n 3
#BSUB -e /somefolder/errors/%J.err
#BSUB -o /somefolder/errors/%J.out
while read line; do
name=`cat $line | awk '{print $1":"$2"-"$3}'`
four=`cat $line | awk '{print $4}' | cut -d: -f4`
fasta="$name.fa"
op="$name.rs"
genome="/somefolder/rn4/Rattus_norvegicus/UCSC/rn4/Sequence/WholeGenomeFasta/genome.fa"
echo $name | xargs samtools faidx "$genome" > "$fasta"
Process -F "$fasta" -M "list_$four.txt" -p 0.003 | awk '($5 >= 0.67)' > "$op"
if [ -s "$op" ]
then
cat "$line" >> "../Positive_Strand/$file.cons"
fi
rm "$lne" "$op" "$fasta"
EOF
echo " done < \"$1\""
}
for file in ../Positive/*.txt_rn; do
bsub_helper "$file" |bsub
done
I created a helper function because I needed to get the input in two commands. I am assuming that $file is the only variable in that block that you want interpreted. I also surrounded that variable (among others) with quotes so that the code can support file names with spaces in them. The final line of the helper has nested double quotes for this reason.
I left your echo $name | xargs … line alone because it's so odd. Without quotes around $name, xargs will take each whitespace-separated entry as its own file. With quotes, xargs will only supply one (likely invalid) file name to samtools.
If $name is a single file, try:
samtools faidx "$genome" "$name" > "$fasta"
If $name is multiple files and none of them have spaces, try:
samtools faidx "$genome" $name > "$fasta"
The only reason to use xargs here would be if you have too much content for one command line, but if you're running echo $name | xargs then you'll run into the same problem.

Accesing remote server in linux using bash script

n=rijotests143.revsw.net
echo $n
me=$(echo $n | sed 's/"/ /g')
echo $me
u=$(echo $me | sed 's/\./\_/g')
echo $u
ssh -i m_bp.pem ubuntu#95.68.74.51 'bash -s' < domain_delbp.sh
I am getting accessed to the remote server and passing values in domain_delbp.sh file. In this i need to pass a value of $u in domain_delbp.sh and need to execute the command
cat /opt/rijo/var/$u.json
Please help me
You can simply parse the value to the script and in the script you can access that value as $1 variable.
n=rijotests143.revsw.net
echo $n me=$(echo $n | sed 's/"/ /g')
echo $me u=$(echo $me | sed 's/./_/g')
echo $u
ssh -i m_bp.pem ubuntu#95.68.74.51 'bash -s' < domain_delbp.sh $u

Verifying if a file exist (script shell)

This is my code:
nb_lignes=`wc -l $1 | cut -d " " -f1`
for i in $(seq $(($nb_lignes - 1)) )
do
machine=`head $1 -n $i | tail -1`
machine1=`head $1 -n $nb_lignes | tail -1`
ssh root#$machine -x " scp /home/file.txt root#$machine1:/home && rm -r /home/file.txt"
done
I'd like to verify if file.txt exist in such machine before the scp et rm ,Please i ask how can i modify this script ?
Thank you.
You can use test command: test -f file && scp ...
ssh root#$machine -x " test -f /home/file.txt && scp /home/file.txt
root#$machine1:/home && rm -r /home/file.txt"

Bulk account creation from text file

I have the following script, it reads from users.txt the first and second fields and uses them to generate the username and password and creates the accounts for each line. the problem is that the script is only creating accounts for the first 2 lines and not for the rest
#!/bin/bash
FILE=/home/knoppix/users.txt
USERSH=/bin/bash
while IFS=":" read GECOS USRGRP ; do
groupadd -f $USRGRP
USERNM="u$(cat /dev/urandom| tr -dc '0-9' | fold -w 6| head -n 1)"
USERPW=$(cat /dev/urandom| tr -dc 'a-zA-Z0-9' | fold -w 6| head -n 1)
useradd $USERNM -p $USERPW -g $USRGRP -c "$GECOS,$USRGRP" -d $HOME/$USERNM -s $USERSH -m
ACCNT=$(grep $USRNM /etc/passwd)
echo "${tgrn}Account creation successful!${tr}"
echo "$ACCNT"
echo "Credentials"
echo "${tred}Username:${tr} $USERNM ${tred}Password:${tr} $USERPW"
echo
done < $FILE
#!/bin/bash
while IFS=: read GECOS USRGRP; do
# your groupadd and useradd commands here
done < /home/knoppix/users.txt
#!/bin/bash
for line in $file; do
# make the account
done
rm $file
touch $file

Resources