While loop fail to read all line from file - bash

So I got this bash script that perform a backup command from remote server, each server hostname is defined on file called /tmp/quarantine.list. The content is like this:
server1
server2
server1 and server2 is an ssh alias to my remote server, so if I need to access either of them I just run e.g. ssh server1.
I need to backup each server in /tmp/quarantine.list, so I loop each line using while loop. Here is my code.
# Reading from /tmp/quarantine.list
while IFS= read -r list; do
if [ "$list" == "server4" ]; then
users="adminjoe"
fi
tempfile="${logdir}/rsync_$list.log"
SECONDS=0
set +x
if rsync -ca --stats --info=progress2 --remove-source-files $list:/home/${users}/backup/20* $backup_location/$list >> $tempfile 2>&1; then
ssh $list "find /home/${users}/backup/20* -type d -empty -delete"
else
echo "Error while running rsync"
exit 1
fi
set -x
elapsed_time=$SECONDS
transferred_size=$(gawk 'BEGIN { FS=":" } {gsub(/^[ \t]+|[ \t]+$/, "", $2)} /Total transferred file size/ { print $2 }' $tempfile | tr -cd "[:digit:]")
echo "Backup of $list done at $(/bin/date +'%Y-%m-%d_%H:%M:%S') --> $(bytesToHumanReadable $transferred_size)" | tee -a "${HOME}/backup.log"
echo "<b>$list --> $(bytesToHumanReadable $transferred_size)</b>" >> $konten
echo "$transferred_size $list" >> $summaryfile_size
echo "$elapsed_time $list" >> $summaryfile_time
done < "/tmp/quarantine.list"
rm "/tmp/quarantine.list"
The problem is my script only pick the first line of /tmp/quarantine.list, so for example if there was two remote in quarantine.
server2
server6
The script only process server2, while server6 remains untouched. What's weird is, when I put --dry-run option to rsync, it will process all line in /tmp/quarantine.list without problem !
Without --dry-run
+ echo 'Backup of server2 done at 2021-10-28_16:55:26 --> 3.28 GB'
Backup of server2 done at 2021-10-28_16:55:26 --> 3.28 GB
++ bytesToHumanReadable 3280769140
++ S=("bytes" "kB" "MB" "GB" "TB" "PB" "EB" "YB" "ZB")
++ local i=3280769140 d= s=0 S
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 14
++ i=3280769
++ s=1
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 76
++ i=3280
++ s=2
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 28
++ i=3
++ s=3
++ (( i > 1000 && s < 9-1 ))
++ echo '3.28 GB'
+ echo '<b>server2 --> 3.28 GB</b>'
+ echo '3280769140 server2'
+ echo '1883 server2'
+ IFS=
+ read -r list
+ rm /tmp/quarantine.list
With --dry-run
+ echo 'Backup of server2 done at 2021-10-28_16:07:14 --> 3.28 GB'
Backup of server2 done at 2021-10-28_16:07:14 --> 3.28 GB
++ bytesToHumanReadable 3280769140
++ S=("bytes" "kB" "MB" "GB" "TB" "PB" "EB" "YB" "ZB")
++ local i=3280769140 d= s=0 S
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 14
++ i=3280769
++ s=1
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 76
++ i=3280
++ s=2
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 28
++ i=3
++ s=3
++ (( i > 1000 && s < 9-1 ))
++ echo '3.28 GB'
+ echo '<b>server2 --> 3.28 GB</b>'
+ echo '3280769140 server2'
+ echo '32 server2'
+ IFS=
+ read -r list
+ '[' server6 == server4 ']'
As you can see, the latter continues with server6 as opposed of the former. I just wondering what I'd do wrong here?

This can happen if the last line of the file does not end with a newline. Then read will actually read the line, but it will return a non-zero status (ref: https://www.gnu.org/software/bash/manual/bash.html#index-read). This non-zero status causes the while loop to end.
The idiomatic way to handle this is:
while IFS= read -r list || [[ -n $list ]]; do

Related

Bash Script - sourceFile not printing to targetFile

The following code should be printing the contents of the sourceFile to targetFile, line by line, with each line having in front 'wireless-key s:', but it only prints 'wireless-key s:' to the targetFile.
#!/bin/bash
sourceFile="file1.log"
targetFile="/etc/network/interfaces"
numLines="$(wc -l < "${sourceFile}")"
counter=5
lineNumber=5
if (( counter >= "$numLines" )) || [[ ! -f "${sourceFile}" ]]; then
echo "invaild file" >82; exit 1
fi
while [ "$counter" -le "$numLines" ]; do
sed -i "${lineNumber} s/.*/wireless-key s: $(sed -n ${counter}p <<< " ${sourceFile}")/" "${targetFile}"
counter=$((counter + 1))
done
Thank you
perhaps a different approach...
awk '{print NR,"wireless_key",$0}' srcfile > dstfile

Generate case range in bash

I want to input two and three digit numbers into a function and output a case range, i.e., input 33 and 66 and output 3[3-9] | [4-5][0-9] | 6[0-6] or input 33 and 666 and output 3[3-9] | [4-9][0-9] | [1-5][0-9][0-9] | 6[0-5][0-9] | 66[0-6]
What are some ideas as to how? Thanks!
Just want to share the final code: https://gist.github.com/zaydek/1627329f88c444f6d71d5cbc0cda616f :)
function enc {
a=${1: -3:1}; b=${1: -2:1}; c=${1: -1}
d=${2: -3:1}; e=${2: -2:1}; f=${2: -1}
if (( $[a] < $[d] )); then
if (( $b$c == 00 && $e$f == 99 )); then echo [$a-$d][0-9][0-9]
elif (( $b$c == 00 )); then if (( $[a+1] < $d )); then echo [$a-$[d-1]][0-9][0-9] "|"; echo $(enc $[d]00 $d$e$f)
else echo $a[0-9][0-9] "|"; echo $(enc $[a+1]00 $d$e$f); fi
else echo $(enc $a$b$c $[a]99) "|"; echo $(enc $[a+1]00 $d$e$f); fi
elif (( $b < $e )); then
if (( $c == 0 && $f == 9 )); then echo $a[$b-$e][0-9]
elif (( $c == 0 )); then if (( $[b+1] < $e )); then echo $a[$b-$[e-1]][0-9] "|"; echo $(enc $a$[e]0 $d$e$f)
else echo $a$b[0-9] "|"; echo $(enc $a$[e]0 $d$e$f); fi
else echo $(enc $a$b$c $a$[b]9) "|"; echo $(enc $a$[b+1]0 $d$e$f); fi
else
if (( $c == $f )); then echo $a$b$c
else echo $a$b[$c-$f]; fi; fi
}
function int {
if [[ $1 = *.* ]]; then
integer=${1%.*}
decimal=${1#*.}
if (( ${decimal:0:1} >= 5 )); then integer=$[integer+1]; fi
echo $integer
else echo $1; fi
}
function cse {
minimum=$(int $(echo $1 $2 | awk '{ print ($1/$2)*( 90) }'))
maximum=$(int $(echo $1 $2 | awk '{ print ($1/$2)*(110) }'))
echo $(enc $minimum $maximum)
}
cse $1 $2
cse takes two inputs, converts them to decimals, and multiples each from a range, i.e., 90 and 110 for -10% and +10% error range, then passes off the minimum and maximum outputs as inputs to enc which is a recursive function that generates case range syntax (with syntax optimization).
Please note, bash does not appear to handle expressions for case ranges as expected or intended. Though, this code can be used to generate case ranges for all 2 to 3 digit numbers and then copied and pasted as inline code.
Here is the max 2 digits case for 0-99, I'd not used character's operations indeed I'd tried math arithmetic.
The following are some output:
./range.sh 1 39 -> 0[1-9] | [1-2][0-9] | 3[0-9]
./range.sh 21 49 -> 2[1-9] | [3-4][0-9]
./range.sh 33 66 -> 3[3-9] | [4-5][0-9] | 6[0-6]
./range.sh 33 99 -> 3[3-9] | [4-8][0-9] | 9[0-9]
Here is my code: range.sh, hope it can provide some help for you:
#!/bin/bash
LHS=$1
RHS=$2
B2=10
D2=$(echo "($RHS-$LHS)/$B2" | bc)
if [[ ${D2} -gt 0 ]]; then
S2=(`seq $(echo "$LHS/$B2" | bc) $(echo "$LHS/$B2+$D2" | bc)`)
#echo ${S2[#]}
C=${#S2[#]}
#echo $C
L1=$(echo "$LHS%$B2" | bc)
if [[ ${L1} -lt 9 ]]; then
L1R="[${L1}-9]"
else
L1R="9"
fi
R1=$(echo "$RHS%$B2" | bc)
if [[ ${R1} -gt 0 ]]; then
R1R="[0-${R1}]"
else
R1R="[0-9]"
fi
if [[ ${C} -gt 3 ]]; then
echo -n $(echo "$LHS/$B2" | bc)$L1R
echo -n " | "[${S2[1]}-${S2[$C-2]}][0-9]
echo -n " | "${S2[$C-1]}$R1R
elif [[ ${C} -eq 3 ]]; then
if [[ ${S2[0]} -eq 0 ]]; then
echo -n 0$L1R
else
echo -n $(echo "$LHS/$B2" | bc)$L1R
fi
echo -n " | "[${S2[1]}-${S2[$C-1]}][0-9]
fi
else
echo [$LHS-$RHS]
fi
echo
Here is the logic for 2 digit numbers:
#!/bin/bash
l=$1
h=$2
function get_nearest_upper_10th_mutlitple()
{
v=$(($1+10))
echo "${v%?}0"
}
function get_nearest_lower_10th_mutlitple()
{
v=$1
echo "${v%?}0"
}
temp="$(printf '%d' $l | wc -m)$(printf '%d' $h | wc -m)"
case $temp in
11) echo "[$l-$h]" ;;
22|12)
a=$([[ $(($l%10)) -eq 0 ]] && echo $l || echo $(($(get_nearest_upper_10th_mutlitple $l)-1)))
if [[ $(printf '%d' $a | wc -m) -eq 1 ]]; then
echo [$l-$a]
else
[[ $a -gt $l ]] && echo "${l%?}[${l#?}-${a#?}]" || echo $a
fi
b=$([[ $(($h%10)) -eq 0 ]] && echo $h || get_nearest_lower_10th_mutlitple $h)
x=$((a+1))
y=$((b-1))
echo "[$(echo ${x%?}~${y%?} | sed 's/~/\n/g' | sort -n | uniq | paste -s -d '-')][${x#?}-${y#?}]"
[[ $b -lt $h ]] && echo "${h%?}[${b#?}-${h#?}]" || echo $b
esac
Here are sample outputs:
$ ./script.bash 2 8
[2-8]
$ ./script.bash 2 6
[2-6]
$ ./script.bash 2 68
[2-9]
[1-5][0-9]
6[0-8]
$ ./script.bash 23 68
2[3-9]
[3-5][0-9]
6[0-8]
$ ./script.bash 23 99
2[3-9]
[3-8][0-9]
9[0-9]
$ ./script.bash 23 80
2[3-9]
[3-7][0-9]
80
$ ./script.bash 40 80
40
[4-7][1-9]
80
$ ./script.bash 40 80 | paste -s -d '|'
40|[4-7][1-9]|80
$ ./script.bash 35 55
3[5-9]
[4][0-9]
5[0-5]
You can use paste -s -d '|' to combine the individual ranges into a single ORed range.
You can use the existing functions which I've made to extend it for 3 digit numbers (I'll post the 3 digit version too after some time with different cases for 13 and 12).

Bash FIFOs not working as expected - how to debug it

I was given the following assignment. We have a prebuilt GUI in a binary form, kept in $GAME_BIN. I have to write a script which connects the GUI with the AI engine. This is my code, which is pretty self descriptive. We have in this case: GAME_BIN=./sredniowiecze_gui, ai1=./idle_ai.sh, ai2=./idle_ai.sh
#!/bin/bash
# arguments parsing and setting $args - not posting this here
gui_outpipe=$(mktemp -u)
gui_inpipe=$(mktemp -u)
ai1_outpipe=$(mktemp -u)
ai1_inpipe=$(mktemp -u)
ai2_outpipe=$(mktemp -u)
ai2_inpipe=$(mktemp -u)
mkfifo $gui_outpipe $gui_inpipe $ai1_outpipe $ai1_inpipe $ai2_inpipe $ai2_outpipe
printinit 1 > $ai1_inpipe &
printinit 2 > $ai2_inpipe &
$GAME_BIN $args < $gui_inpipe &
$ai1 < $ai1_inpipe > $ai1_outpipe &
$ai1 < $ai2_inpipe > $ai2_outpipe &
while true; do
echo "Started the loop"
while true; do
read line < $ai1_outpipe || echo "Nothing read"
echo $line
if [[ $line ]]; then
echo "$line" > $gui_inpipe
echo "$line" > $ai2_inpipe
if [[ "$line" == "END_TURN" ]]; then
break
fi
fi
done
sleep $turndelay
while true; do
read line < $ai2_outpipe || echo "nothing read"
echo $line
if [[ $line ]]; then
echo "$line" > $gui_inpipe
echo "$line" > $ai2_inpipe
if [[ "$line" == "END_TURN" ]]; then
break
fi
fi
done
sleep $turndelay
done
wait
I created a simple idle AI contained in idle_ai.sh
#!/bin/sh
while true; do
echo END_TURN
done
Then the END_TURN message from the GUI is not received at all. On the other hand, the second END_TURN in line (*) is not received by the script. If I use my own C-written AI - very long code, not posting it here, no information from the AI is received in the second run of the while loop
I have absolutely no idea how to debug it.
Since I'm not eager to run binaries unsandboxed, I'm calling the script by firejail ./game.sh [irrelevant parameters]
EDIT after adding set -x the output is
INIT 10 3 1 1 1 3 9
+ [[ -n ./idle_ai.sh ]]
+ [[ -n '' ]]
+ [[ -n ./idle_ai.sh ]]
+ printinit 1
+ ./sredniowiecze_gui -human2
+ true
+ echo 'Started the loop'
Started the loop
+ true
+ read line
+ ./idle_ai.sh
+ echo 'INIT 10 3 1 1 1 3 9'
+ echo END_TURN
END_TURN
+ [[ -n END_TURN ]]
+ echo END_TURN
+ [[ END_TURN == \E\N\D\_\T\U\R\N ]]
+ break
+ true
+ read line
+ echo MOVE 5 9 5 8
MOVE 5 9 5 8
+ [[ -n MOVE 5 9 5 8 ]]
+ echo 'MOVE 5 9 5 8'
In AI vs AI mode:
INIT 10 3 1 1 1 3 9
+ [[ -n ./idle_ai.sh ]]
+ [[ -n ./idle_ai.sh ]]
+ printinit 1
+ printinit 2
+ ./sredniowiecze_gui
+ ./idle_ai.sh
+ true
+ echo 'Started the loop'
Started the loop
+ echo 'INIT 10 3 1 1 1 3 9'
+ true
+ read line
+ ./idle_ai.sh
+ echo 'INIT 10 3 2 1 1 3 9'
+ echo END_TURN
END_TURN
+ [[ -n END_TURN ]]
+ echo END_TURN
+ echo END_TURN
+ [[ END_TURN == \E\N\D\_\T\U\R\N ]]
+ break
+ sleep 1
+ true
+ read line
+ echo END_TURN
END_TURN
+ [[ -n END_TURN ]]
+ echo END_TURN
+ echo END_TURN
+ [[ END_TURN == \E\N\D\_\T\U\R\N ]]
+ break
+ sleep 1
+ true
+ echo 'Started the loop'
Started the loop
+ true
+ read line
EDIT2
I did the suggested changes, now my code is:
printinit 1 > $ai1_inpipe &
printinit 2 > $ai2_inpipe &
$GAME_BIN $args < $gui_inpipe &
$ai1 < $ai1_inpipe > $ai1_outpipe &
echo $!
$ai2 < $ai2_inpipe > $ai2_outpipe &
echo $!
while true; do
echo "Started the loop"
while true; do
read -u3 line || echo "Nothing read"
echo $line
if [[ $line ]]; then
echo "$line" > $gui_inpipe
echo "$line" > $ai2_inpipe
if [[ "$line" == "END_TURN" ]]; then
break
fi
fi
done
sleep $turndelay
while true; do
read -u4 line || echo "nothing read"
echo $line
if [[ $line ]]; then
echo "$line" > $gui_inpipe
echo "$line" > $ai1_inpipe
if [[ "$line" == "END_TURN" ]]; then
break
fi
fi
done
sleep $turndelay
done 3<$ai1_outpipe 4<$ai2_outpipe
And now the script gets stuck on the echo "$line" > $ai1_inpipe line, although the $ai2 process is still running.
EDIT3. Now the log with set -x is
INIT 10 3 1 1 1 3 9
+ [[ -n ./idle_ai.sh ]]
+ [[ -n ./idle_ai.sh ]]
+ printinit 1
+ printinit 2
+ ./sredniowiecze_gui
+ echo 26
26
+ ./idle_ai.sh
+ echo 'INIT 10 3 1 1 1 3 9'
+ echo 27
27
+ ./idle_ai.sh
+ echo 'INIT 10 3 2 1 1 3 9'
+ true
+ echo 'Started the loop'
Started the loop
+ true
+ read -u3 line
+ echo END_TURN
END_TURN
+ [[ -n END_TURN ]]
+ echo END_TURN
+ echo END_TURN
+ [[ END_TURN == \E\N\D\_\T\U\R\N ]]
+ break
+ sleep 1
+ true
+ read -u4 line
+ echo END_TURN
END_TURN
+ [[ -n END_TURN ]]
+ echo END_TURN
+ echo END_TURN
If you add an echo FOO before and after the call, like this:
echo FOO
echo "$line" > $ai1_inpipe
echo BAR
then echo FOO is executed and echo BAR not.
You're using read < input, which sucks all the input and only uses the first line.
Instead of doing that, you should have read read from open file descriptors, like this:
EDIT: Same thing with writing to the fifo files with echo
while true; do
echo "Started the loop"
while true; do
read -u3 line || echo "Nothing read"
...
echo "$line" >&5
echo "$line" >&6
...
done
sleep $turndelay
while true; do
read -u4 line || echo "nothing read"
...
echo "$line" >&5
echo "$line" >&7
...
sleep $turndelay
done 3<$ai1_outpipe 4<$ai2_outpipe 5>$gui_inpipe 6>$ai2_inpipe 7>$ai1_inpipe
See these links for more help on the topic:
BashGuide Named pipes
BashFAQ 085

Bash incrementation breaking scripts

I'm having a weird issue incrementing a bash variable that seems
to be breaking after my first attempt at incremntation that I cannot
pin down, here is a sample of what I am doing and the debug output,
anyone see any reason this should NOT work?
I am currently on GNU bash, version 4.2.45(1)-release (i686-pc-linux-gnu)
#!/bin/bash
set -ex
declare -i elem=0
echo $elem # 0
(( elem++ )) # breaks
echo $elem # 1 but never encountered
while IFS=$'\n' read -r line || [[ -n "$line" ]]; do
(( elem++ ))
echo $elem
done <"${1}" # foo\nbar\nbaz
Output
./incr.sh test
+ declare -i elem=0
+ echo 0
0
+ (( elem++ ))
The weirdest part is by changing the initial incrementor to (( elem+=1 ))
the entire program increments correctly, this seems extremely buggy to the eye...
#!/bin/bash
set -ex
declare -i elem=0
echo $elem
(( elem+=1 ))
echo $elem
while IFS=$'\n' read -r line || [[ -n "$line" ]]; do
(( elem++ ))
echo $elem
done <"${1}" # foo\nbar\nbaz
Output
+ declare -i elem=0
+ echo 0
0
+ (( elem+=1 ))
+ echo 1
1
+ IFS='
'
+ read -r line
+ (( elem++ ))
+ echo 2
2
+ IFS='
'
+ read -r line
+ (( elem++ ))
+ echo 3
3
+ IFS='
'
+ read -r line
+ (( elem++ ))
+ echo 4
4
+ IFS='
'
+ read -r line
+ [[ -n '' ]]
set -e makes your script exit when any command returns failure.
(( 0 )), and equivalently elem=0; (( elem++ )) returns failure.
Therefore, the script exits.
If you set -e and want to run commands whose status you don't care, about, you can use
(( elem++ )) || true

Bash script to list all IPs in prefix

I'm trying to create script that I can input a set of prefixes, which will then list all IP addresses within the prefixes (including network/host/broadcast).
An example would be:
./convert-prefix-to-IPs.sh 192.168.0.0/23 203.20.0.0/16
192.168.0.0
192.168.0.1
...
192.168.0.255
192.168.1.0
..
192.168.1.255
203.20.0.0
..
203.20.255.255
There are some python/perl scripts which can do this, but I'm hoping to have a simple bash script, as it may be used on systems without perl/python (yes.. i know.. )
Here is what I use to generate all the IP addresses in a given CIDR block
nmap -sL -n 10.10.64.0/27 | awk '/Nmap scan report/{print $NF}'
From the nmap man page, the flags are:
-sL: List Scan - simply list targets to scan
-n: Never do DNS resolution
Just that simple
The above command outputs this
10.10.64.0
10.10.64.1
10.10.64.2
10.10.64.3
10.10.64.4
10.10.64.5
10.10.64.6
10.10.64.7
10.10.64.8
10.10.64.9
10.10.64.10
10.10.64.11
10.10.64.12
10.10.64.13
10.10.64.14
10.10.64.15
10.10.64.16
10.10.64.17
10.10.64.18
10.10.64.19
10.10.64.20
10.10.64.21
10.10.64.22
10.10.64.23
10.10.64.24
10.10.64.25
10.10.64.26
10.10.64.27
10.10.64.28
10.10.64.29
10.10.64.30
10.10.64.31
I too was looking for this solution and found that #scherand script worked great. I also have added to this script to give you more option. Help File below.
THIS SCRIPT WILL EXPAND A CIDR ADDRESS.
SYNOPSIS
./cidr-to-ip.sh [OPTION(only one)] [STRING/FILENAME]
DESCRIPTION
-h Displays this help screen
-f Forces a check for network boundary when given a STRING(s)
-i Will read from an Input file (file should contain one CIDR per line) (no network boundary check)
-b Will do the same as –i but with network boundary check
EXAMPLES
./cidr-to-ip.sh 192.168.0.1/24
./cidr-to-ip.sh 192.168.0.1/24 10.10.0.0/28
./cidr-to-ip.sh -f 192.168.0.0/16
./cidr-to-ip.sh -i inputfile.txt
./cidr-to-ip.sh -b inputfile.txt
#!/bin/bash
############################
## Methods
############################
prefix_to_bit_netmask() {
prefix=$1;
shift=$(( 32 - prefix ));
bitmask=""
for (( i=0; i < 32; i++ )); do
num=0
if [ $i -lt $prefix ]; then
num=1
fi
space=
if [ $(( i % 8 )) -eq 0 ]; then
space=" ";
fi
bitmask="${bitmask}${space}${num}"
done
echo $bitmask
}
bit_netmask_to_wildcard_netmask() {
bitmask=$1;
wildcard_mask=
for octet in $bitmask; do
wildcard_mask="${wildcard_mask} $(( 255 - 2#$octet ))"
done
echo $wildcard_mask;
}
check_net_boundary() {
net=$1;
wildcard_mask=$2;
is_correct=1;
for (( i = 1; i <= 4; i++ )); do
net_octet=$(echo $net | cut -d '.' -f $i)
mask_octet=$(echo $wildcard_mask | cut -d ' ' -f $i)
if [ $mask_octet -gt 0 ]; then
if [ $(( $net_octet&$mask_octet )) -ne 0 ]; then
is_correct=0;
fi
fi
done
echo $is_correct;
}
#######################
## MAIN
#######################
OPTIND=1;
getopts "fibh" force;
shift $((OPTIND-1))
if [ $force = 'h' ]; then
echo ""
echo -e "THIS SCRIPT WILL EXPAND A CIDR ADDRESS.\n\nSYNOPSIS\n ./cidr-to-ip.sh [OPTION(only one)] [STRING/FILENAME]\nDESCRIPTION\n -h Displays this help screen\n -f Forces a check for network boundary when given a STRING(s)\n -i Will read from an Input file (no network boundary check)\n -b Will do the same as –i but with network boundary check\n\nEXAMPLES\n ./cidr-to-ip.sh 192.168.0.1/24\n ./cidr-to-ip.sh 192.168.0.1/24 10.10.0.0/28\n ./cidr-to-ip.sh -f 192.168.0.0/16\n ./cidr-to-ip.sh -i inputfile.txt\n ./cidr-to-ip.sh -b inputfile.txt\n"
exit
fi
if [ $force = 'i' ] || [ $force = 'b' ]; then
old_IPS=$IPS
IPS=$'\n'
lines=($(cat $1)) # array
IPS=$old_IPS
else
lines=$#
fi
for ip in ${lines[#]}; do
net=$(echo $ip | cut -d '/' -f 1);
prefix=$(echo $ip | cut -d '/' -f 2);
do_processing=1;
bit_netmask=$(prefix_to_bit_netmask $prefix);
wildcard_mask=$(bit_netmask_to_wildcard_netmask "$bit_netmask");
is_net_boundary=$(check_net_boundary $net "$wildcard_mask");
if [ $force = 'f' ] && [ $is_net_boundary -ne 1 ] || [ $force = 'b' ] && [ $is_net_boundary -ne 1 ] ; then
read -p "Not a network boundary! Continue anyway (y/N)? " -n 1 -r
echo ## move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]; then
do_processing=1;
else
do_processing=0;
fi
fi
if [ $do_processing -eq 1 ]; then
str=
for (( i = 1; i <= 4; i++ )); do
range=$(echo $net | cut -d '.' -f $i)
mask_octet=$(echo $wildcard_mask | cut -d ' ' -f $i)
if [ $mask_octet -gt 0 ]; then
range="{$range..$(( $range | $mask_octet ))}";
fi
str="${str} $range"
done
ips=$(echo $str | sed "s, ,\\.,g"); ## replace spaces with periods, a join...
eval echo $ips | tr ' ' '\n'
else
exit
fi
done
nmap is useful, but an overkill.
You can use prips instead. Saves you the hassle of grepping out the extra output from nmap and using awk.
Calling prips 192.168.0.0/23 will print what you need.
I use the following to skip the network address and broadcast: prips "$subnet" | sed -e '1d; $d'
Prips also has other useful options, e.g. being able to sample every n-th IP.
It's available via apt,brew,rpm and as tar.gz.
This short script will print all the IP addresses in a CIDR range in a few lines of Bash. (I named it prips after the Ubuntu command of the same name. Obviously, if that command is available, use that.)
prips() {
local cidr="$1" ; local lo hi a b c d e f g h
# range is bounded by network (-n) & broadcast (-b) addresses.
lo="$(ipcalc -n "$cidr" | cut -f2 -d=)"
hi="$(ipcalc -b "$cidr" | cut -f2 -d=)"
IFS=. read -r a b c d <<< "$lo"
IFS=. read -r e f g h <<< "$hi"
eval "echo {$a..$e}.{$b..$f}.{$c..$g}.{$d..$h}"
}
Note that I assume the RedHat Linux (Erik Troan, Preston Brown) version of ipcalc, not the Krischan Jodies version that is installed on some platforms (e.g. Mac OS X).
Examples:
$ prips 10.0.0.128/27
10.0.0.128 10.0.0.129 10.0.0.130 10.0.0.131 10.0.0.132 10.0.0.133 10.0.0.134 10.0.0.135 10.0.0.136 10.0.0.137 10.0.0.138 10.0.0.139 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.143 10.0.0.144 10.0.0.145 10.0.0.146 10.0.0.147 10.0.0.148 10.0.0.149 10.0.0.150 10.0.0.151 10.0.0.152 10.0.0.153 10.0.0.154 10.0.0.155 10.0.0.156 10.0.0.157 10.0.0.158 10.0.0.159
Calculates correct number of addresses in a /23 networks:
$ prips 10.0.0.0/23 | wc -w
512
Inspecting a few of those addresses using cut:
$ prips 10.0.0.0/23 | cut -f1-10,256-266 -d' '
10.0.0.0 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7 10.0.0.8 10.0.0.9 10.0.0.255 10.0.1.0 10.0.1.1 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6 10.0.1.7 10.0.1.8 10.0.1.9
And maybe too slow but also correctly generates the 16 million addresses in a /8 network:
$ date ; prips 10.0.0.0/8 | wc -w ; date
Sat May 20 18:06:00 AEST 2017
16777216
Sat May 20 18:06:41 AEST 2017
I recently wrote a function to generate all IP addresses from a given network address. The function takes the network address as argument and accepts CIDR and subnet masks. The script then stores all IPs in the array variable $ips.
Code
network_address_to_ips() {
# create array containing network address and subnet
local network=(${1//\// })
# split network address by dot
local iparr=(${network[0]//./ })
# if no mask given it's the same as /32
local mask=32
[[ $((${#network[#]})) -gt 1 ]] && mask=${network[1]}
# convert dot-notation subnet mask or convert CIDR to an array like (255 255 255 0)
local maskarr
if [[ ${mask} =~ '.' ]]; then # already mask format like 255.255.255.0
maskarr=(${mask//./ })
else # assume CIDR like /24, convert to mask
if [[ $((mask)) -lt 8 ]]; then
maskarr=($((256-2**(8-mask))) 0 0 0)
elif [[ $((mask)) -lt 16 ]]; then
maskarr=(255 $((256-2**(16-mask))) 0 0)
elif [[ $((mask)) -lt 24 ]]; then
maskarr=(255 255 $((256-2**(24-mask))) 0)
elif [[ $((mask)) -lt 32 ]]; then
maskarr=(255 255 255 $((256-2**(32-mask))))
elif [[ ${mask} == 32 ]]; then
maskarr=(255 255 255 255)
fi
fi
# correct wrong subnet masks (e.g. 240.192.255.0 to 255.255.255.0)
[[ ${maskarr[2]} == 255 ]] && maskarr[1]=255
[[ ${maskarr[1]} == 255 ]] && maskarr[0]=255
# generate list of ip addresses
local bytes=(0 0 0 0)
for i in $(seq 0 $((255-maskarr[0]))); do
bytes[0]="$(( i+(iparr[0] & maskarr[0]) ))"
for j in $(seq 0 $((255-maskarr[1]))); do
bytes[1]="$(( j+(iparr[1] & maskarr[1]) ))"
for k in $(seq 0 $((255-maskarr[2]))); do
bytes[2]="$(( k+(iparr[2] & maskarr[2]) ))"
for l in $(seq 1 $((255-maskarr[3]))); do
bytes[3]="$(( l+(iparr[3] & maskarr[3]) ))"
printf "%d.%d.%d.%d\n" "${bytes[#]}"
done
done
done
done
}
Example
network_address_to_ips 10.0.1.0/255.255.255.240
network_address_to_ips 10.1.0.0/24
This script should do. It's (almost) pure Bash. The seq part can be replaced if a completely pure bash is required.
Since Bash apparently uses signed two-complement 4-byte integers, the script is limited to /8 mask maximum. I found ranges larger than /16 impractical anyway so this doesn't bother me at all. If someone knows a simple way to overcome this, please share :)
#!/usr/bin/env bash
BASE_IP=${1%/*}
IP_CIDR=${1#*/}
if [ ${IP_CIDR} -lt 8 ]; then
echo "Max range is /8."
exit
fi
IP_MASK=$((0xFFFFFFFF << (32 - ${IP_CIDR})))
IFS=. read a b c d <<<${BASE_IP}
ip=$((($b << 16) + ($c << 8) + $d))
ipstart=$((${ip} & ${IP_MASK}))
ipend=$(((${ipstart} | ~${IP_MASK}) & 0x7FFFFFFF))
seq ${ipstart} ${ipend} | while read i; do
echo $a.$((($i & 0xFF0000) >> 16)).$((($i & 0xFF00) >> 8)).$(($i & 0x00FF))
done
Usage:
./script.sh 192.168.13.55/22
Tested with Bash version 4.4.23. YMMV.
I think this little script I hacked together does the trick. If not, it's definitely a starting point! Good luck.
#!/bin/bash
############################
## Methods
############################
prefix_to_bit_netmask() {
prefix=$1;
shift=$(( 32 - prefix ));
bitmask=""
for (( i=0; i < 32; i++ )); do
num=0
if [ $i -lt $prefix ]; then
num=1
fi
space=
if [ $(( i % 8 )) -eq 0 ]; then
space=" ";
fi
bitmask="${bitmask}${space}${num}"
done
echo $bitmask
}
bit_netmask_to_wildcard_netmask() {
bitmask=$1;
wildcard_mask=
for octet in $bitmask; do
wildcard_mask="${wildcard_mask} $(( 255 - 2#$octet ))"
done
echo $wildcard_mask;
}
#######################
## MAIN
#######################
for ip in $#; do
net=$(echo $ip | cut -d '/' -f 1);
prefix=$(echo $ip | cut -d '/' -f 2);
bit_netmask=$(prefix_to_bit_netmask $prefix);
wildcard_mask=$(bit_netmask_to_wildcard_netmask "$bit_netmask");
str=
for (( i = 1; i <= 4; i++ )); do
range=$(echo $net | cut -d '.' -f $i)
mask_octet=$(echo $wildcard_mask | cut -d ' ' -f $i)
if [ $mask_octet -gt 0 ]; then
range="{0..$mask_octet}";
fi
str="${str} $range"
done
ips=$(echo $str | sed "s, ,\\.,g"); ## replace spaces with periods, a join...
eval echo $ips | tr ' ' '\012'
done
I have extended #rberg script a little.
check if the "network" you provide really is a network (use -f to skip the check)
handle netmasks greater than /24
Maybe this is of use for someone.
#!/bin/bash
############################
## Methods
############################
prefix_to_bit_netmask() {
prefix=$1;
shift=$(( 32 - prefix ));
bitmask=""
for (( i=0; i < 32; i++ )); do
num=0
if [ $i -lt $prefix ]; then
num=1
fi
space=
if [ $(( i % 8 )) -eq 0 ]; then
space=" ";
fi
bitmask="${bitmask}${space}${num}"
done
echo $bitmask
}
bit_netmask_to_wildcard_netmask() {
bitmask=$1;
wildcard_mask=
for octet in $bitmask; do
wildcard_mask="${wildcard_mask} $(( 255 - 2#$octet ))"
done
echo $wildcard_mask;
}
check_net_boundary() {
net=$1;
wildcard_mask=$2;
is_correct=1;
for (( i = 1; i <= 4; i++ )); do
net_octet=$(echo $net | cut -d '.' -f $i)
mask_octet=$(echo $wildcard_mask | cut -d ' ' -f $i)
if [ $mask_octet -gt 0 ]; then
if [ $(( $net_octet&$mask_octet )) -ne 0 ]; then
is_correct=0;
fi
fi
done
echo $is_correct;
}
#######################
## MAIN
#######################
OPTIND=1;
getopts "f" force;
shift $(( OPTIND-1 ));
for ip in $#; do
net=$(echo $ip | cut -d '/' -f 1);
prefix=$(echo $ip | cut -d '/' -f 2);
do_processing=1;
bit_netmask=$(prefix_to_bit_netmask $prefix);
wildcard_mask=$(bit_netmask_to_wildcard_netmask "$bit_netmask");
is_net_boundary=$(check_net_boundary $net "$wildcard_mask");
if [ $force != 'f' ] && [ $is_net_boundary -ne 1 ]; then
read -p "Not a network boundary! Continue anyway (y/N)? " -n 1 -r
echo ## move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]; then
do_processing=1;
else
do_processing=0;
fi
fi
if [ $do_processing -eq 1 ]; then
str=
for (( i = 1; i <= 4; i++ )); do
range=$(echo $net | cut -d '.' -f $i)
mask_octet=$(echo $wildcard_mask | cut -d ' ' -f $i)
if [ $mask_octet -gt 0 ]; then
range="{$range..$(( $range | $mask_octet ))}";
fi
str="${str} $range"
done
ips=$(echo $str | sed "s, ,\\.,g"); ## replace spaces with periods, a join...
eval echo $ips | tr ' ' '\012'
fi
done
Wanted to comment on an answer above but don't have the rep yet.
Using the top solution with NMAP I added this to my .bashrc
expand-ip() {
nmap -sL -n -iL "$1" | awk '/Nmap scan report/{print $NF}'
}
Now I can use this with just expand-ip targs.
You can use this script (you need to have "bc" installed on your system):
for ip in $# ;do
net=$(echo $ip | cut -d '/' -f 1);
prefix=$(echo $ip | cut -d '/' -f 2);
o1=$(echo $net | cut -d '.' -f4);
o2=$(echo $net | cut -d '.' -f3);
o3=$(echo $net | cut -d '.' -f2);
o4=$(echo $net | cut -d '.' -f1);
len=$(echo "2^(32 - $prefix)"|bc);
for i in `seq $len`;do
echo "$o4.$o3.$o2.$o1";
o1=$(echo "$o1+1"|bc);
if [ $o1 -eq 256 ]; then
o1=0;
o2=$(echo "$o2+1"|bc);
if [ $o2 -eq 256 ]; then
o2=0;
o3=$(echo "$o3+1"|bc);
if [ $o3 -eq 256 ]; then
o3=0;
o4=$(echo "$o4+1"|bc);
fi
fi
fi
done
done
fping -Aaqgr 1 10.1.1.0/24
Simplicity works best

Resources