I have a raw file with IP ranges (xx.xx.xx.xx-yy.yy.yy.yy)
I want to create a new list with the range converted into single IP addresses.
(All ranges are in a 1-255 range)
conditions
(1) If the difference between the fourth IP octet on each line is less or equal to the max
variable (say 5) It will loop and report each iteration as a single /32 address.
(2) IP address with more than the max variable will be reported as ip address with /24
The following bash script works fine but it is slow on files of 50,000 lines?
Any help would be appreciated. Its part of a script that does other functions so I need to stay in BASH.
for i in $data; do
A=$(echo $i | sed 's/-.*//'); B=$(echo $i | sed 's/^.*-//')
A1=$(echo $A | cut -d '.' -f 4); B1=$(echo $B | cut -d '.' -f 4)
diff=`expr $B1 - $A1`
if [ "$diff" == "0" ]; then
echo $A >> $outfile
elif [ "$diff" -gt "0" -a "$diff" -le $max ]; then
echo $A >> $outfile
for a in $(jot "$diff"); do
count=`expr $A1 + $a`
echo $A | sed "s/\.[0-9]*$/.$count/" >> $outfile
done
else
echo $A | sed 's/\.[0-9]*$/.0\/24/' >> $outfile
fi
done
The likely reason your script is so slow for 50000 lines is that you having bash call a lot of external programs (sed, cut, jot, expr), several times in each iteration of your inner and outer loops. Forking external processes adds a lot of time overhead, when compounded over multiple iterations.
If you want to do this in bash, and improve performance, you'll need to make use of the equivalent features that are built into bash. I took a stab at this for your script and came up with this. I have tried to keep the functionality the same:
for i in $data; do
A="${i%-*}"; B="${i#*-}"
A1="${A##*.}"; B1="${B##*.}"
diff=$(($B1 - $A1))
if [ "$diff" == "0" ]; then
echo $A >> $outfile
elif [ "$diff" -gt "0" -a "$diff" -le $max ]; then
echo $A >> $outfile
for ((a=1; a<=$diff; a++)); do
count=$(($A1 + $a))
echo "${A%.*}.$count" >> $outfile
done
else
echo "${A%.*}.0/24" >> $outfile
fi
done
In particular I've made a lot of use of parameter expansions and arithmetic expansions. I'd be interested to see what kind of speedup (if any) this has over the original. I think it should be significantly faster.
If you are okay with using python, install (download, extract and run sudo python setup.py install) ipaddr library https://pypi.python.org/pypi/ipaddr, then write something like this
import ipaddr
for ip in (ipaddr.IPv4Network('192.0.2.0/24')):
print ip
Related
I need to accept input from user (i.e. 'read').
This input can be either a single positive number or a range of numbers (in the form X-Y ).
I then need to validate this input and perform an iterative loop through the range of numbers (or just once in the case of only X).
examples:
1) User supplies: "8" or "8-"
Loop runs only a single time supplying the number 8
2) User supplies: "13-22"
Loop runs 11 times (13 through 22) referencing the number 13.
3) User supplies: "22-13"
Probably should behave like #2 above...but I'm open to other clean ways to validate the input.
I have worked on the following so far, which isn't very clean, complete, or even 100% correct for what I was asking - but it shows the effort and idea I'm going for:
echo "line?"; read -r deleteline
case "$deleteline" in
''|*[!0-9\-]*) echo "not a number";;
[1-9]|[1-9][0-9]*);;
esac
deleteline_lb=$(echo $deleteline|awk -F "-" '{print $1}')
deleteline_ub=$(echo $deleteline|awk -F "-" '{print $2}')
if [ ! $deleteline_lb = "" ] && [ ! "$deleteline_ub" = "" ]; then
delete_line_count=1
delete_line_count=$(expr $deleteline_ub - $deleteline_lb)
if [ $delete_line_count -le 0 ]; then
delete_line_count=1
fi
fi
i=1; while [ $i -le $delete_line_count ]; do
echo $deleteline_lb $i
i=$(($i + 1))
done
This needs to run in sh, things like seq are not supported - so stick with posix compliant methods...
To clarify I am looking to do the following (pseudo-code):
1) accept input from user
2) validate if input is in the form "#" or "#-#" (range).
3) Execute chosen (arbitrary) code path based on proper/improper input.
4) If single # is given then store that to variable to perform future operations against.
5) If range is given, store both numbers in variable to be able to perform the operation against the lower # up to the higher number. More specifically it would be "(higher #) - (lower #) + 1". So if range were 12-17 then we need to perform operation against 12, 6x. (17 - 12 + 1). IOW, 12-17 inclusive.
6) A way to easily denote if data set is range vs single number is also desired so that code path to each can be easily branched.
thanks for helping!
UPDATE:
Using my basic code I reworked it (with a bit of input from a friend), and basically came up with this:
while true;do
printf "\\delete lines? [e=Exit] ";read -r deleteline
case "$deleteline" in
[Ee]) break;;
*)
echo "$deleteline" | egrep -q '^[[:digit:]-]*$'
if [ $? -ne 0 ]; then
printf "\\n input is not a number.\\n"
else
delete_range_start=`echo $deleteline|awk -F "-" '{print $1}'`
delete_range_end=`echo $deleteline|awk -F "-" '{print $2}'`
if [ $delete_range_end -lt $delete_range_start ]; then
printf "\\n upper range must be higher than lower range.\\n"
else
if [ "$delete_range_end" = "" ]; then
delete_range_end=$delete_range_start
elif [ $delete_range_end -gt $lineNumbers ]; then
printf "\\Setting range to last entry\\n"
fi
break
fi
fi
;;
esac
done
deleteline=$delete_range_start
deleteloop=`expr $delete_range_end - $delete_range_start + 1`
i=1
while [ $i -le $deleteloop ]; do
# Insert all processing code in here
i=`expr $i + 1`
done
If you have a posix compliant awk, try this:
echo "$userInput" | awk -F- '
($1+0==$1&&$2+0==$2){
for(i=$1;($1<$2?i<=$2:i>=$2);)
print ($1<$2?i++:i--);
next
}
$1+0==$1{
print $1;
next
}
$2+0==$2{
print $2;
next
}
($1+0!=$1&&$2+0!=$2){
exit 1
}'
The script check if the 2 fields (separated with -) are numbers. If so, it prints these numbers in an ascending or descending way depending if the first number is greater or lower than the second one.
If only one input, the script just prints it.
If none of the field are number, it exits with a non zero value.
This script could be the validation step of a shell script like this:
$ cat test.sh
#!/bin/sh
echo -n "range: "
read -r range
validated_input=$(echo "$range" | awk -F- '($1+0==$1&&$2+0==$2){for(i=$1;($1<$2?i<=$2:i>=$2);)print ($1<$2?i++:i--);next}$1+0==$1{print $1;next}$2+0==$2{print $2;next}($1+0!=$1&&$2+0!=$2){exit 1}')
if [ $? -ne 0 ]; then
echo "Incorrect range" >&2
fi
for i in $validated_input; do
echo "$i"
done
Examples:
$ ./test.sh
range: 10-6
10
9
8
7
6
$ ./test.sh
range: 8-
8
$ ./test.sh
range: hello
Incorrect range
my script check if the arguments are files or folders
if it is a file, he count the number of lines
after that, if the number of lines is great then 20 or less he do some instructions
the problem is in this instructionn= cat $a | wc -l
My script:
#!/usr/bin/env bash
echo 'Hello this is the test of' `date`
echo 'arguments number is ' $#
if [ $# -eq 4 ]
then
for a in $#
do
if [ -d $a ]
then
ls $a > /tmp/contenu
echo "contenu modified"
elif [ -f $a ]
then
# this instruction must set a numeric value into n
echo "my bad instruction"
n= cat $a | wc -l
echo "number of lines = " $n
# using the numeric value in a test (n must be numeric and takes the number of lines in the current file)
if [ $n -eq 0 ]
then
echo "empty file"
elif [ $n -gt 20 ]
then
echo ` head -n 10 $a `
else
cat $a
fi
else
echo "no file or directory found"
fi
done
else
echo "args number must be 4"
fi
This is the output of the execution of the incorrect instruction
my bad instruction
5
number of lines =
ExamenEx2.sh: line 19: [: -eq : opérateur unaire attendu
The line n= cat $a | wc -l is an offending instruction. Always remember that bash shell scripting is extremely case-sensitive. Your command is interpreted by the shell as having to run two separate commands
n= cat $a | wc -l
#^^ ^^^^^^^^^^^^^^
#1 2
The first part just stores an empty string to the variable n and the next prints the line count of the file stored in variable a. Notice that the shell does not throw errors for this. Because it is not violating the syntax (just the semantics are wrong). But the line count is never assigned to the variable n.
The error is seen when the conditional if [ $n -eq 0 ] is hit when you are doing a comparison with an empty variable on the LHS.
You wanted to run a command and store its output, you need command-substitution($(..)) for that. Assuming the $a contains a name of a file just do
n=$(wc -l < "$a")
Note, that I've removed the useless cat usage and piping it to wc. But wc can read from an input stream directly.
Also note that you have multiple bad practices in your script. Remember to do the following
Always double-quote the shell variables - "$#", "$#", [ -f "$a" ], [ -d "$a" ]
Don't use the `` for command-substitution, because it is not easily nestable and you might have issues related to quoting also.
You can use conditional expression [[ if you are sure if the script is running under bash in which a variable containing spaces can be used without quoting on the LHS
Can anybody tell me what's wrong in this script, it's not working. When I run it, there is no output/error on the screen.
The script is to monitor a log file to check the value of one of the columns, and if it is more than 20 it will echo a message.
#!/bin/bash
while true ; do
COUNT=`tail -f /monitoring/log.20160121|cut -d" " -f39`
echo $COUNT
if [ $COUNT -gt 20 ] ;then
echo "Count is high"
break
fi
sleep 10
done
tail -f does not exit, so your script gets stuck there. I assume you are just interested in the last line of the log; tail -n 1 does that.
Other points:
Indentation: not sure how much got lost while copy pasting, but proper indentation massively increases readability of your code
Variable names: all uppercase variable names are discouraged as they might clash with reserved (environment) variable names
Command substitution with backticks (` `) is discouraged and the form $( ) is preferred; makes for example nesting easier
Since you're using Bash, you can use the (( )) conditional construct, which is better suited for comparing numbers than [ ]
Together:
#!/bin/bash
while true; do
count=$(tail -n 1 /monitoring/log.20160121 | cut -d " " -f 39)
echo $count
if (( count > 20 )); then
echo "Count is high"
break
fi
sleep 10
done
The output comes from a command I run from our netscaler. It outputs the following ... One thing to note is that the middle two numbers change but the even/odd criteria is always on the last digit. We never have more than 2 digits, so we'll never hit 10.
WC-01-WEB1
WC-01-WEB4
WC-01-WEB3
WC-01-WEB5
WC-01-WEB8
I need to populate a file called "even" and "odds." If we're dealing with numbers I can figure it out, but having the number within a string is throwing me off.
Example code but I'm missing the part where I need to match the string.
if [ $even_servers -eq 0 ]
then
echo $line >> evenfile
else
echo $line >> oddfile
fi
This is a simple awk command:
awk '/[02468]$/{print > "evenfile"}; /[13579]$/{print > "oddfile"}' input.txt
There must be better way.
How about this version:
for v in `cat <my_file>`; do export type=`echo $v | awk -F 'WEB' '{print $2%2}'`; if [ $type -eq 0 ]; then echo $v >> evenfile ; else echo $v >> oddfile; fi; done
I assume your list of servers is stored in the filename <my_file>. The basic idea is to tokenize on WEB using awk and process the chars after WEB to determine even-ness. Once this is known, we export the value to a variable type and use this to selectively dump to the appropriate file.
For the case when the name is the output of another command:
export var=`<another command>`; export type=`echo $var | awk -F 'WEB' '{print $2%2}'`; if [ $type -eq 0 ]; then echo $var >> evenfile ; else echo $var >> oddfile; fi;
Replace <another command> with your perl script.
As always grep is your friend:
grep "[2468]$" input_file > evenfile
grep "[^2468]$" input_file > oddfile
I hope this helps.
In bash I need to run a script that loops from i=1 to i=99999999 but it always run out of memory. Is there any workaround? or is there a max value for i?
first=1
last=99999999
randomString="CXCXQOOPSOIS"
for val in $( seq $first $last )
do
padVal=$( printf "%010d\n" $val )
hash=$( echo -n $randomString$padVal | md5sum )
if [[ "$hash" =~ ^000000) ]]; then
echo "Number: $val" >> log_000000
echo "$val added to log - please check."
fi
done
bash provides C-like syntax for loop:
first=1
last=99999999
randomString="CXCXQOOPSOIS"
for ((val=$first; val<$last; val++))
do
padVal=$( printf "%010d\n" $val )
hash=$( echo -n $randomString$padVal | md5sum )
if [[ "$hash" =~ ^000000) ]]; then
echo "Number: $val" >> log_000000
echo "$val added to log - please check."
fi
done
Your seq command generates 100 million numbers (bar a couple) and requires 800 MiB or so of memory to hold just the list of digits (probably an under-estimate; each number might be held in a separate memory allocation, which might mean 8 bytes for a pointer and 16 bytes for the allocated block, which triples the storage space estimate).
You can improve things dramatically by using:
for millions in $(seq 0 99)
do
for smallstuff in $(seq -f "%6.0f" 0 999999)
do
val="$millions$smallstuff"
...
done
done
This dramatically reduces the amount of memory needed; the only issue to watch is that it tests 0 which your original code did not.
If you still want to use seq => therefore separate seq and the loop using a pipe: |
This solution is more portable and can be used on other shells.
The memory print is still reduced, but this script requires to process two threads.
first=1
last=99999999
randomString="CXCXQOOPSOIS"
seq $first $last |
while read val
do
padVal=$( printf "%010d\n" $val )
hash=$( echo -n $randomString$padVal | md5sum )
if [[ "$hash" =~ ^000000) ]]; then
echo "Number: $val" >> log_000000
echo "$val added to log - please check."
fi
done