Logical Approach on bash - bash

#!/bin/bash
echo -n "Enter the domain name > "
read name
dig -t ns "$name" | cut -d ";" -f3 | cut -d ":" -f2| grep ns | cut -f6 > registerfile.txt;
cat registerfile.txt | while read line; do dig axfr "#$line" "$name"; done | cut -d"." -f-4 > nmap.txt
It is done till this section. Below, it could be not matched the line and name parameters. How should be changed?
cat nmap.txt | while read line; do if [ "$line" == "$name" ]; then host "$line"; fi; done > ping.txt
cat ping.txt | cut -d " " -f4 | while read line; do if [[ "$line" =~ ^[0-9]+$ ]]; then nmap -sS "$line";fi ;done

It's not clear where exactly things are going wrong, but here is a refactoring which might hopefully at least nudge you in the right direction.
#!/bin/bash
read -p "Enter the domain name > " name
dig +short -t ns "$name" |
tee registerfile.txt |
while read line; do
dig axfr "#$line" "$name"
done |
cut -d"." -f-4 |
tee nmap.txt |
while read line; do
if [ "$line" = "$name" ]; then
host "$line"
fi
done > ping.txt
cut -d " " -f4 ping.txt |
grep -E '^[0-9]+$' |
xargs -r -n 1 nmap -sS
Your remark in comments that if [ "$line" = "$name" ]; then host "$line"; fi isn't working suggests that the logic there is somehow wrong. It currently checks whether each line is identical to the original domain name, and then looks it up over and over again in those cases, which seems like a curious thing to do; but given only the code and the "does not work", it's hard to say what it's really supposed to accomplish. If you actually want something else, you need to be more specific about what you require. Perhaps you are actually looking for something like
... tee nmap.txt |
# Extract the lines which contain $name at the end
grep "\.$name\$" |
xargs -n 1 dig +short |
tee ping.txt |
grep -E '^[0-9]+$' ...
The use of multiple statically-named files is an antipattern; obviously, if these files serve no external purpose, just take out the tee commands and run the entire pipeline with no in-between output files. If you do need these files, having them overwritten on each run seems problematic -- maybe add a unique date stamp suffix to the file names?

Related

How do I fix my error saying the home directory is invalid?

I'm still learning how to work with bash but my program is part of part 1 which is posted below but they are wanting us to use the file cs.rosters.txt to generate the new_users.txt file.
The cs.rosters.txt looks like this:
CMPSC 1513 03|doejan|Doe, Jane|0510|0350
CMPSC 1513 03|smijoh|Smith, John|0510
CMPSC 1133 01|cp2stu3| CPII, Student3 Neither|2222|0020
Below is what I created for part 1 which runs correctly:
#!/bin/bash
while read -r line || [[ -n "$line" ]]; do
username=$(echo "$line" | cut -d: -f1)
GECOS=$(echo "$line" | cut -d: -f5)
homedir=$(echo "$line" | cut -d: -f6)
echo "adduser -g '$GECOS' -d '$homedir' -s /bin/bash '$username'"
done < "new_users.txt"
Here's where I'm struggling
The code above pretty much displays the information in the new_users.txt file
I'm struggling on the second part, my code is posted below:
What I want to do is create a script that generates the cs.rosters.txt file like I did above.
#!/bin/bash
while read line; do
user=$(echo $line | cut -d'|' -f2)
pass=$(echo $line | cut -d'|' -f3)
home=$(echo $line | cut -d'|' -f4)
useradd -m -d $home -s /bin/bash $user
echo $pass | passwd --stdin $user
done < cs_roster.txt
I'm getting two errors: one is saying the Computer directory is invalid and the function --stdin is invalid. Can you help me?

How can I make cut take less time to process?

I have a text file with a whole bunch of lines (1000 exactly) and they all have 4 bits of text, seperated by a ;.
Here is the for loop I'm using, to go through each line:
while IFS= read -r line; do
let liner++
if [[ liner -eq "1" ]]; then
continue
fi
name=$(echo "${line}" | cut -d';' -f1)
fullname=$(echo "${line}" | cut -d';' -f2)
id=$(echo "${line}" | cut -d';' -f3)
test=$(echo "${line}" | cut -d';' -f4)
echo "${GREEN}$(($liner-1))) ${name} ${ORANGE}v${test} ${RED}(${id})${NC}"
stuff+=("${fullname}")
done < list.txt
It takes about 5 seconds before it finishes running and I believe it's from all those cut (name, fullname, id, test) variables. What would be the best solution to speed this up?
Awk undoubtedly provides a better solution, but if you don't want to learn Awk right now, you could speed your function up a lot by just using read to split the lines into fields:
liner=0
stuff=()
while IFS=\; read -r name fullname id test; do
echo "$GREEN$((++liner))) $name ${ORANGE}v$test $RED($id)$NC"
stuff+=("$fullname")
done < <(tail -n+2 1000num.txt)

Intermittent piping failure in bash

I have a code snippet that looks like this
while grep "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#"; do
grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do
lineno=$(echo $line | cut -d':' -f1)
spaces=$(sed "${lineno}!d" /tmp/kubernetes/$basefile | awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" /tmp/kubernetes/$basefile
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" /tmp/kubernetes/$basefile
lineno=$((lineno+1))
done < "/tmp/secrets.yaml"
done
done
in /tmp/kubernetes/$basefile, the string {{SECRETS}} appears twice 100% of the time.
Almost every single time, this completes fine. However, very infrequently, the script errors on its second loop through the file. like so, according to set -x
...
IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
exit code 1
When it works, the set -x looks like this, and continues processesing the file correctly.
...
+ IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
+ grep '{{SECRETS}}' /tmp/kubernetes/deployment.yaml
+ grep -v '#'
I have no answer for how this can only happen occasionally, so I think there's something about bash piping's parallelism I don't understand. Is there something in grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do that could lead to out-of-order execution somehow? Based on the error, it seems like it's trying to read a line, but can't because previous commands didn't work. But there's no indication of that in the set -x output.
A likely cause of the problem is that the pipeline containing the inner loop both reads and writes the "basefile" at the same time. See How to make reading and writing the same file in the same pipeline always “fail”?.
One way to fix the problem is do a full read of the file before trying to update it. Try:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
while
line=$(grep -n "{{SECRETS}}" "$basepath" | grep -v "#" | head -n1)
[[ -n $line ]]
do
lineno=$(echo "$line" | cut -d':' -f1)
spaces=$(sed "${lineno}!d" "$basepath" \
| awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" "$basepath"
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" "$basepath"
lineno=$((lineno+1))
done < "$secretspath"
done
(I introduced the variables basepath and secretspath to make the code easier to test.)
As an aside, it's also possible to do this with pure Bash code:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
updated_lines=()
is_updated=0
while IFS= read -r line || [[ -n $line ]] ; do
if [[ $line == *'{{SECRETS}}'* && $line != *'#'* ]] ; then
spaces=${line%%[^[:space:]]*}
while IFS= read -r secretline || [[ -n $secretline ]]; do
updated_lines+=( "${spaces}${secretline}" )
done < "$secretspath"
is_updated=1
else
updated_lines+=( "$line" )
fi
done <"$basepath"
(( is_updated )) && printf '%s\n' "${updated_lines[#]}" >"$basepath"
The whole updated file is stored in memory (in the update_lines array) but that shouldn't be a problem because any file that's too big to store in memory will almost certainly be too big to process line-by-line with Bash. Bash is generally extremely slow.
In this code spaces holds the actual space characters for indentation, not the number of them.

grepping text within a multi-column list with a provided string

I'm writing a script to handle multiple uses for searching lists. Long story short, I am querying a DB and have a basic list such as this:
sdc 10:0 KQJWBE11
sdd 10:1 KSDJFBQK
sde 10:2 13KN13DD
sdf 10:3 123DJN1O
sdg 10:4 213JBDKJ
sdh 10:5 N2QQWMNE
sdi 10:6 QKEWJDQJ
sdj 10:7 QKWJEDWE
sdk 20:0 QEDQWEDQ
sdl 20:1 1234E13L
sdm 20:2 KQNE2OUN
sdn 20:3 QN2NK3JN
sdo 20:4 23J23EN2
sdp 20:5 2WBNEKNW
sdq 20:6 QWEDKJNW
sdr 20:7 QWEDQEDD
These exist in the variable "${TABLE_FORMAT}" and are formatted as just as above into a table.
#... other logic above this
# Query via primary and secondary location. Example: DISK_ARG="10:1"
elif [[ ${DISK_ARG} =~ ([[:digit:]]:[[:digit:]])+$ ]]; then
DISK_ARG_PRIMARY=$(echo "${DISK_ARG}" | cut -d: -f1)
DISK_ARG_SECONDARY=$(echo "${DISK_ARG}" | cut -d: -f2)
echo -e "${HEADER}"
echo -e "${TABLE_FORMAT}" | grep -Ei "($DISK_ARG_PRIMARY):($DISK_ARG_SECONDARY)"
fi
# Query secondary location. Example: DISK_ARG="5"
elif [[ ${DISK_ARG} =~ ([[:digit:]])+$ ]]; then
DISK_ARG_F2="$(echo "${F2}" | grep -Ei "([[:digit:]]):(${DISK_ARG})")"
DISK_ARG_PRIMARY=$(echo "${DISK_ARG_F2}" | cut -d: -f1)
DISK_ARG_SECONDARY=$(echo "${DISK_ARG_F2}" | cut -d: -f2)
echo -e "${TABLE_FORMAT}" | grep -Ei "($DISK_ARG_PRIMARY):($DISK_ARG_SECONDARY)"
fi
else :
fi
The offending line that doesn't work is in the second elif:
echo -e "${TABLE_FORMAT}" | grep -Ei "($DISK_ARG_PRIMARY):($DISK_ARG_SECONDARY)"
grep: Unmatched ( or \(
The current variables at this point are:
DISK_ARG_PRIMARY="10 20"
DISK_ARG_SECONDARY="5 5"
I want the following rendered as output:
sdh 10:5 N2QQWMNE
sdp 20:5 2WBNEKNW
I'm not sure if this could be accomplished with building some type of array in grep or modifying the IFS somehow. I want the script to handle many inputs and look for matches off of the relevant fields.
In your case
DISK_ARG_PRIMARY=$(echo "${DISK_ARG}" | cut -d: -f1)
DISK_ARG_SECONDARY=$(echo "${DISK_ARG}" | cut -d: -f2)
could be replaced with
DISK_ARG_PRIMARY=${DISK_ARG%:*} # remove :* suffix
DISK_ARG_SECONDARY=${DISK_ARG#*:} # remove *: prefix
to avoid subshell pipe and cut
also add -w option to grep to avoid to match substrings for example 5 will not match 51.
DISK_ARG_PRIMARY="10 20"
DISK_ARG_SECONDARY="5 5"
To have
grep -Ew '(10|20):(5|5)'
grep -Ew "($(set -- $DISK_ARG_PRIMARY;IFS=\|;echo "$*")):($(set -- $DISK_ARG_SECONDARY;IFS=\|;echo "$*"))"

Bash Script to batch-convert IP Addresses to CIDR?

Ok, here's the problem.
I have a plaintext list of IP addresses that I'm blocking on my servers, growing more and more unwieldy every day (added 3000+ entries today alone).
It's already been sorted for duplicates so that's not a problem. What I'd like to do is write a script to go through it and consolidate the entries a bit better for mass blocking.
For example, take this:
2.132.35.104
2.132.79.240
2.132.99.87
2.132.236.34
2.132.245.30
And turn it into this:
2.132.0.0/16
Any suggestions on how to code that in a bash script?
UPDATE: I've worked out part-way how to do what I'm needing. Converting it to /24 is easy, as follows:
cat /usr/local/blocks/blocks.txt | while read line; do
oc1=`echo "$line" | cut -d '.' -f 1`
oc2=`echo "$line" | cut -d '.' -f 2`
oc3=`echo "$line" | cut -d '.' -f 3`
oc4=`echo "$line" | cut -d '.' -f 4`
echo "$oc1.$oc2.$oc3.0/24" >> twentyfour.srt
done
sort -u twentyfour.srt > twentyfour.txt
rm -f twentyfour.srt
ori=`cat /usr/local/blocks/blocks.txt | wc -l`
new=`cat twentyfour.txt | wc -l`
echo "$ori"
echo "$new"
That reduced it down from 4,452 entries to 4,148 entries.
Instead of having:
109.86.9.93
109.86.26.77
109.86.55.225
109.86.70.224
109.86.87.199
109.86.89.202
109.86.95.248
109.86.100.19
109.86.110.43
109.86.145.216
109.86.152.86
109.86.155.238
109.86.156.54
109.86.187.91
109.86.228.86
109.86.234.51
109.86.239.61
I now have:
109.86.100.0/24
109.86.110.0/24
109.86.145.0/24
109.86.152.0/24
109.86.155.0/24
109.86.156.0/24
109.86.187.0/24
109.86.228.0/24
109.86.234.0/24
109.86.239.0/24
109.86.26.0/24
109.86.55.0/24
109.86.70.0/24
109.86.87.0/24
109.86.89.0/24
109.86.9.0/24
109.86.95.0/24
All well and good. BUT, there's 17 entries from the 109.86.. area. In a case where the first 2 octets match more than say 5 entries on /24, I'd like to reduce that to /16.
That's where I'm stuck.
UPDATE 2:
For Steve: Here's the block list for today. And here's the result so far. Apparently it's not removing the near-duplicate entries from twentyfour that are in sixteen.
I wish I could tell you this is a simple filter. However, all of the 2.0.0.0/8 network is registered to RIPE NCC. There's just way too many different ranges of blocked IP addresses, its easier to just narrow down the scope of visitors you do want versus what you don't want.
You could also use various tools you can use to block attacks automatically.
Map to identify which is which. https://www.iana.org/numbers
Here's a script I just made for you. Then you can create the major block lists for each of the primary registries. Afrinic, Lacnic, Apnic, Ripe, and Arin.
create_tables_by_registry.sh
Just run this script... Then run the following registry.sh files. (E.g; ripe.sh)
#!/bin/bash
# Author: Steve Kline
# Date: 03-04-2014
# Designed and tested to run on properly on CentOS 6.5
#Grab Updated IANA Address Space Assignments only if Newer Version
wget -N https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.txt
assigned=ipv4-address-space.txt
arrayregistry=( afrinic apnic arin lacnic ripe )
for registry in "${arrayregistry[#]}"
do
#Clean up the ipv4-address-space.txt file and keep useable IPs
grep "$registry" $assigned | sed 's/\/8/\.0\.0\.0\/8/g'| colrm 15 > $registry-tmp1.txt
ip=($(cat $registry-tmp1.txt))
echo "#!/bin/bash" > $registry.sh
for ip in "${ip[#]}"
do
echo $ip | sed -e 's/" "//g' > $registry-tmp2.txt
#INSERT OR MODIFY YOUR COMPATIBLE FIREWALL RULES HERE
#This section creates the country to block.
echo "iptables -A INPUT -s $ip -j DROP" >> $registry.sh
chmod +x $registry.sh
done
rm $registry-tmp1.txt -f
rm $registry-tmp2.txt -f
done
Ok! Well I'm back, a little insane here and a little nutty there... I think I helped figure this out for you. I'm sure you can piece together a modification to better fit your needs.
#MODIFY FOR YOUR LIST OF IP ADDRESSES
BADIPS=block.ip
twentyfour=./twentyfour.ips #temp file for all IPs converted to twentyfour net ids
sixteen=./sixteen.ips #temp file for sixteen bit
twentyfourlst1=./twentyfour1.txt #temp file for 24 bit IDs
twentyfourlst2=./twentyfour2.txt #temp file for 24 bit IDs filtered by 16 bit IDs that match
sixteenlst=./sixteen.txt #temp file for parsed sixteenbit
#MODIFY FOR YOUR OUTPUT OF CIDR ADDRESSES
finalfile=./blockips.list #Final file post-merge
cat $BADIPS | while read line; do
oc1=`echo "$line" | cut -d '.' -f 1`
oc2=`echo "$line" | cut -d '.' -f 2`
oc3=`echo "$line" | cut -d '.' -f 3`
oc4=`echo "$line" | cut -d '.' -f 4`
echo "$oc1.$oc2.$oc3.0/24" >> $twentyfour
echo "$oc1.$oc2.0.0/16" >> $sixteen
done
awk '{i=1;while(i <= NF){a[$(i++)]++}}END{for(i in a){if(a[i]>4){print i,a[i]}}}' $sixteen | sed 's/ [0-9]\| [0-9][0-9]\| [0-9][0-9][0-9]//g' > $sixteenlst
sort -u $twentyfour > twentyfour.txt
# THIS FINDS NEAR DUPLICATES MATCHING FIRST TWO OCTETS
cat $sixteenlst | while read line; do
oc1=`echo "$line" | cut -d '.' -f 1`
oc2=`echo "$line" | cut -d '.' -f 2`
oc3=`echo "$line" | cut -d '.' -f 3`
oc4=`echo "$line" | cut -d '.' -f 4`
grep "\b$oc1.$oc2\b" twentyfour.txt >> duplicates.txt
done
#THIS REMOVES THE NEAR DUPLICATES FROM THE TWENTYFOUR FILE
fgrep -vw -f duplicates.txt twentyfour.txt > twentyfourfinal.txt
#THIS MERGES BOTH RESULTS
cat twentyfourfinal.txt $sixteenlst > $finalfile
sort -u $finalfile
ori=`cat $BADIPS | wc -l`
new=`cat $finalfile | wc -l`
echo "$ori"
echo "$new"
#LAST MIN CLEANUP
rm -f $twentyfour $twentyfourlst $sixteen $sixteenlst duplicates.txt twentyfourfinal.txt
Going Back to fix: I noted a problem... Originally unsuccessful.
`grep "$oc1.$oc1" twentyfour.txt > duplicates.txt
For Example: The old script had bad results with this test IP range... the updated version now above... Does exactly as its intended. match the octet exactly.. and not a similar.
192.168.1.1
192.168.2.50
192.168.5.23
192.168.14.10
192.168.10.5
192.168.24.25
192.165.20.10
10.192.168.30
5.76.10.20
5.76.20.30
5.76.250.10
5.76.34.10
5.76.50.30
95.76.30.1 - Old script matched this to 5.76
20.20.5.5
20.20.10.10
20.20.16.50
20.20.205.20
20.20.60.20
205.20.16.20 - not a problem
20.205.150.150 - Old script matched this to 20.20
220.20.16.0 - Also failed without adding -w parameter to the last grep to only match exact strings.

Resources