I am running a loop like so:
for i in $(cat ips_uniq.txt)
do
whois $i | grep 'netname|country|org-name' | sed ':a;N;$!ba;s/\\n//g'
done
Output:
netname: NETPLUScountry: INcountry: INcountry: IN
netname: NETPLUScountry: INcountry: INcountry: IN
This is good however my ips_uniq.txt contains over 300 uniq IP addresses so Ideally I want the IP address to be on the same line of each output.
My version:
#!/bin/bash
while IFS= read -r i
do
if [[ "$i" != "" ]]
then
results=$(whois "$i" | grep -iE 'netname|country|org-name' | tr '\n' ' ')
echo "$i $results"
fi
done < ips_uniq.txt
using the while read method is a safe way to read files line per line, avoiding all problems with spaces or weird line formats. Read https://mywiki.wooledge.org/BashFAQ/001 for details. Not required in your case, but a good construct to learn about.
the if is to avoid empty lines.
then store the result of the whois | grep combination in a variable.
note that I replaced your sed with a simple tr which removes the \n, and adds a space to split fields. Modify as required.
then the final echo adds the IP address prefix to the results line.
I tested with ips_uniq.txt equal to
8.8.8.8
74.6.231.20
and the result is
8.8.8.8 NetName: LVLT-ORG-8-8 Country: US NetName: LVLT-GOGL-8-8-8 Country: US
74.6.231.20 NetName: INKTOMI-BLK-6 Country: US
Using printf you could format the output to be better (i.e. fixed length columns for example).
Related
Here is a sample bash script:
#!/bin/bash
array[0]="google.com"
array[1]="yahoo.com"
array[2]="bing.com"
pasteCommand="/usr/bin/paste -d'|'"
for val in "${array[#]}"; do
pasteCommand="${pasteCommand} <(echo \$(/usr/bin/dig -t A +short $val)) "
done
output=`$pasteCommand`
echo "$output"
Somehow it shows an error:
/usr/bin/paste: invalid option -- 't'
Try '/usr/bin/paste --help' for more information.
How can I fix it so that it works fine?
//EDIT:
Expected output is to get result from the 3 dig executions in a string delimited with | character. Mainly I am using paste that way because it allows to run the 3 dig commands in parallel and I can separate output using a delimiter so then I can easily parse it and still know the dig output to which domain (e.g google.com for first result) is assigned.
First, you should read BashFAQ/050 to understand why your approach failed. In short, do not put complex commands inside variables.
A simple bash script to give intended output could be something like that:
#!/bin/bash
sites=(google.com yahoo.com bing.com)
iplist=
for site in "${sites[#]}"; do
# Capture command's output into ips variable
ips=$(/usr/bin/dig -t A +short "$site")
# Prepend a '|' character, replace each newline character in ips variable
# with a space character and append the resulting string to the iplist variable
iplist+=\|${ips//$'\n'/' '}
done
iplist=${iplist:1} # Remove the leading '|' character
echo "$iplist"
outputs
172.217.18.14|98.137.246.7 72.30.35.9 98.138.219.231 98.137.246.8 72.30.35.10 98.138.219.232|13.107.21.200 204.79.197.200
It's easier to ask a question when you specify input and desired output in your question, then specify your try and why doesn't it work.
What i want is https://i.postimg.cc/13dsXvg7/required.png
$ array=("google.com" "yahoo.com" "bing.com")
$ printf "%s\n" "${array[#]}" | xargs -n1 sh -c '/usr/bin/dig -t A +short "$1" | paste -sd" "' _ | paste -sd '|'
172.217.16.14|72.30.35.9 98.138.219.231 98.137.246.7 98.137.246.8 72.30.35.10 98.138.219.232|204.79.197.200 13.107.21.200
I might try a recursive function like the following instead.
array=(google.com yahoo.com bing.com)
paster () {
dn=$1
shift
if [ "$#" -eq 0 ]; then
dig -t A +short "$dn"
else
paster "$#" | paste -d "|" <(dig -t A +short "$dn") -
fi
}
output=$(paster "${array[#]}")
echo "$output"
Now finally clear with expected output:
domains_arr=("google.com" "yahoo.com" "bing.com")
out_arr=()
for domain in "${domains_arr[#]}"
do
mapfile -t ips < <(dig -tA +short "$domain")
IFS=' '
# Join the ips array into a string with space as delimiter
# and add it to the out_arr
out_arr+=("${ips[*]}")
done
IFS='|'
# Join the out_arr array into a string with | as delimiter
echo "${out_arr[*]}"
If the array is big (and not just 3 sites) you may benefit from parallelization:
array=("google.com" "yahoo.com" "bing.com")
parallel -k 'echo $(/usr/bin/dig -t A +short {})' ::: "${array[#]}" |
paste -sd '|'
I'm really new to bash. I have a list of domains in a .txt file (URLs.txt). I also want to have a .csv file which consists of 3 columns separated by , (myFile.csv). My code reads each line of URLs.txt (each domain), finds its IP address and then inserts them into myFile.csv (domain in the first column, its IP in the 2nd column.
Name, IP
ex1.com, 10.20.30.40
ex2.com, 20.30.40.30
ex3.com, 10.45.60.20
ex4.com, 10.20.30.40
Here is my code:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
echo "$line,$ipValue" >> myFile.csv
fi
done < URLs.txt
I want to add another column to myFile.csv for keeping open ports of each IP. So output would be like this:
Name, IP, Port
ex1.com, 10.20.30.40, 21/tcp
ex2.com, 20.30.40.30, 20/tcp
ex3.com, 10.45.60.20, 33/tcp
ex4.com, 10.20.30.40, 21/tcp
I want to use Nmap to do this. After I choose an IP address from the 2nd column of myFile.csv and find its open ports using Nmap, I want to write the Nmap result to the corresponding cell of the 3rd column.
Also, if there is another similar IP in the 2nd column I want to write the Nmap result for that line too. I mean I don't want to run Nmap again for the duplicated IP. For example, in my example, there are two "10.20.30.40" in the 2nd column. I want to use Nmap just once and for the 1st "10.20.30.40" (and write the result for the 2nd "10.20.30.40" as well, Nmap should not be run for the duplicated IP).
For this to happen, I changed the first line of my code to this:
echo "Name,IP,Port" > myFile.csv
and also here is the Nmap code to find the open ports:
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
but I don't know what to do next and how to apply these changes to my code.
I updated my code to something like this:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
echo "$line,$ipValue,$port" >> myFile.csv
fi
done < URLs.txt
but this way, Nmap was used for finding the open ports of the duplicated IPs too, but I didn't want this. What should I do?
Here's a modified version of your script that roughly does what you want:
#!/usr/bin/env bash
# cache maps from IP addresses to open ports
declare -A cache
getports() {
local ip=$1
nmap -v -Pn -p 1-100 "$ip" -oG - \
| awk -F '\t' '
/Ports:/ {
n = split($2, a, /,? /)
printf "%s", a[2]
for (i = 3; i <= n; ++i)
printf ":%s", a[i]
}
'
}
{
echo 'Name,IP,Port'
while IFS= read -r url; do
# Read filtered dig output into array
readarray -t ips < <(dig +short "$url" | grep -E '^([0-9]+\.){3}[0-9]+$')
# Build array of open ports
unset ports
for ip in "${ips[#]}"; do
ports+=("${cache["$ip"]:=$(getports "$ip")}")
done
# Output
printf '%s,%s,%s\n' \
"$url" \
"$(IFS='-'; echo "${ips[*]}")" \
"$(IFS='-'; echo "${ports[*]}")"
done < URLs.txt
} > myFile.csv
The readarray line reads the filtered output from dig into an array of IP addresses; if that array has length zero, the rest of the loop is skipped.
Then, for each elements in the ips array, we get the ports. To avoid calling nmap if we've seen the IP address before, we use the ${parameter:=word} parameter expansion: if ${cache["$ip"]} is non-empty, use it, otherwise call the getports function and store the output in the cache associative array.
getports is called for IP addresses we haven't seen before; I've used -oG ("grepable output") to make parsing easier. The awk command filters for lines containing Ports:, which look something like
Host: 52.94.225.242 () Ports: 80/open/tcp//http/// Ignored State: closed (99)
with tab separated fields. We then split the second field on the regular expression /,? / (an optional comma followed by a blank) and store all but the first field of the resulting array, colon separated.
Finally, we print the line of CSV data; if ips or ports contain more than one element, we want to join the elements with -, which is achieved by setting IFS in the command substitution and then printing the arrays with [*].
The initial echo and the loop are grouped within curly braces so output redirection has to happen just once.
I wanted to modify dig command to automatically apply a reverse lookup to any A record I receive in output.
Therefore, I've created the following function:
dt ()
{
remove=$(echo $# | sed 's^https*^^' | sed 's^[,/*:]^^g' );
dig any +trace +nocl +nodnssec $remove | sed "s/\($remove.*A\s*\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\)/dig -x \2 +short | xargs echo \"\1 ->\"/e"
}
With this code I have the following output (only A record part is shown so to avoid the question getting bigger):
domain.com. 1200 A 198.54.115.174 -> server224-3.web-hosting.com.
However, now I also need to make a whois lookup using the IP I receive from dig output, but only in case dig -x \2 +short doesn't give any result (stackoverflow.com can be a good example of a domain with A records that do not have PTR).
I tried something like this to check exit code of regular host command (since dig implies that output is successful even if it's empty) and execute proper command depending on the result:
dig any +trace +nocl +nodnssec "$remove" | sed -e "s/\($remove.*A\s*\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\)/echo -n \"\1 -> \"; host \2 | grep -q ''; if [ ${PIPESTATUS[0]} -eq 0 ]; then dig -x \2 +short; else whois \2 | grep 'network:Network-Name:';fi; /e"
But it seems that sed somehow affects the value of ${PIPESTATUS[0]} array.
I wanted to do these modifications in sed because I needed something that will print lines on the go. If I use variables and modify the output from them, the function will work slower, at least visually.
Maybe awk can be of use here, but I am not sure how I should write the code using this command.
Is there a way around this problem? Can sed be used for this purpose or should I use some other tool? Thanks.
The good old' bash gives you variety of tools grep, awk, xargs, while read and so on. Using sed with e command with inside checking PIPESTATUS and executing xargs.... is just unreadable and very long too read. It's not clear to me what do you want to do, as too much happens on one line.
The dig command results zero status on success - you can check it's output if it has zero length when it "fails".
Consider the following script:
dt () {
remove=$(
<<<$1 \
sed 's#^http[s]?://\([^/]*\).*#\1#'
)
dig any +trace +nocl +nodnssec "$remove" |
tr -s '\t' |
awk -v "remove=${remove}." '{
if ($1 == remove && $3 == "A") {
print
}
}' |
while IFS= read -r line; do
IFS=$'\t' read -r _ _ _ ip _ <<<"$line"
if ! dig_result=$(dig -x "$ip" +short); then
echo "Dig error" >&2
exit 1
fi
if [ -z "$dig_result" ]; then
printf "%s -> no-result-from-dig\n" "$line"
else
printf "%s -> %s\n" "$line" "$dig_result"
fi
done
}
First dig is run. Notice the "$remove" is quoted.
Just a precaution - I squeze the tabulators in the output.
Then I filter - the first column should have "$remove". with a dot and the third column should be an "A"
Then for each line (as there are meny)
I get the ip address (ok, maybe ip=$(<<<"$line" cut -f4) would be cleaner and simpler).
I get the result from the dig -x using the ip address
If the result is empty I print "no-result-from-dig" if it not, I print the result.
Running:
dt stackoverflow.com
dt google.pl
Outputs:
stackoverflow.com. 300 A 151.101.65.69 -> no-result-from-dig
stackoverflow.com. 300 A 151.101.193.69 -> no-result-from-dig
stackoverflow.com. 300 A 151.101.129.69 -> no-result-from-dig
stackoverflow.com. 300 A 151.101.1.69 -> no-result-from-dig
google.pl. 300 A 216.58.215.67 -> waw02s16-in-f3.1e100.net.
The reason your command did not work is that $PIPESTATUS was quoted inside " quotes thus expanded before running the shell. You should escape the $ so that's \$PIPESTATUS or pass it inside nonexpading single quotes " ... "'$PIPESTATUS'" ... ".
From every line of nmap network scan output I want to store the hosts and their IPs in variables (for further use additionaly the "Host is up"-string):
The to be processed output from nmap looks like:
Nmap scan report for samplehostname.mynetwork (192.168.1.45)
Host is up (0.00047s latency).
thats my script so far:
#!/bin/bash
while IFS='' read -r line
do
host=$(grep report|cut -f5 -d' ')
ip=$(grep report|sed 's/^.*(//;s/)$//')
printf "Host:$host - IP:$ip"
done < <(nmap -sP 192.168.1.1/24)
The output makes something I do not understand. It puts the "Host:" at the very beginning, and then it puts "IP:" at the very end, while it completely omits the output of $ip.
The generated output of my script is:
Host:samplehostname1.mynetwork
samplehostname2.mynetwork
samplehostname3.mynetwork
samplehostname4.mynetwork
samplehostname5.mynetwork - IP:
In separate, the extraction of $host and $ip basically works (although there might a better solution for sure). I can either printf $host or $ip alone.
What's wrong with my script? Thanks!
Your two grep commands are reading from standard input, which they inherit from the loop, so they also read from nmap. read gets one line, the first grep consumes the rest, and the second grep exits immediately because standard input is closed. I suspect you meant to grep the contents of $line:
while IFS='' read -r line
do
host=$(grep report <<< "$line" |cut -f5 -d' ')
ip=$(grep report <<< "$line" |sed 's/^.*(//;s/)$//')
printf "Host:$host - IP:$ip"
done < <(nmap -sP 192.168.1.1/24)
However, this is inefficient and unnecessary. You can use bash's built-in regular expression support to extract the fields you want.
regex='Nmap scan report for (.*) \((.*)\)'
while IFS='' read -r line
do
[[ $line =~ $regex ]] || continue
host=${BASH_REMATCH[1]}
ip=${BASH_REMATCH[2]}
printf "Host:%s - IP:%s\n" "$host" "$ip"
done < <(nmap -sP 192.168.1.1/24)
Try this:
#!/bin/bash
while IFS='' read -r line
do
if [[ $(echo $line | grep report) ]];then
host=$(echo $line | cut -f5 -d' ')
ip=$(echo $line | sed 's/^.*(//;s/)$//')
echo "Host:$host - IP:$ip"
fi
done < <(nmap -sP it-50)
Output:
Host:it-50 - IP:10.0.0.10
I added an if clause to skip unwanted lines.
I'm attempting to grep IPs from a number of files, look them up in DNS and compare them to the hostnames already in the same files to ensure both are correct. Then print out anything that is wrong.
I've gathered I need to put the information into arrays and diff them somehow.
Here is my horrible bash code which does not work. I'm pretty sure at least my for loop is wrong:
declare -a ipaddr=(`grep -h address *test.com.cfg | awk '{print $2}'`)
declare -a host_names=(`grep -h address *test.com.cfg | awk '{print $2}'`)
for i in "${ipaddr[#]}"
do
lookedup_host_names=( $(/usr/sbin/host ${ipaddr[#]} | awk '{print $5}' | cut -d. -f1-4 | tr '[:upper:]' '[:lower:]'))
done
if [[ -z diff <(printf "%s\n" "${lookedup_host_names[#]}"| sort ) <(printf "%s\n" "${host_names[#]}"| sort) ]]
then
printf "%s\n" "${lookedup_host_names[#]}"
fi
I don't see a difference between your arrays ipaddr and host_names. Supposed your files contain lines like
address 1.2.3.4 somehost.tld
a script like this may do what you want.
cat *test.com.cfg | grep address | while read line; do
IP=$(awk {'print $2'});
CO=$(awk {'print $3'});
CN=$(host $CO | cut -d ' ' -f 4)
[ "$CN" = "$IP" ] || echo "Error with IP $IP";
done
The two principal problems are that your for loop overwrites the array each time rather than appending, and your diff check is invalid.
To quickly fix the for loop, you could use += instead of =, .e.g lookedup_host_names+=( ... ).
To do the diff, you don't really need a condition. You could just run
diff <(printf "%s\n" "${host_names[#]}"| sort ) <(printf "%s\n" "${lookedup_host_names[#]}"| sort)
and it would show any differences between the two in diff format which most Unix users are familiar with (note that I switched the arguments around, since the first argument is supposed to be original).
If, like in your example, you actually do want to compare them and show the entire final list if there is a difference, you could do
if diff <(printf "%s\n" "${host_names[#]}"| sort ) <(printf "%s\n" "${lookedup_host_names[#]}"| sort) > /dev/null
then
printf "%s\n" "${lookedup_host_names[#]}"
fi