I have a list (text file) with the following data:
app1 example1.google.com
app2 example2.google.com
dev1 device1.google.com
cell1 iphone1.google.com
I want to check the ip address of the URLs/hostnames and update the text file with the gathered ip. Example:
app1 example1.google.com 192.168.1.10
app2 example2.google.com 192.168.1.55
dev1 device1.google.com 192.168.1.53
cell1 iphone1.google.com 192.168.1.199
You can use dig to get the IP (but the domains must exist). Not tested for IPv6.
#! /bin/bash
while read name url ; do
ip=$(dig -4 $url | grep '^[^;]' | grep -o '\([0-9]*[.:]\)\+[0-9.:]*$')
printf '%s %s %s\n' "$name" "$url" "$ip"
done < data.txt
If there are only two columns in the file, this might help:
awk '{"dig +short " $2 | getline ip ; print $1, $2, ip}' file
First we run a subshell (not really a good idea to run this for zillions of records) and in it a "dig +short" (the shortest possibility which came to my mind to get IP address only) with the FQDN of a machine (found in the second column). Then we print all the original columns and the new one too (with the IP address). Output can be redirected to a new file with a single >. I wouldn't consider "safe" to edit original files.
Related
I have written a bash script that reads a text file containing URLs and finds its IP. Each line of this file contains a URL. I want to create a .csv file as output which has 2 columns, the first column for URL and the 2nd column for its IP. Here is the script:
#!/bin/bash
while IFS= read -r line; do
ip=$(dig +short $line)
echo "${line}, ${ip}" >> ipfile.csv
done < domains
It works fine. The problem is that sometimes when I use dig +short example.com, instead of returning only the IP of the "example.com", it returns something like: example2.com IP. In this case the 2nd column saves example2.com and the corresponding IP moves to the 1st column of the next row.
So my question is: "How can I ignore the first part (example2.com) and only extract and save the "IP" part in the second column"?
I tried to split the text by space and newline character, but unfortunately it didn't work for me.
#!/bin/bash
while IFS= read -r line; do
ip=$(dig +short $line)
if [[ $ip = *\n* ]]
then
bar=${ip##*\n}
echo "{line}, ${bar}" >> ipfile.csv
else
echo "${line}, ${ip}" >> ipfile.csv
fi
done < domain
You may extract anything that looks like an IPv4 address using the following grep expression:
echo 'hello 11.22.33.44 world' | grep -E -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"
Explanation:
-E: Use regex
-o: Print only the matching part
[0-9]+\.: Match a sequence of digits, followed by a period (escaped because a period in regex has special meaning)
This is repeated 4 times, excluding the final period
This solution has some false positives (9999.0000.1.2 passes the pattern) but assuming dig doesn't output something seriously messed up, this will do.
Also it doesn't support IPv6, which might be a problem for you, but it is trivial to modify for IPv6 so it is left as an exercise to the reader :)
Assuming dig +shirt only returns two forms :
IP
name IP
We can use this script :
#!/usr/bin/env bash
while IFS= read -r line; do
read name ip <<<"$(dig +short $line)" # get name and IP
echo "${line}, ${ip:-$name}" >> ipfile.csv # If second column is absent, take the first column
done < domains
I need to perform a whois lookup on a file containing IP addresses and output both the country code and the IP address into a new file. In my command so far I find the IP addresses and get a unique copy that doesn't match allowed ranges. Then I run a whois lookup to find out who the foreign addresses are. Finally it pulls the country code out. This works great, but I can't get it show me the IP alongside the country code since that isn't included in the whois output.
What would be the best way to include the IP address in the output?
awk '{match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); ip = substr($0,RSTART,RLENGTH); print ip}' myInputFile \
| sort \
| uniq \
| grep -v '66.33\|66.128\|75.102\|216.106\|66.6' \
| awk -F: '{ print "whois " $1 }' \
| bash \
| grep 'country:' \
>> myOutputFile
I had thought about using tee, but am having troubles lining up the data in a way that makes sense. The output file should be have both the IP Address and the country code. It doesn't matter if they are a single or double column.
Here is some sample input:
Dec 27 04:03:30 smtpfive sendmail[14851]: tBRA3HAx014842: to=, delay=00:00:12, xdelay=00:00:01, mailer=esmtp, pri=1681345, relay=redcondor.itctel.c
om. [75.102.160.236], dsn=4.3.0, stat=Deferred: 451 Recipient limit exceeded for this se
nder
Dec 27 04:03:30 smtpfive sendmail[14851]: tBRA3HAx014842: to=, delay=00:00:12, xdelay=00:00:01, mailer=esmtp, pri=1681345, relay=redcondor.itctel.c
om. [75.102.160.236], dsn=4.3.0, stat=Deferred: 451 Recipient limit exceeded for this se
nder
Thanks.
In general: Iterate over your inputs as shell variables; this then lets you print them alongside each output from the shell.
The below will work with bash 4.0 or newer (requires associative arrays):
#!/bin/bash
# ^^^^- must not be /bin/sh, since this uses bash-only features
# read things that look vaguely like IP addresses into associative array keys
declare -A addrs=( )
while IFS= read -r ip; do
case $ip in 66.33.*|66.128.*|75.102.*|216.106.*|66.6.*) continue;; esac
addrs[$ip]=1
done < <(grep -E -o '[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+')
# getting country code from whois for each, printing after the ip itself
for ip in "${!addrs[#]}"; do
country_line=$(whois "$ip" | grep -i 'country:')
printf '%s\n' "$ip $country_line"
done
An alternate version which will work with older (3.x) releases of bash, using sort -u to generate unique values rather than doing that internal to the shell:
while read -r ip; do
case $ip in 66.33.*|66.128.*|75.102.*|216.106.*|66.6.*) continue;; esac
printf '%s\n' "$ip $(whois "$ip" | grep -i 'country:')"
done < <(grep -E -o '[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+' | sort -u)
It's more efficient to perform input and output redirection for the script as a whole than to put a >> redirection after the printf itself (which would open the file before each print operation and close it again after, incurring a substantial performance penalty), which is why suggested invocation for this script looks something like:
countries_for_addresses </path/to/logfile >/path/to/output
I need to resolve a host name to an IP address in a shell script. The code must work at least in Cygwin, Ubuntu and OpenWrt(busybox).
It can be assumed that each host will have only one IP address.
Example:
input
google.com
output
216.58.209.46
EDIT:
nslookup may seem like a good solution, but its output is quite unpredictable and difficult to filter. Here is the result command on my computer (Cygwin):
>nslookup google.com
Unauthorized answer:
Serwer: UnKnown
Address: fdc9:d7b9:6c62::1
Name: google.com
Addresses: 2a00:1450:401b:800::200e
216.58.209.78
I've no experience with OpenWRT or Busybox but the following one-liner will should work with a base installation of Cygwin or Ubuntu:
ipaddress=$(LC_ALL=C nslookup $host 2>/dev/null | sed -nr '/Name/,+1s|Address(es)?: *||p')
The above works with both the Ubuntu and Windows version of nslookup. However, it only works when the DNS server replies with one IP (v4 or v6) address; if more than one address is returned the first one will be used.
Explanation
LC_ALL=C nslookup sets the LC_ALL environment variable when running the nslookup command so that the command ignores the current system locale and print its output in the command’s default language (English).
The 2>/dev/null avoids having warnings from the Windows version of nslookup about non-authoritative servers being printed.
The sed command looks for the line containing Name and then prints the following line after stripping the phrase Addresses: when there's more than one IP (v4 or 6) address -- or Address: when only one address is returned by the name server.
The -n option means lines aren't printed unless there's a p commandwhile the-r` option means extended regular expressions are used (GNU sed is the default for Cygwin and Ubuntu).
If you want something available out-of-the-box on almost any modern UNIX, use Python:
pylookup() {
python -c 'import socket, sys; print socket.gethostbyname(sys.argv[1])' "$#" 2>/dev/null
}
address=$(pylookup google.com)
With respect to special-purpose tools, dig is far easier to work with than nslookup, and its short mode emits only literal answers -- in this case, IP addresses. To take only the first address, if more than one is found:
# this is a bash-specific idiom
read -r address < <(dig +short google.com | grep -E '^[0-9.]+$')
If you need to work with POSIX sh, or broken versions of bash (such as Git Bash, built with mingw, where process substitution doesn't work), then you might instead use:
address=$(dig +short google.com | grep -E '^[0-9.]+$' | head -n 1)
dig is available for cygwin in the bind-utils package; as bind is most widely used DNS server on UNIX, bind-utils (built from the same codebase) is available for almost all Unix-family operating systems as well.
Here's my variation that steals from earlier answers:
nslookup blueboard 2> /dev/null | awk '/Address/{a=$3}END{print a}'
This depends on nslookup returning matching lines that look like:
Address 1: 192.168.1.100 blueboard
...and only returns the last address.
Caveats: this doesn't handle non-matching hostnames at all.
TL;DR; Option 2 is my preferred choice for IPv4 address. Adjust the regex to get IPv6 and/or awk to get both. There is a slight edit to option 2 suggested use given in EDIT
Well a terribly late answer here, but I think I'll share my solution here, esp. because the accepted answer didn't work for me on openWRT(no python with minimal setup) and the other answer errors out "no address found after comma".
Option 1 (gives the last address from last entry sent by nameserver):
nslookup example.com 2>/dev/null | tail -2 | tail -1 | awk '{print $3}'
Pretty simple and straight forward and doesn't really need an explanation of piped commands.
Although, in my tests this always gave IPv4 address (because IPv4 was always last line, at least in my tests.) However, I read about the unexpected behavior of nslookup. So, I had to find a way to make sure I get IPv4 even if the order was reversed - thanks regex
Option 2 (makes sure you get IPv4):
nslookup example.com 2>/dev/null | sed 's/[^0-9. ]//g' | tail -n 1 | awk -F " " '{print $2}'
Explanation:
nslookup example.com 2>/dev/null - look up given host and ignore STDERR (2>/dev/null)
sed 's/[^0-9. ]//g' - regex to get IPv4 (numbers and dots, read about 's' command here)
tail -n 1 - get last 1 line (alt, tail -1)
awk -F " " '{print $2} - Captures and prints the second part of line using " " as a field separator
EDIT: A slight modification based on a comment to make it actually more generalized:
nslookup example.com 2>/dev/null | printf "%s" "$(sed 's/[^0-9. ]//g')" | tail -n 1 | printf "%s" "$(awk -F " " '{print $1}')"
In the above edit, I'm using printf command substitution to take care of any unwanted trailing newlines.
Bonjour,
I launch nslookup someServer. I consider I get a serie of object (as in powers hell) separated by empty lines and not simply a stdout.
$ nslookup someServer
Server: 10.0.0.1
Address: 10.0.0.1#53
Name: someServer
Address: 10.0.0.5
$
How to get the object who have both properties Name and Address?
nslookup someServer | haveboth Name Address | wc -l
Does it exists in GNU utilities?
are you simply looking for a way to check that you have both of these values in your output?
Then you could use perl in oneline mode (probably not the most pretty solution imaginable, but does what you want and could be expanded to check more things easily)
nslookup someServer | perl -ne '$v{"NAME"}++ if /Name/; $v{"ADDRESS"}++ if /Address/; END{ print "Has both values\n" if $v{"NAME"} && $v{"ADDRESS"} }'
this goes through your output and counts the occurrences of Name and Address and then prints a message if it has more than zero of both.
If using perl in this way is a viable option, then I can recommend this page for further reading on perl oneliners
EDIT:
in case you want to have access to the values that are stored in your property: you can use
nslookup someServer | perl -F: -lane '$v{"NAME"} = $F[1] if /Name/; $v{"ADDRESS"} = $F[1] if /Address/; END{ print $v{"ADDRESS"}." ".$v{"NAME"} if exists $v{"ADDRESS"} && $v{"NAME"} }'
This will split a given line on : as field separator and store the value in the variable instead of simple counting occurrences. Note that done in this way it would only store the last occurrence.
I am trying to write a bash script that will from a domain name find it MX records, from them figure out which is the primary (they are not always in order) and then find its IP.
(when there are more then one primary MX the first one found would be ok)
For example:
./findmxip.sh gmail.com
Would give me 173.194.71.26. For me to do this I need to host gmail.com
then find the primary MX in the results and host it, getting its IP.
To get exactly 0 or 1 answers:
dig +short gmail.com mx | sort -n | nawk '{print $2; exit}' | dig +short -f -
You'll need a non-ancient dig that supports +short.
As noted there may be more than one "primary" MX as the preferences need not be unique. If you want all the IP addresses of all of the lowest preference records then:
dig +short oracle.com mx | sort -n |
nawk -v pref=65536 '($1<=pref) {pref=$1; print $2}' |
dig +short -f - | uniq
This does not handle the case where there is no MX record and the A record accepts email, an uncommon but perfectly valid configuration.
Sadly all the dig versions I've tested return 0 whether the domain exists or not (NXDOMAIN), and whether any MX records exist or not. You can catch a DNS time-out (rc=9) though. The related host command does return a non-zero rc with NXDOMAIN, but its behaviour is a little inconsistent, it's messy to script and the output harder to parse.
A poor man's error-checking version (inspired by tripleee's comment) that might be slightly more robust depending on your host command is:
DOMAIN=gmail.com
if ! host -t any $DOMAIN >/dev/null 2>&1 ; then
echo "no such domain"
elif ! host -t mx $DOMAIN >/dev/null 2>&1; then
echo "no MX records"
else
dig +short $DOMAIN mx | sort -n | nawk '{print $2; exit}' | dig +short -f -
fi
(Perversely, you may require an older version of host (bind-8.x) for the -t mx test to work, newer versions just return 0 instead.)
This is just about the point people start backing away nervously asking why you're not using perl/python/$MFTL.
If you really need to write a robust version in bash, check out the djbdns CLI tools and debugging tools which are rather easier to parse (though sadly don't set user exit codes either).