I’m doing simple bash script that is basically working with files and basic bash functions. However I’ve crossed something I don’t know how to deal with. I need to convert 1st column of inputed file IPv4/v6 to their hostname and prints that out. w
I know, that to print out 1st column is good to use awk | ‘{print $1}’, and to find host of IP is good nslookup, but I can’t quite put those together to echo out that IP column as HOSTNAME column.
This is how I print out 1st column. Parameter passed into function is a filename.
function ip_echo(){
cat $1 | awk '{print $1}’
}
You can look here to get sense how to do it.
In short, your command should use dig and reverse lookup to translate the IP's into their hostnames.
Finally, your bash command should be look like this -
cat $1 | awk '{print $1}' | while read count ip; do printf "%d " $count; printf "%s " $ip; dig +noall +answer -x $ip; done
Related
I have a script that read log files and parse the data to insert them to mysql table..
My script looks like
while read x;do
var=$(echo ${x}|cut -d+ -f1)
var2=$(echo ${x}|cut -d_ -f3)
...
echo "$var,$var2,.." >> mysql.infile
done<logfile
The Problem is that log files are thousands of lines and taking hours....
I read that awk is better, I tried, but don't know the syntax to parse the variables...
EDIT:
inputs are structure firewall logs so they are pretty large files like
#timestamp $HOST reason="idle Timeout" source-address="x.x.x.x"
source-port="19219" destination-address="x.x.x.x"
destination-port="53" service-name="dns-udp" application="DNS"....
So I'm using a lot of grep for ~60 variables e.g
sourceaddress=$(echo ${x}|grep -P -o '.{0,0}
source-address=\".{0,50}'|cut -d\" -f2)
if you think perl will be better I'm open to suggestions and maybe a hint how to script it...
To answer your question, I assume the following rules of the game:
each line contains various variables
each variable can be found by a different delimiter.
This gives you the following awk script :
awk 'BEGIN{OFS=","}
{ FS="+"; $0=$0; var=$1;
FS="_"; $0=$0; var2=$3;
...
print var1,var2,... >> "mysql.infile"
}' logfile
It basically does the following :
set the output separator to ,
read line
set the field separator to +, re-parse the line ($0=$0) and determine the first variable
set the field separator to '_', re-parse the line ($0=$0) and determine the second variable
... continue for all variables
print the line to the output file.
The perl script below might help:
perl -ane '/^[^+]*/;printf "%s,",$&;/^([^_]*_){2}([^_]*){1ntf "%s\n",$+' logfile
Since, $& can result in performance penalty, you could also use the /p modifier like below :
perl -ane '/^[^+]*/p;printf "%s,",${^MATCH};/^([^_]*_){2}([^_]*){1}_.*/;printf "%s\n",$+' logfile
For more on perl regex matching refer to [ PerlDoc ]
if you're extracting the values in order, something like this will help
$ awk -F\" '{for(i=2;i<=NF;i+=2) print $i}' file
idle Timeout
x.x.x.x
19219
x.x.x.x
53
dns-udp
DNS
you can easily change the output format as well
$ awk -F\" -v OFS=, '{for(i=2;i<=NF;i+=2)
printf "%s", $i ((i>NF-2)?ORS:OFS)}' file
idle Timeout,x.x.x.x,19219,x.x.x.x,53,dns-udp,DNS
I have a file "IP" with number of blocked different subnet IP addresses, I want to change last octal digits of subnet, like 1.1.1.5 to 1.1.1.0/24. I wrote script. Issue is that awk command out doesn't assigned to any variable.
Example:
sed "s/139.196.8.79/139.196.8.0\/24/g" ip |a=$(awk -F. '{print $4}')
But when I run awk -F. '{print S4}', it returns output but doesn't assign value to variable.
Thanks in advance.
Not sure about your expected output but... looking your example here above, maybe, you're looking for something like this:
$ a=$(sed "s/139.196.8.79/139.196.8.0\/24/g" ip | awk -F. '{print $4}')
$ echo $a
0/24
WHat you were trying to write is:
a="$(sed "s/139.196.8.79/139.196.8.0\/24/g" ip | awk -F. '{print $4}' )"
but you need to escape the RE metacharacters (.s) and you don't need sed when you're using awk:
a="$(awk -F. '{gsub(/139\.196\.8\.79/,"139.196.8.0/24"); print $4}' ip)"
The above is untested and may be wrong or not the best approach since you didn't provide any sample input and expected output in your question.
Note that the above should really be using word boundaries if an IP addr like 139.196.8.790 can occur in your data but again without sample input and expected output....
A pipe creates a sub process. This means that you create the variable in a sub shell, which gets discarded right after. In order to access the variable, you have to group your commands into curly braces:
sed "s/139.196.8.79/139.196.8.0\/24/g" ip | {
a=$(awk -F. '{print $4}')
echo $a
}
Everything you want to do with a must be done in the sub shell, in which a has been defined.
I have a CSV list that is two columns (col1 is Share Name, col2 is file system path). I need two variables for either everything BEFORE the comma, or everything AFTER the column. My issue is that either column potentially has spaces, and even though these are quoted in the output, my script isn't handling them properly.
CSV:
ShareName,/path/to/sharename
"Share with spaces",/path/to/sharewithspaces
ShareWithSpace,"/path/to/share with spaces"
I was using this awk statement to get either field 1 or field 2:
echo $line | awk -F "\"*,\"*" '{print $2}'
BUT, I soon realized that it wasn't handling the spaces properly, even when passing that command to a variable and quoting the variable.
So, then after googling my brain out, I was trying this:
echo $line | cut -d, -f2
Which works, EXCEPT when echoing the variable $line. If I echo the string, it works perfectly, but unfortunately I'm using this in a while/read/do.
I am fairly certain my issue is having to define fields and having whitespace, but I really only need before or after a comma.
Here's the stripped down version so there's no sensitive data.
#!/usr/bin/bash
ssh <ip> <command> > "2_shares.txt"
<command> > "1_shares.txt"
file1="1_shares.txt"
file2="2_shares.txt"
while read -r line
do
share=`echo "$line" | awk -F "\"*,\"*" '{print $1}'`
path=`echo "$line" | awk -F "\"*,\"*" '{print $2}'`
if grep "$path" $file2 > /dev/null;
then
:
else
echo "SHARE NEEDS CREATED FOR $line"
case $path in
*)
blah blah blah
;;
esac
fi
done < "$file1"
You could simply do like this,
awk -F',' '{print $2}' file
To skip the first line.
awk -F',' 'NR>1{print $2}' file
Your issue is simply that you aren't quoting your shell variables. ALWAYS quote shell variables unless you have a very specific reason not to and are fully aware of all of the consequences.
I strongly suspect the rest of your script is completely wrong in it's approach since you apparently didn't know to quote variables and are talking about shell loops and echoing one line at time to awk so please do post a followup question if you'd like help.
Howdie do,
I'm currently working on a script that will take a list of IPs, store them in a LIST, loop through that list and compare the IP's to those in two text files. If the IP is duplicated in both files, it will remove the IP from one of the files.
The two files that contain the duplicates:
cat jeremy
209.240.105.0
cat jeremy2
209.240.105.0
Now the code is pretty simple:
LIST="$(cat /STORAGE/ips | awk -F ':' '{print $1}')"
for I in $LIST
do
DUP1=$(grep -rwl "$I" /STORAGE/jeremy/ | awk -F '/' '{print $4}' | sed 2d)
DUP2=$(grep -rwl "$I" /STORAGE/jeremy/ | awk -F '/' '{print $4}' | sed 1d)
cat $DUP1 | while read IP; do sed -i "/^${IP}$/d" $DUP2 ; done
done
That actually works and removes the duplicate IP from the $DUP2 file as it should, but it seems to get stuck in an infinite loop.
I saw this as after I run the script, it will remove the duplicate as it should, but the script just keeps running.
If you press enter while the script is turning it's wheels, it spits out:
sed: no input files
sed: no input files
But you can clearly see the duplicate IP has been removed:
[/STORAGE/jeremy]# cat jeremy
[/STORAGE/jeremy]# cat jeremy2
209.240.105.0
So it does it's job, but the sed command seems to be stuck in a loop. I've only today really started to learn more about sed it's capabilites, but is there an equivalent to break; like c++ or c#?
I just need sed to break out of the while read loop
The Input and Output files are posted below and also, this is not a duplicate question. I did raise a question earlier about this script, but that was just to get better understanding of how to use the regex with the sed and awk.
IP Input file that generates $LIST
209.240.105.0:255.255.255.255:209.240.105.0
209.240.105.1:255.255.255.255:209.240.105.1
The two files that I'm testing on just contain a list of one IP at the time:
Test file #1 jeremy:
209.240.105.0
Test file #2 jeremy2:
209.240.105.0
Once the script runs, it should only remove the IP from the Test File #2:
Test file #1 jeremy:
209.240.105.0
Test file #2 jeremy2:
Which the script currently does. It's just that I have to kill the script manually instead of it breaking out of the while read loop
Let's start with this, uses GNU awk for "\<" word-delimiters:
gawk -F':' '
NR==FNR{ gsub(/\./,"\\."); ips["\\<" $1 "\\>"]; next }
{
for (ip in ips) {
if ( match($0,ip) ) {
print ip, FILENAME, RSTART, RLENGTH
}
}
}
' /STORAGE/ips /STORAGE/jeremy/* |
sort
That should print, for each IP address, the file name(s) it occurs in plus the character position it first occurs in on each line, and the length of the IP address.
Does it?
Once you post some sample input and expected output we can go further.
Protagonists
The Admin
Pipes
The Cron Daemon
A bunch of text processing utilities
netstat
>> the Scribe
Setting
The Cron Daemon is repeatedly performing the same job where he forces an innocent netstat to show the network status (netstat -n). Pipes then have to pick up the information and deliver it to bystanding text processing utilities (| grep tcp | awk '{ print $5 }' | cut -d "." -f-4). >> has to scribe the important results to a file. As his highness, The Admin, is a lazy and easily annoyed ruler, >> only wants to scribe new information to the file.
*/1 * * * * netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file
Soliloquy by >>
To append, or not append, that is the question:
Whether 'tis new information to bother The Admin with
and earn an outrageous Fortune,
Or to take Arms against `netstat` and the others,
And by opposing, ignore them? To die: to sleep;
note by the publisher: For all those that had problems understanding Hamlet, like I did, the question is, how do I check if the string is already included in the file and if not, append it to the file?
Unless you are dealing with a very big file, you can use the uniq command to remove the duplicate lines from the file. This means you will also have the file sorted, I don't know if this is an advantage or disadvantage for you:
netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file && sort /tmp/file | uniq > /tmp/file.uniq
This will give you the sorted results without duplicates in /tmp/file.uniq
What a piece of work is piping, how easy to reason about,
how infinite in use cases, in bash and script,
how elegant and admirable in action,
how like a vim in flexibility,
how like a gnu!
Here is a slightly different take:
netstat -n | awk -F"[\t .]+" '/tcp/ {print $9"."$10"."$11"."$12}' | sort -nu | while read ip; do if ! grep -q $ip /tmp/file; then echo $ip >> /tmp/file; fi; done;
Explanation:
awk -F"[\t .]+" '/tcp/ {print $9"."$10"."$11"."$12}'
Awk splits the input string by tabs and ".". The input string is filtered (instead of using a separate grep invocation) by lines containing "tcp". Finally the resulting output fields are concatenated with dots and printed out.
sort -nu
Sorts the IPs numerically and creates a set of unique entries. This eliminates the need for the separate uniq command.
if ! grep -q $ip /tmp/file; then echo $ip >> /tmp/file; fi;
Greps for the ip in the file, if it doesn't find it, the ip gets appended.
Note: This solution does not remove old entries and clean up the file after each run - it merely appends - as your question implied.