Host file make a unique file for all servers - sorting

Question

I have many hosts files. I collect them from all servers and i put them together in host_files.txt and then I must make one hosts file for all servers.
I do this command to make a unique file, but some rows share the same ip address or hostname.
awk '!a[$0]++' host_files.txt
Here is my host_files.txt
#backup server IPs
95.23.23.56
95.23.23.57
#ftp server IPs
45.89.67.5
45.89.67.3
#apache
12.56.35.36
12.56.35.35
#ftp server IPs
95.23.23.50
#apache
12.56.35.37
I want to output file, but I need to keep the comment line
#backup server IPs <= comment line, i need to keep them
95.23.23.56
95.23.23.57
#ftp server IPs <= comment line, i need to keep them
45.89.67.5
45.89.67.3
95.23.23.50
#apache <= comment line, i need to keep them
12.56.35.36
12.56.35.35
12.56.35.37
i already try :
sort -ur host_files.txt
cat host_files.txt | uniq > ok_host.txt
I need the ip without # just need ip adresse please help me
Thanks in advance

Answer 1

In GNU awk for using multidimensional arrays:
$ awk '
/^#/ { k=$0; next } # group within identical comments, k is key to hash
/./ { a[k][$1]=$0 } # remove empty records and hash ips
END { for(k in a) { # after everything, output
print k
for(i in a[k])
print a[k][i]
}
}' file*
#apache
12.56.35.35 #apacheprivate
12.56.35.36 #apachepub
12.56.35.37 #apachepub
#ftp server IPs
45.89.67.3 #ftpssh
45.89.67.5 #ftpmain
95.23.23.50 #ftp
#backup server IPs
95.23.23.56 #masterbasckup
95.23.23.57 #agentbasckup
The output is random order because of for(k in a), ie. comment groups and ips within groups are in no particular order.

Answer 2

This will work in any awk:
$ cat tst.awk
/^#/ { key = $0; next }
NF && !seen[$0]++ {
ips[key] = ips[key] $0 ORS
}
END {
for (key in ips) {
print key ORS ips[key]
}
}
$ awk -f tst.awk file
#apache
12.56.35.36 #apachepub
12.56.35.35 #apacheprivate
12.56.35.37 #apachepub
#ftp server IPs
45.89.67.5 #ftpmain
45.89.67.3 #ftpssh
95.23.23.50 #ftp
#backup server IPs
95.23.23.56 #masterbasckup
95.23.23.57 #agentbasckup
Output order will be random due to use of the in operator, if that's a problem it's just a couple more lines of code to change.

Answer 3

If awk is not a requirement.
#!/bin/ksh
cat host_files.txt | while read line ; do
[[ $line =~ ^$ ]] && { continue; } # skip empty lines
[[ $line =~ ^# ]] && { group=$line; continue; } # remember the group name
print "$group|$line" # print with group name in front
done | sort \
| while read line ; do
if [[ ${line%\|*} != $last ]]; then # if the group name changed
print "\n${line%\|*}" # print the group name
last=${line%\|*} # remember the new group name
fi
print "${line#*\|}" # print the entry without the group name
done
put the group name in front of the line
sort
detect changing group name and print it
print entry without group name
Using the same concept with awk (avoiding the while loop in shell).
awk '
/^#/ { k=$0; next }
/./ { print k "|" $0 }
' host_files.txt | sort | awk -F '|' '{
if ( k != $1 ) { print "\n" $1; k = $1; }
print $2
}' -
Because it does not use an array it would not loose lines due to duplicate keys.
And, thinking a bit more, the second awk can be avoided. Adding the key to each line. For the header without 'x'. So the header is sorted above the rest. In the output, just remove the added sort-key.
awk '
/^#/ { k=$0; print k "|" $0; next; }
/./ { print k "x|" $0 }
' t18.dat | sort -u | cut -d '|' -f 2