Network Monitor: Don't Show Same Line Immediately Twice [duplicate] - macos

Question

I am writing a network monitoring script in bash. The base command I am using is ettercap -T -M ARP -i en1 // //. Then I pipe egrep --color 'Host:|GET' into it.
A sample output I am getting looks like this:
GET /images/srpr/logo11w.png HTTP/1.1.
Host: www.google.com.
GET /en-us/us/products HTTP/1.1.
Host: www.caselogic.com.
My desired output is this:
Title: logo11w.png
URL: www.google.com/images/srpr/logo11w.png HTTP/1.1.
Title: Products - Case Logic
URL: www.caselogic.com/en-us/us/products
Things to notice: HTTP/1.1. and the . at the end of the host are gone. They also are formed into one URL and there is a blank line after each Title/URL listing. I attempted forming them into one URL by parsing the commands output into a variable with
var=`sudo ettercap -T -M ARP -i en1 // // | egrep --color 'Host:|GET'` | echo $var
but obviously that doesn't work because the input to the variable is a command the isn't done until the user requests a stop (CTRL + C).
To get the title of an HTML page, I use the command wget -qO- 'https://url.goes/here' | perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si'. If it is something that doesn't have a title, such as an image, no title is fine.
Any help is greatly appreciated, and sorry if what I wrote is hard to read, feel free to ask questions.

Answer 1

Try this:
title_host.pl
#!/usr/bin/env perl
use warnings;
use strict;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
my ($get,$host,$title);
while (<>) {
if (m|^GET (\S+) |) {
$get = $1;
} elsif ( m|^Host: (\S+)\.| ) {
$host = $1;
} else {
# Unrecognized line...reset
$get = $host = $title = '';
}
if ($get and $host) {
my ($title) = $get =~ m|^.*\/(.+?)$|; # default title
my $url = 'http://' . $host . $get;
$mech->get($url);
if ($mech->success) {
# HTML may have title, images will not
$title = $mech->title() || $title;
}
print "Title: $title\n";
print "URL: $url\n";
print "\n";
$get = $host = $title = '';
}
}
input
GET /images/srpr/logo11w.png HTTP/1.1.
Host: www.google.com.
GET /en-us/us/products HTTP/1.1.
Host: www.caselogic.com.
now just pipe your input into the perl script:
cat input | perl title_host.pl
output:
Title: logo11w.png
URL: http://www.google.com/images/srpr/logo11w.png
Title: Products - Case Logic
URL: https://www.caselogic.com/en-us/us/products