I'm trying to program a little "dirty" website filter - e.g. an user wants to visit an erotic website (based on domain name)
So basically, I got something like
#!/bin/bash
sudo tshark -i any tcp port 80 or tcp port 443 -V | grep "Host.*keyword"
It works great but now I need to do some actions after I find something (iptables and DROPing packets...). The problem I got is that tcp dumping is still running. If I had a complete file with data, the thing I'm trying to reach is easy to solve.
In pseudocoude, I'd like to have something like:
if (tshark and grep found something)
iptables - drop packets
sleep 600 # a punishment for an user
iptables accept packets I was dropping
else
still look for a match in the tcp dump that's still running
Thanks for your help.
Maybe you could try something like the following:
tshark OPTIONS 2>&1 | grep --line-buffered PATTERN | while read line; do
# actions for when the pattern is found, the matched input is in $line
break
done
The 2>&1 is important so that when PATTERN is matched and the while loop terminates, tshark has nowhere to write to and terminates because of the broken pipe.
If you want to keep tshark running and analyze future output, just remove the break. This way, the while loop never terminates and it keeps reading the filtered output from tshark.
Related
Running this:
ping google.com | grep -o 'PING'
Will print PING to the terminal, so I assume that means that the stdout of grep was captured by the terminal.
So why doesn't the follow command print anything? The terminal just hangs:
ping google.com | grep -o 'PING' | grep -o 'IN'
I would think that the stdout of the first grep command would be redirected to the stdin of the second grep. Then the stdout of the second grep would be captured by the terminal and printed.
This seems to be what happens if ping is replaced with echo:
echo 'PING' | grep -o 'PING' | grep -o 'IN'
IN is printed to the terminal, as I would expect.
So what's special about ping that prevents anything from being printed?
You could try being more patient :-)
ping google.com | grep -o 'PING' | grep -o 'IN'
will eventually display output, but it might take half an hour or so.
Under Unix, the standard output stream handed to a program when it starts up is "line-buffered" if the stream is a terminal; otherwise it is fully buffered, typically with a buffer of 8 kilobytes (8,192 characters). Buffering means that output is accumulated in memory until the buffer is full, or, in the case of line-buffered streams, until a newline character is sent.
Of course, a program can override this setting, and programs which produce only small amounts of output -- like ping -- typically make stdout line-buffered regardless of what it is. But grep does not do so (although you can tell Gnu grep to do that by using the --line-buffered command-line option.)
"Pipes" (which are created to implement the | operator) are not considered terminals. So the grep in the middle will have a fully-buffered output, meaning that its output will be buffered until 8k characters are written. That will take a while in your case, because each line contains only five characters (PING plus a newline), and they are produced once a aecond. So the buffer will fill up after about 1640 seconds, which is almost 28 minutes.
Many unix distributions come with a program called stdbuf which can be used to change buffering for standard streams before running a program. (If you have stdbuf, you can find out how it works by typing man 1 stdbuf.) Programming languages like Perl generally provide other mechanisms to call the stdbuf standard library function. (In Perl, you can force a flush after every write using the builtin variable $|, or the autoflush(BOOL) io handle method.)
Of course, when a program successfully terminates, all output buffers are "flushed" (srnt to their respective streams). So
echo PING | grep -o 'PING' | grep -o 'IN'
will immediately output its only output line. But ping does not terminate unless you provide a count command-line option (-c N; see man ping). So if you need immediate piped throughput, you may need to modify buffering behaviour.
I would like to see whenever the ping to a specific server goes down.
So at the moment I have something like this: ping 8.8.8.8 | grep time=11 && date
This pings google and prints every line that has a ping of 11.x that is just for testing so that I have some output that is not there all the time but often enough that I can test this. And then I have && date that I thought would print the date when grep finds the string.. But it does not. So my question now Is how can I get it so that in every line that grep spits out I can append the current timestamp. Or what would be even better is to have it as a prefix in front of every line that grep spits out. Is this possible with one command? Or do I need a simple shell script?
Thanks in advance. I hope I get an answer soon. :)
Also merry x-mas to everyone!
See if it works for you:
ping 8.8.8.8 | while read line; do echo "$(date): $line"; done | grep time=13
You can also turn the setup around to check when a remote host becomes available by using while ! ping ... Example, in case your path to 8.8.8.8 goes down:
declare -i cnt=0; while ! ping -c 1 8.8.8.8; do ((cnt++)); echo "test '$cnt' - network down on $(date)"; sleep 60; done
Which will check once per-minute if the host 8.8.8.8 is available and report, for example:
test '10' - network down on Sat Dec 26 16:38:48 CST 2015
The script will terminate on its own once the network path is established.
I'm having trouble removing the output from the netstat command when I check if a current port is being used. I dont need the output of the command but rather just the error code.
Running netstat -anp | grep 1521 &>/dev/null results in
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
being shown as the output. I have also tried:
2>/dev/null| cut -d' ' -f45-50 | sed 's/[^0-9]*//g'
which just results in a bunch of empty lines after the Not all processes... output but still displays that in the output.
ended up solving my own problem.
Use the following
netstat -taepn 2>/dev/null | grep -Po "\b(\d|\.)+:1521\b" 1>/dev/null
This is specific for my situation when I just want the error code and no output
The accepted answer doesn't work for me, CentOS6.5 32bit here. It outputs nothing no matter what state the port is.
I found my solution:
netstat -antp 2>/dev/null | grep [port-num]
This will omit the first two lines of netstat if it is executed by regular user.
The warning two lines needed to be omitted:
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
and works for two situations:
the port is "LISTEN" state
the port is "TIME_WAIT" state.
Instead of
netstat -anp
you may check using
sudo netstat -anp
This worked for me.
Good day. I was reading another post regarding resolving hostnames to IPs and only using the first IP in the list.
I want to do the opposite and used the following script:
#!/bin/bash
IPLIST="/Users/mymac/Desktop/list2.txt"
for IP in 'cat $IPLIST'; do
domain=$(dig -x $IP +short | head -1)
echo -e "$domain" >> results.csv
done < domainlist.txt
I would like to give the script a list of 1000+ IP addresses collected from a firewall log, and resolve the list of destination IP's to domains. I only want one entry in the response file since I will be adding this to the CSV I exported from the firewall as another "column" in Excel. I could even use multiple responses as semi-colon separated on one line (or /,|,\,* etc). The list2.txt is a standard ascii file. I have tried EOF in Mac, Linux, Windows.
216.58.219.78
206.190.36.45
173.252.120.6
What I am getting now:
The domainlist.txt is getting an exact duplicate of list2.txt while the results has nothing. No error come up on the screen when I run the script either.
I am running Mac OS X with Macports.
Your script has a number of syntax and stylistic errors. The minimal fix is to change the quotes around the cat:
for IP in `cat $IPLIST`; do
Single quotes produce a literal string; backticks (or the much preferred syntax $(cat $IPLIST)) performs a command substitution, i.e. runs the command and inserts its output. But you should fix your quoting, and preferably read the file line by line instead. We can also get rid of the useless echo.
#!/bin/bash
IPLIST="/Users/mymac/Desktop/list2.txt"
while read IP; do
dig -x "$IP" +short | head -1
done < "$IPLIST" >results.csv
Seems that in your /etc/resolv.conf you configured a nameserver which does not support reverse lookups and that's why the responses are empty.
You can pass the DNS server which you want to use to the dig command. Lets say 8.8.8.8 (Google) for example:
dig #8.8.8.8 -x "$IP" +short | head -1
The commands returns the domain with a . appended. If you want to replace that you can additionally pipe to sed:
... | sed 's/.$//'
I am trying to get some numbers from tcpdump inside a shell script and print that number.
Here is my script
while true
do
{
b=`tcpdump -n -i eth1 | awk -F'[, ]' '{print $10}'`
echo $b
}
done
When I execute this script, I get this
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
Is there anything special I need to do to capture tcpdump o/p inside shell script ?
By default, tcpdump runs forever (or until it's interrupted by Control-C or something similar). The
b=`tcpdump ...`
construct runs until tcpdump exits... which is never ... and then puts its output into $b. If you want to capture the output from a single packet, you can use tcpdump -c1 ... (or -c5 to capture groups of 5, or similar). Alternately, you could let it run forever but capture its output one line at a time with a while read loop (although you need to use tcpdump -l to prevent excessive buffering):
tcpdump -l -n -i eth1 | awk -F'[, ]' '{print $10}' | while read b; do
echo $b
done
I'm not entirely sure what your script is supposed to do, but I see some other issues. First, unless your version of tcpdump is much more consistent than mine, printing the 10th comma-delimited field of each packet will not get you anything meaningful. Here's some sample output from my computer:
00:05:02:ac:54:1e
1282820004:1282820094
90
73487384:73487474
1187212630:1187212720
90
90
host
2120673524
Second, what's the point of capturing the output into a variable, then printing it? Why not just run the command and let it output directly? Finally, echo $b may garble the output due to word splitting and wildcard expansion (for example, if $b happened to be "*", it would print a list of files in the current directory). For this reason, you should double-quote variables when you use them (in this case, echo "$b").
It's been so long since this question was asked but the simplest way to accomplish the goal of what the script was intending to catch would be to simply record a pcap matching only the packets you're interestedin seeing; as an example, to write a pcap file consisting only of packets where the ack flag is set and the acknowledgment number is a value between 19000 and 20000:
tcpdump -c2500 -iany 'tcp[8:4]>=19000&&tcp[8:4]<=20000&&tcp[13]&16!=0' -Uw./TCP_ACKs.pcap