I am trying to get some numbers from tcpdump inside a shell script and print that number.
Here is my script
while true
do
{
b=`tcpdump -n -i eth1 | awk -F'[, ]' '{print $10}'`
echo $b
}
done
When I execute this script, I get this
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
Is there anything special I need to do to capture tcpdump o/p inside shell script ?
By default, tcpdump runs forever (or until it's interrupted by Control-C or something similar). The
b=`tcpdump ...`
construct runs until tcpdump exits... which is never ... and then puts its output into $b. If you want to capture the output from a single packet, you can use tcpdump -c1 ... (or -c5 to capture groups of 5, or similar). Alternately, you could let it run forever but capture its output one line at a time with a while read loop (although you need to use tcpdump -l to prevent excessive buffering):
tcpdump -l -n -i eth1 | awk -F'[, ]' '{print $10}' | while read b; do
echo $b
done
I'm not entirely sure what your script is supposed to do, but I see some other issues. First, unless your version of tcpdump is much more consistent than mine, printing the 10th comma-delimited field of each packet will not get you anything meaningful. Here's some sample output from my computer:
00:05:02:ac:54:1e
1282820004:1282820094
90
73487384:73487474
1187212630:1187212720
90
90
host
2120673524
Second, what's the point of capturing the output into a variable, then printing it? Why not just run the command and let it output directly? Finally, echo $b may garble the output due to word splitting and wildcard expansion (for example, if $b happened to be "*", it would print a list of files in the current directory). For this reason, you should double-quote variables when you use them (in this case, echo "$b").
It's been so long since this question was asked but the simplest way to accomplish the goal of what the script was intending to catch would be to simply record a pcap matching only the packets you're interestedin seeing; as an example, to write a pcap file consisting only of packets where the ack flag is set and the acknowledgment number is a value between 19000 and 20000:
tcpdump -c2500 -iany 'tcp[8:4]>=19000&&tcp[8:4]<=20000&&tcp[13]&16!=0' -Uw./TCP_ACKs.pcap
Related
Running this:
ping google.com | grep -o 'PING'
Will print PING to the terminal, so I assume that means that the stdout of grep was captured by the terminal.
So why doesn't the follow command print anything? The terminal just hangs:
ping google.com | grep -o 'PING' | grep -o 'IN'
I would think that the stdout of the first grep command would be redirected to the stdin of the second grep. Then the stdout of the second grep would be captured by the terminal and printed.
This seems to be what happens if ping is replaced with echo:
echo 'PING' | grep -o 'PING' | grep -o 'IN'
IN is printed to the terminal, as I would expect.
So what's special about ping that prevents anything from being printed?
You could try being more patient :-)
ping google.com | grep -o 'PING' | grep -o 'IN'
will eventually display output, but it might take half an hour or so.
Under Unix, the standard output stream handed to a program when it starts up is "line-buffered" if the stream is a terminal; otherwise it is fully buffered, typically with a buffer of 8 kilobytes (8,192 characters). Buffering means that output is accumulated in memory until the buffer is full, or, in the case of line-buffered streams, until a newline character is sent.
Of course, a program can override this setting, and programs which produce only small amounts of output -- like ping -- typically make stdout line-buffered regardless of what it is. But grep does not do so (although you can tell Gnu grep to do that by using the --line-buffered command-line option.)
"Pipes" (which are created to implement the | operator) are not considered terminals. So the grep in the middle will have a fully-buffered output, meaning that its output will be buffered until 8k characters are written. That will take a while in your case, because each line contains only five characters (PING plus a newline), and they are produced once a aecond. So the buffer will fill up after about 1640 seconds, which is almost 28 minutes.
Many unix distributions come with a program called stdbuf which can be used to change buffering for standard streams before running a program. (If you have stdbuf, you can find out how it works by typing man 1 stdbuf.) Programming languages like Perl generally provide other mechanisms to call the stdbuf standard library function. (In Perl, you can force a flush after every write using the builtin variable $|, or the autoflush(BOOL) io handle method.)
Of course, when a program successfully terminates, all output buffers are "flushed" (srnt to their respective streams). So
echo PING | grep -o 'PING' | grep -o 'IN'
will immediately output its only output line. But ping does not terminate unless you provide a count command-line option (-c N; see man ping). So if you need immediate piped throughput, you may need to modify buffering behaviour.
I've attempted numerous times and tried different methods but cannot seem to get this to work. I am trying to run a python script and grep the output to see if it is contained in a file and if it is not I want to append it to said file.
$./scan_network.py 22 192.168.1.1 192.168.1.20 | if ! grep -q - ./results.log; then - >> results.log; fi
I understand that it macOS grep does not understand - as stdout and that then - >> would not work because it would not pick up stdout either. I am not sure what to do.
As stated before the primary goal is to check the output of the script against a file and if the IP address is not found in the file, it needs to be appended.
Edit:
results.log is currently an empty file. Output of scan_network.py on would be 192.168.1.6 for now. When I go to run it on another network the output would be numerous addresses in a range example being 10.234.x.y where x and y would be any number between 0 and 255.
One simple solution is to merge the log file and the output of the program into a new log file:
sort -u <(./scan_network.py 22 192.168.1.1 192.168.1.20) results.log > newresults.log
The -u flag causes duplicate lines to be removed from the output, so you will get only one of each line.
That has the side effect of reordering the lines (so that they are sorted alphabetically). It is possible to preserve order if necessary, but it gets more complicated.
With a reasonably modern gnu sort, you can use a "version number" sort, which will do a reasonable job of keeping IP numbers in logical order; you can use the -V flag to do that. Or you can sort the octets individually with sort -u -t. -k1,1n -k2,2n -k3,3n -k4,4n .... Or you can just live with lexicographic ordering. Do not just use -n for standard numeric sorting, because it will only examine the first octet, and that will have an unfortunate interaction with the -u option, since two lines which compare equal are considered duplicates. As numeric sort only considers the numeric prefix, there will be many false duplicates.
If you don't mind sorting and rewriting your log file, rici's helpful answer works well (note that simply using -V for true per-component numerical IP-address sorting is not an option on macOS, unfortunately).[1].
Here's an alternative that only appends to the existing log file on demand, in-place, without reordering existing lines:
grep -f results.log -xFv <(./scan_network.py 22 192.168.1.1 192.168.1.20) >> results.log
Note: This assumes that ./scan_network.py's output is line-based; pipe to tr to transform to line-based output, if necessary.
-f treats each line in the specified file as a separate search term, where a match of any term is considered an overall match.
-x matches lines in full
-F performs literal matching (doesn't interpret search terms as regular expressions)
-v only outputs lines that do not match
The net effect is that only lines output by ./scan_network.py ... that aren't already present in results.log are appended to results.log.
Note, however, that performance will likely suffer the larger results.log becomes, so rici's approach may be preferable in the long run, particularly, if the log file keeps growing and/or you want the log sorted by IP addresses anyway.
As for what you've tried:
Both GNU and BSD/macOS grep optionally accept - as a placeholder for stdin to accept the input from, but note that this operand is never needed, because grep reads input from stdin by default.
By contrast, only GNU grep accepts - as the option-argument to -f, i.e., the file containing the search terms to apply.
BSD/macOS requires either an explicit filename, a process substitution (as above), or, in a pinch, /dev/stdin to refer to stdin.
The logic of your search must be reversed: as in the command above, the existing log file contents must serve as the search terms (passed to -f), and the ./scan_network.py ... output must serve as the input in order to determine which lines are not (-v) already in the log file.
using - to represent stdin or stdout, depending on context, is a mere convention that only works as a command argument, so your attempt to refer to stdout output with if ...; then - >> results.log cannot work, because - is invariably interpreted as a command name.
If you use grep -q, stdout output is by definition suppressed, so there's nothing to pass on (even if you used a pipe).
[1] macOS's (OS X's) sort does not support -V for per-component version-number sorting (which can be applied to IP addresses too). Even though the macOS sort is a GNU sort, it is an ancient one - v5.93 as of macOS 10.12 - that predates support for -V.
Assuming that your script returns a single line of text, you can store the output in a variable and then grep for that string. For example:
logfile="results.log"
# save output to a shell variable
str=$(./scan_network.py 22 192.168.1.1 192.168.1.20)
# don't call grep twice for the same pattern
grep=$(grep -F "$str" "$logfile")
# append if grep results are empty
if [[ -z "$grep" ]]; then
echo "$grep" >> "$logfile"
fi
( ping -n apple.com | ack -o --flush '((?<=icmp_seq=)[0-9]+|(?<=time=)[0-9]+[.][0-9]+)' | paste - - ) > ~/Desktop/ping_ouput.txt
Doesn't seem to be working for some reason, whereas if I take away the
| paste - -
section, it works just fine. I need to join every other line together with a tab instead of newline. Any ideas are appreciated.
Update:
By default, ping produces output indefinitely, until terminated, so your pipe will keep producing output and growing the output file indefinitely, and your command will never finish by itself.
Thus, you need to limit the number of pings performed; e.g.:
using the -c count option to stop after the specified number of pings.
alternatively, on BSD-like systems, you can use -o to stop after the first successful ping.
alternatively, you can stop after a specified timeout in seconds, regardless of how many packets were received:
Linux: -w timeout
BSD-like systems: -t timeout
Incidentally, in order to redirect a pipeline to a file, there's no need to enclose it in (...) as a whole, which needlessly creates a(nother) subshell.
The following was written before (a) the specific symptom was known and (b) before the OP revealed in a comment that their platform is OSX. The commands below show alternatives to the OP's extraction command; -c 2 has been added to limit ping to 2 attempts.
tivn's answer has found at least potential problem in your command: the assumption that time values always have a decimal point, which does not hold on Linux platforms; on BSD/OSX platforms, however, there are always 3 decimal places.
You can bypass this issue as well as the need to merge consecutive lines as follows (I'm using 127.0.0.1 instead of apple.com for easier testing):
Using either GNU sed (Linux) or BSD sed (BSD-like systems, including OSX):
ping -c 2 -n 127.0.0.1 |
sed -E '1d; s/^.* icmp_seq=([^ ]+).* time=([^ ]+).*$/\1'$'\t''\2/'
Alternative solutions, using awk:
ping -c 2 -n 127.0.0.1 | awk -F'[ =]' -v OFS='\t' 'NR>1 { print $6, $10 }'
If you'd rather not rely on field indices, here's an alternative (still relies on field icmp_seq to come before time, and for both to be preceded by a space):
ping -c 2 -n 127.0.0.1 | awk -F' (icmp_seq|time)=' -v OFS='\t' '
NR>1 { sub(" .+$", "", $2); sub(" .+$", "", $3)
print $2, $3 }'
You don't provide sample input you got from the ping, so it is not clear what your problem is. But I guess your issue is because the time part from ping result may not always contains dot ., for example: 64 bytes from x.x.x.x: icmp_seq=1 ttl=63 time=137 ms . You may want to try with the following command :
( ping -n yahoo.com \
| ack -o --flush '((?<=icmp_seq=)\d+|(?<=time=)[\d.]+)' \
| stdbuf -o0 paste - - \
) > ~/Desktop/ping_ouput.txt
Edit: after comment from OP, the issue may actually come from buffered paste command. Please try with adding stdbuf -o0 to the paste command.
Good day. I was reading another post regarding resolving hostnames to IPs and only using the first IP in the list.
I want to do the opposite and used the following script:
#!/bin/bash
IPLIST="/Users/mymac/Desktop/list2.txt"
for IP in 'cat $IPLIST'; do
domain=$(dig -x $IP +short | head -1)
echo -e "$domain" >> results.csv
done < domainlist.txt
I would like to give the script a list of 1000+ IP addresses collected from a firewall log, and resolve the list of destination IP's to domains. I only want one entry in the response file since I will be adding this to the CSV I exported from the firewall as another "column" in Excel. I could even use multiple responses as semi-colon separated on one line (or /,|,\,* etc). The list2.txt is a standard ascii file. I have tried EOF in Mac, Linux, Windows.
216.58.219.78
206.190.36.45
173.252.120.6
What I am getting now:
The domainlist.txt is getting an exact duplicate of list2.txt while the results has nothing. No error come up on the screen when I run the script either.
I am running Mac OS X with Macports.
Your script has a number of syntax and stylistic errors. The minimal fix is to change the quotes around the cat:
for IP in `cat $IPLIST`; do
Single quotes produce a literal string; backticks (or the much preferred syntax $(cat $IPLIST)) performs a command substitution, i.e. runs the command and inserts its output. But you should fix your quoting, and preferably read the file line by line instead. We can also get rid of the useless echo.
#!/bin/bash
IPLIST="/Users/mymac/Desktop/list2.txt"
while read IP; do
dig -x "$IP" +short | head -1
done < "$IPLIST" >results.csv
Seems that in your /etc/resolv.conf you configured a nameserver which does not support reverse lookups and that's why the responses are empty.
You can pass the DNS server which you want to use to the dig command. Lets say 8.8.8.8 (Google) for example:
dig #8.8.8.8 -x "$IP" +short | head -1
The commands returns the domain with a . appended. If you want to replace that you can additionally pipe to sed:
... | sed 's/.$//'
I'm trying to program a little "dirty" website filter - e.g. an user wants to visit an erotic website (based on domain name)
So basically, I got something like
#!/bin/bash
sudo tshark -i any tcp port 80 or tcp port 443 -V | grep "Host.*keyword"
It works great but now I need to do some actions after I find something (iptables and DROPing packets...). The problem I got is that tcp dumping is still running. If I had a complete file with data, the thing I'm trying to reach is easy to solve.
In pseudocoude, I'd like to have something like:
if (tshark and grep found something)
iptables - drop packets
sleep 600 # a punishment for an user
iptables accept packets I was dropping
else
still look for a match in the tcp dump that's still running
Thanks for your help.
Maybe you could try something like the following:
tshark OPTIONS 2>&1 | grep --line-buffered PATTERN | while read line; do
# actions for when the pattern is found, the matched input is in $line
break
done
The 2>&1 is important so that when PATTERN is matched and the while loop terminates, tshark has nowhere to write to and terminates because of the broken pipe.
If you want to keep tshark running and analyze future output, just remove the break. This way, the while loop never terminates and it keeps reading the filtered output from tshark.