I'm looking for a way to pipe multiple log files on multiple remote servers, and then pipe the result to another program.
Right now I'm using multitail, but it does not exactly do what I need, or maybe I'm doing something wrong!
I would like to be able to send the merge of all log files, to another program. For example jq. Right now if I do:
multitail --mergeall -l 'ssh server1 "tail -f /path/to/log"' -l 'ssh server2 "tail -f /path/to/log"' -l 'ssh server3 "tail -f /path/to/log"' | jq .
for instance, I get this:
parse error: Invalid numeric literal at line 1, column 2
But more generally, I would like to give the output of this to another program I use to parse and display logs :-)
Thanks everybody!
One way to accomplish this feat would be to pipe all your outputs together into a named pipe and then deal with the output from that named pipe.
First, create your named pipe: $ mknod MYFIFO p
For each location you want to consolidate lines from, $ tail -f logfile > MYFIFO (note, the tail -f can be run through an ssh session).
Then have another process take the data out of the named pipe and handle it appropriately. An ugly solution could be:
$ tail -f MYFIFO | jq
Season to taste.
Related
I am trying to to provide a file for my shell as an input which in return should test if the file contains a specific word and decide what command to execute. I am not figuring out yet where the mistake might lie. Please find the shell script that i wrote:
#!/bin/(shell)
input_file="$1"
output_file="$2"
grep "val1" | awk -f ./path/to/script.awk $input_file > $output_file
grep "val2" | sh ./path/to/script.sh $input_file > $output_file
when I input the the file that uses awk everything get executed as expected, but for the second command I don't even get an output file. Any help is much appreciated
Cheers,
You haven't specified this in your question, but I'm guessing you have a file with the keyword, e.g. file cmdfile that contains x-g301. And then you run your script like:
./script "input_file" "output_file" < cmdfile
If so, the first grep command will consume the whole cmdfile on stdin while searching for the first pattern, and nothing will be left for the second grep. That's why the second grep, and then your second script, produces no output.
There are many ways to fix this, but choosing the right one depends on what exactly you are trying to do, and how does that cmdfile look like. Assuming that's a larger file with other things than just the command pattern, you could pass that file as a third argument to your script, like this:
./script "input_file" "output_file" "cmdfile"
And have your script handle it like this:
#!/bin/bash
input_file="$1"
output_file="$2"
cmdfile="$3"
if grep -q "X-G303" "$cmdfile"; then
awk -f ./mno/script.awk "$input_file" > t1.json
fi
if grep -q "x-g301" "$cmdfile"; then
sh ./mno/tm.sh "$input_file" > t2.json
fi
Here I'm also assuming that your awk and sh scripts don't really need the output from grep, since you're giving them the name of the input file.
Note the proper way to use grep for existence search is via its exit code (and the muted output with -q). Instead of the if we could have used shortcircuiting (grep ... && awk ...), but this way is probably more readable.
I need to continuously read a pcap file recorded by a pcap capture program.
tcpdump -i eth0 -w foo.pcap
tcpstat -r foo.pcap -o "pps: %p\n" 1
For an example, assume I am recording a pcap by above tcpdump command and trying to read with tcpstat. Here, tcpstat is executing current foo.pcap and finishing its job. What I want is that make it wait for new packages and process them too. You can think is as a tailf command but reads pcap instead text file.
Note that here tcpdump and tcpstat only 2 example of reader and writer processes. They can be any other programs. Also note that reader processes can be more than one. What is the best way to accomplish that?
Ignoring the pcap stuff (see end of post) to start, and answering the general question:
... here tcpdump and tcpstat only 2 example of reader and writer processes. They can be any other programs. Also note that reader processes can be more than one. What is the best way to accomplish that?
So one writer multiple reader problem.
What's the best solution in Bash? Probably tail -F my.log file. Example:
while true; do echo ${RANDOM} >> my.log; sleep 1; done &
tail -F my.log &
tail -F my.log &
tail -F my.log &
There is also tee but that writes to multiple files not multiple processes. I guess you could setup a bunch of pipes to processes and use tee.
Note there are plenty of better solutions not in Bash that you probably want to use if you want a higher quality solution. One general issue with tail and UNIX pipes is that they are stream orientated, and you probably want a message orientated stream. UNIX message queues, a queue server, or maybe sockets can help with that.
On the pcap case
tail -F won't work because pcap file format has a header and you can't just start reading from the end of the file like tail -F does. You need to read the whole file start to finish then start tailing. -n option to tail does that:
tail -n+0 -F foo.pcap | tcpstat -r- -o "pps: %p\n"
tcpstat can read data from stdin:
-r filename
Read all data from filename, which may be a regular file, a named pipe or "-" to read it's data from standard input.
So:
tcpdump -i eth0 -w foo.pcap &
tail -f foo.pcap | tcpstat -r - -o "pps: %p\n" 1
Also, you would run this commands in different termial windows
I have a Python script which needs a --file xyz.json argument.
Thing is, my JSON is immense, hence it is Gzipped. Of course I could un-gzip it and then run the script, but that seems wasteful. Is there a clever way I can get this to work while doing a zcat xyz.json.gz | myscript.py --file ?????. I don't want to go into modifying myscript.py to read stdin instead of a file unless there's no way to get this done otherwise.
Thanks!
Try:
myscript.py --file <(zcat xyz.json.gz)
A file descriptor containing the pipe is returned. Provided that the script just reads the file, and does not search forward and backward, this should work.
The <( ... ) is called process substitution.
As an elaboration on what happens:
% awk 'BEGIN{print "filename:", ARGV[1]};1' <(echo first; sleep 1; echo second)
filename: /proc/self/fd/11
first
second
The second gets printed after a delay. So: Awk gets the filename /proc/self/fd/11, and starts to process it. It will immediately see the first line, and print it out. Then, after the sleep, it will see the second line, and print it as well.
You can use /dev/stdin or (equivalently) /dev/fd/0:
zcat xyz.json.gz | myscript.py --file /dev/stdin
zcat xyz.json.gz | myscript.py --file /dev/fd/0
I am unable to see what I receive through the MQTT/mosquitto stream by means of echoing.
My code is as follows:
#!/bin/bash
`mosquitto_sub -d -t +/# >>mqtt_log.csv`
mqtt_stream_variable=`sed '$!d' mqtt_log.csv`
echo "$mqtt_stream_variable"
First line subscribes to the mqtt stream and appends the output to the mqtt_log.csv file. Then I sed '$!d' mqtt_log.csv so I get the last lines value assigned to the mqtt_stream variable, I later echo this.
When I execute this - I don't see any echoing I was curious to know how I could do this? When I cat mqtt_log.csv there are things in there. So the mosquitto_sub -d -t +/# >>mqtt_log.csv part is working. It's just the echoing that is being problematic.
Ideally after mqtt_stream=``sed '$!d' mqtt_log.csv I would like to play around with the values in mqtt_log.csv [as it's a csv string]. So by means of echoing I can see what the mqtt_stream_variable variable holds
The mosquitto_sub command will never return and sed will read the empty file before any messages are written to it and then exit.
How about something like this
#!/bin/bash
mosquitto_sub -d -t +/# | tee -a mqtt_log.csv | sed '$!d'
No need for all the sub shells and pipes will get you what you want.
The only other thing is why the need for both wild cards in the topic? +/# should be the same as just # (you will probably need to wrap the # in quotes on it's own)
I'm looking for a unix shell command to append the contents of a file as the parameters of another shell command. For example:
command << commandArguments.txt
xargs was built specifically for this:
cat commandArguments.txt | xargs mycommand
If you have multiple lines in the file, you can use xargs -L1 -P10 to run ten copies of your command at a time, in parallel.
xargs takes its standard in and formats it as positional parameters for a shell command. It was originally meant to deal with short command line limits, but it is useful for other purposes as well.
For example, within the last minute I've used it to connect to 10 servers in parallel and check their uptimes:
echo server{1..10} | tr ' ' '\n' | xargs -n 1 -P 50 -I ^ ssh ^ uptime
Some interesting aspects of this command pipeline:
The names of the servers to connect to were taken from the incoming pipe
The tr is needed to put each name on its own line. This is because xargs expects line-delimited input
The -n option controls how many incoming lines are used per command invocation. -n 1 says make a new ssh process for each incoming line.
By default, the parameters are appended to the end of the command. With -I, one can specify a token (^) that will be replaced with the argument instead.
The -P controls how many child processes to run concurrently, greatly widening the space of interesting possibilities..
command `cat commandArguments.txt`
Using backticks will use the result of the enclosed command as a literal in the outer command