I am unable to see what I receive through the MQTT/mosquitto stream by means of echoing.
My code is as follows:
#!/bin/bash
`mosquitto_sub -d -t +/# >>mqtt_log.csv`
mqtt_stream_variable=`sed '$!d' mqtt_log.csv`
echo "$mqtt_stream_variable"
First line subscribes to the mqtt stream and appends the output to the mqtt_log.csv file. Then I sed '$!d' mqtt_log.csv so I get the last lines value assigned to the mqtt_stream variable, I later echo this.
When I execute this - I don't see any echoing I was curious to know how I could do this? When I cat mqtt_log.csv there are things in there. So the mosquitto_sub -d -t +/# >>mqtt_log.csv part is working. It's just the echoing that is being problematic.
Ideally after mqtt_stream=``sed '$!d' mqtt_log.csv I would like to play around with the values in mqtt_log.csv [as it's a csv string]. So by means of echoing I can see what the mqtt_stream_variable variable holds
The mosquitto_sub command will never return and sed will read the empty file before any messages are written to it and then exit.
How about something like this
#!/bin/bash
mosquitto_sub -d -t +/# | tee -a mqtt_log.csv | sed '$!d'
No need for all the sub shells and pipes will get you what you want.
The only other thing is why the need for both wild cards in the topic? +/# should be the same as just # (you will probably need to wrap the # in quotes on it's own)
Related
I'm trying to temporarily disable dhcp on all connections in a computer using bash, so I need the process to be reversible. My approach is to comment out lines that contain BOOTPROTO=dhcp, and then insert a line below it with BOOTPROTO=none. I'm not sure of the correct syntax to make sed understand the line number stored in the $insertLine variable.
fileList=$(ls /etc/sysconfig/network-scripts | grep ^ifcfg)
path="/etc/sysconfig/network-scripts/"
for file in $fileList
do
echo "looking for dhcp entry in $file"
if [ $(cat $path$file | grep ^BOOTPROTO=dhcp) ]; then
echo "disabling dhcp in $file"
editLine=$(grep -n ^BOOTPROTO=dhcp /$path$file | cut -d : -f 1 )
#comment out the original dhcp value
sed -i "s/BOOTPROTO=dhcp/#BOOTPROTO=dhcp/g" $path$file
#insert a line below it with value of none.
((insertLine=$editLine+1))
sed "$($insertLine)iBOOTPROTO=none" $path$file
fi
done
Any help using sed or other stream editor greatly appreciated. I'm using RHEL 6.
The sed editor should be able to do the job, without having to to be combine bash, grep, cat, etc. Easier to test, and more reliable.
The whole scripts can be simplified to the below. It performs all operations (substitution and the insert) with a single pass using multiple sed scriptlets.
#! /bin/sh
for file in $(grep -l "^BOOTPROTO=dhcp" /etc/sysconfig/network-scripts/ifcfg*) ; do
sed -i -e "s/BOOTPROTO=dhcp/#BOOTPROTO=dhcp/g" -e "/BOOTPROTO=dhcp/i BOOTPROTO=none" $file
done
As side note consider NOT using path as variable to avoid possible confusion with the 'PATH` environment variable.
Writing it up, your attempt with the following fails:
sed "$($insertLine)iBOOTPROTO=none" $path$file
because:
$($insertLine) encloses $insertLIne in a command substitution which when $insertLIne is evaluated it returns a number which is not a command generating an error.
your call to sed does not include the -i option to edit the file $path$file in place.
You can correct the issues with:
sed -i "${insertLine}i BOOTPROTO=none" $path$file
Which is just sed - i (edit in place) and Ni where N is the number of the line to insert followed by the content to insert and finally what file to insert it in. You add ${..} to insertLine to protect the variable name from the i that follows and then the expression is double-quoted to allow variable expansion.
Let me know if you have any further questions.
(and see dash-o's answer for refactoring the whole thing to simply use sed to make the change without spawning 10 other subshells)
I have a Python script which needs a --file xyz.json argument.
Thing is, my JSON is immense, hence it is Gzipped. Of course I could un-gzip it and then run the script, but that seems wasteful. Is there a clever way I can get this to work while doing a zcat xyz.json.gz | myscript.py --file ?????. I don't want to go into modifying myscript.py to read stdin instead of a file unless there's no way to get this done otherwise.
Thanks!
Try:
myscript.py --file <(zcat xyz.json.gz)
A file descriptor containing the pipe is returned. Provided that the script just reads the file, and does not search forward and backward, this should work.
The <( ... ) is called process substitution.
As an elaboration on what happens:
% awk 'BEGIN{print "filename:", ARGV[1]};1' <(echo first; sleep 1; echo second)
filename: /proc/self/fd/11
first
second
The second gets printed after a delay. So: Awk gets the filename /proc/self/fd/11, and starts to process it. It will immediately see the first line, and print it out. Then, after the sleep, it will see the second line, and print it as well.
You can use /dev/stdin or (equivalently) /dev/fd/0:
zcat xyz.json.gz | myscript.py --file /dev/stdin
zcat xyz.json.gz | myscript.py --file /dev/fd/0
I'm looking for a way to pipe multiple log files on multiple remote servers, and then pipe the result to another program.
Right now I'm using multitail, but it does not exactly do what I need, or maybe I'm doing something wrong!
I would like to be able to send the merge of all log files, to another program. For example jq. Right now if I do:
multitail --mergeall -l 'ssh server1 "tail -f /path/to/log"' -l 'ssh server2 "tail -f /path/to/log"' -l 'ssh server3 "tail -f /path/to/log"' | jq .
for instance, I get this:
parse error: Invalid numeric literal at line 1, column 2
But more generally, I would like to give the output of this to another program I use to parse and display logs :-)
Thanks everybody!
One way to accomplish this feat would be to pipe all your outputs together into a named pipe and then deal with the output from that named pipe.
First, create your named pipe: $ mknod MYFIFO p
For each location you want to consolidate lines from, $ tail -f logfile > MYFIFO (note, the tail -f can be run through an ssh session).
Then have another process take the data out of the named pipe and handle it appropriately. An ugly solution could be:
$ tail -f MYFIFO | jq
Season to taste.
Here is the user email for example...
mburkhar#imap1.tech.com
I want to cut out the mburkhar and also remove the imap. in the email to look like this in a new file. I have been working on this for a while but there are so many different commands I am confused as to what I should actually be using for this problem.
mburkhar mburkhar#tech.com
s=mburkhar#imap1.tech.com
s_name=${s%%#*}
s_adjusted=${s//#imap1./#}
echo "Name is $s_name; adjusted email address is $s_adjusted"
When run, this will have the output:
Name is mburkhar; adjusted email address is mburkhar#tech.com
...of course, for your originally requested output, you could
echo "$s_name $s_adjusted"
...or, to implement this as a one-liner, assuming again your original value in the variable $s:
echo "${s%%#*} ${s//#imap1./#}"
These are parameter expansion operations, performed internally to bash, and thus more efficient than using any external process such as sed.
I would use a quick sed script:
sed 's/\([^#]*\)#imap[0-2]\.\(.*\)$/\1 \1#\2/'
You can use the sed command to do this:
echo "mburkhar#imap1.tech.com" | sed -e 's/^\(.*\)#imap[^\.]*.\(.*\)/\1 \1#\2/'
mburkhar mburkhar#tech.com
This will capture the part before the # ignore any string starting with imap just after the # and capture the end of the address.
If no imap is found the output will be like this:
echo "mburkhar#tech.com" | sed -e 's/^\(.*\)#imap[^\.]*.\(.*\)/\1 \1#\2/'
mburkhar#tech.com
Sorry for the title, i couldn't find proper words to explain my problem.
Here's the code:
wlan_c=$(iwconfig | sed '/^\(w.*$\)/!d;s/ .*//' > ./wifi_iface)
wlan=$(<./wifi_iface)
echo "$wlan"
I get the following output:
lo no wireless extensions.
enp4s0 no wireless extensions.
wlp2s0
The last line is the result of execution the echo "$wlan".
The previous lines coming from the iwconfig, those that are not getting formatted by sed.
And the file ./wifi_iface also has the info i need.
Everything works as intended.
So i really want to get rid of that unwanted output before the wlp2s0 line.
How do i manage to do this?
That output must be going to stderr rather than stdout. Redirect it to /dev/null
iwconfig 2>/dev/null | sed '/^\(w.*$\)/!d;s/ .*//' > ./wifi_iface
There's no need to assign this to wlan_c. Since you're writing to the file, nothing will be written to stdout, so the assignment will always be empty.