How do you make a shell function that you can pipe into to make the output red? - bash

I want something where I can do {some command} | red and it'll output whatever {some command} would have outputted, but red. The best I've been able to do is
function red()
{
while read text
do
echo -e '\033[31m'$text'\033[0m'
done
}
But this removes all the indentation.
This seems like something that should be easy but I just can't seem to figure it out. I was trying to do it just for fun but now I just need to know, 'cause there has to be a simple way to do this that I'm missing.
EDIT:
I've also tried this in C like so:
#include <stdio.h>
#include <stdlib.h>
int main()
{
char* buffer = malloc(8);
int size = 8;
int index = 0;
while ((buffer[index++] = getchar()) != EOF)
{
if (index == size)
{
size *= 2;
buffer = realloc(buffer, size);
}
}
buffer[index] = 0;
printf("%s%s%s", "\\033[31m", buffer, "\\033[0m");
}
But the shell doesn't interpret the escape characters, and so doesn't make the output red.

It was only missing a couple things for it to work:
Firstly you want to neutralize the Internal Field Separator IFS environment variable, because it was the cause of your discarded indentation. This variable caused the read operation to eat leading tabs and spaces as field separator.
Next you wan to read -r. The -r flag prevent the interpretation of escaped characters and have a raw read instead.
Finally, you must not use echo -e. It is a horrible non-standard way of printing \ escaped characters. Use a printf instead.
Additionally, the loop on the read will not handle the last line of text if it does not contain an newline character \n. The shell read returns false in this case after reading the remaining text, and break out of the while loop without processing the last line not ending with a newline.
It is dealt with testing if the line contains a last line of text and print it without a newline for this special last case.
Now here it is all fixed and working as you intended to:
#!/usr/bin/env bash
function red() {
while IFS= read -r line; do
printf '\e[31m%s\e[0m\n' "$line"
done
[ -n "$line" ] && printf '\e[31m%s\e[0m' "$line"
}
red
EDIT:
As #Thsutton commented above, you can use cat in the red function to insert the ANSI CSI sequences before and after the whole text only, rather than for each line of text. It does not need a while loop, so will undoubtedly be more performant. Although note that the per-line method might give better results if the text stream itself contains ANSI CSI sequences changing the terminal colour.
red() {
printf '\e[31m'
cat
printf '\e[0m'
}

Writing the escape codes explicitly is IMO cumbersome. I would do something like
red() {
# Send stdin to stdout, but coloured in red, if the
# terminal supports it.
if tput setaf 196 2>/dev/null
then
cat
tput sgr0 # reset the attribute
else
# No support for changing the colour. Just display it
# using the default foreground colour
cat
fi
}
which also works on terminals which are not configured for color.
The magical number 196 comes from this table.

Related

Bash next line (\) in command causing spacing issues

I have the following function I want to call:
function print_something {
echo "test\
something\
yep"
}
when called it prints:
'test something yep'
I would like it to print:
'testsomethingyep'
I can get this to print if I do:
function print_something {
echo "test\
something\
yep"
}
but I don't think that looks great..
(Root problem is a curl command not an echo)
Consider assembling your pieces an array, and then combining them into a string later. Array definition syntax is far more forgiving -- not requiring backslashes at all, and also allowing comments on each line and between lines.
#!/usr/bin/env bash
# ^^^^- arrays and printf -v require bash, not sh
pieces=(
test # this also lets you use comments
something # and you don't need any backslashes at all!
# one can also have a full-line comment midway through your array
"space here" # plus if you want to add a literal space you can do it
)
printf -v oneword '%s' "${pieces[#]}"
echo "$oneword"
...properly emits:
testsomethingspace here
Here are three ideas:
#!/bin/bash
print_something() {
tr -d \\n <<- EOF
test
something
yep
EOF
echo
}
print_something2() {
echo "test"$(:
)"something"$(:
)"yep"
}
print_something3() {
tr -d \\t <<- EOF
test\
something\
yep
EOF
}
print_something
print_something2
print_something3
The first uses a <<- style heredoc to remove all of the leading indentation (that indentation must be hard-tabs for this to work, and coding styles that mandate the use of spaces for indentation render this solution unusable (this is one reason coding styles that mandate the use of spaces in shell scripts are IMO utterly useless)) and the extra tr to remove the newlines. (The additional echo is then needed to add the trailing newline.). The second uses the $(:) command substitution to discard all the intervening whitespace. The 3rd manually deletes all the hard tabs.

Is it possible to combine bash variable search and replace with substring?

I have this function:
#! /usr/bin/env bash
function underline() {
U="${1//?/${2:--}}"
echo -e "\n$1\n${U:0:${#1}}\n"
}
underline "$1" "^-v-"
will work as expected:
$ ./u.sh "This is all you're going to see"
This is all you're going to see
^-v-^-v-^-v-^-v-^-v-^-v-^-v-^-v
It does what you expect. Originally it assumed that the underline character was just that, 1 character.
For "fun", I extended it to move from a single character underline to a repeating string underline (why? ... well ... because I can I suppose ... almost zero practical value in this exercise!).
And so, being the "let's make this so difficult that you need to write 2 pages of documentation to explain what's going on" sort of guy, I was wondering if the function could be written as a single line. And I don't mean:
function underline() { U="${1//?/${2:--}}"; echo -e "\n$1\n${U:0:${#1}}\n"; }
I don't think you can combine bash's variable search/replace with substring which is what is required.
I'm aware that this will work happily in zsh:
#! /usr/bin/env zsh
function underline() {
echo -e "\n$1\n${${1//?/${2:--}}:0:${#1}}\n"
}
underline $1 "^-v-"
e.g.
$ ./u.sh "This is all you're going to see"
This is all you're going to see
^-v-^-v-^-v-^-v-^-v-^-v-^-v-^-v
but not capable of this in bash (it seems).
No, you cannot combine those in a single parameter expansion in bash. But this single printf (a builtin in bash) should do the trick:
underline() { printf '%s\n%.*s\n' "$1" ${#1} "${1//?/${2:--}}"; }
underline "This is all you're going to see" "^-v-"
outputs
This is all you're going to see
^-v-^-v-^-v-^-v-^-v-^-v-^-v-^-v

Sed replace regex with regex

Given the following file.txt:
this is line 1
# this is line 2
this is line 3
I would like to use sed to replace the lines with # at the beginning with \e[31m${lineContent}\e[0m. This will color that particular line. Additionally, I need the color \e[31m to be in a variable, color. (The desired output of this example would be having the second line colored). I have the following:
function colorLine() {
cat file.txt | sed ''/"$1"/s//"$(printf \e[31m $1 \e[0m)"/g''
}
colorLine "#.*"
The variable color is not included in what I have so far, as I am not sure how to go about that part.
The output of this is (with the second line being red):
this is line 1
#.*
this is line 3
It is apparently interpreting the replace string literally. My question is how do I use the matched line to generate the replace string?
I understand that I could do something much easier like appending \e[31m to the beginning of all lines that start with #, but it is important to use sed with the regexes.
colorLine() {
sed "/$1/s//"$'\e[31m&\e[0m/' file.txt
}
colorLine "#.*"
Multiple fixes, but it uses $1 to identify the pattern from the arguments to the function, and then uses ANSI-C quoting to encode the escape sequences — and fixes the color reset sequence which was (originally) missing the [ after the escape sequence. It also avoids the charge of "UUoC — Useless Use of cat".
The fixed file name is not exactly desirable, but fixing it is left as an exercise for the reader.
What if I needed \e[31m to be a variable, $color. How do I change the quoting?
I have a colour-diff script which contains (in Perl notation — I've translated it to Bash notation using ANSI C quoting as before):
reset=$'\e[0m'
black=$'\e[30;1m' # 30 = Black, 1 = bold
red=$'\e[31;1m' # 31 = Red, 1 = bold
green=$'\e[32;1m' # 32 = Green, 1 = bold
yellow=$'\e[33;1m' # 33 = Yellow, 1 = bold
blue=$'\e[34;1m' # 34 = Blue, 1 = bold
magenta=$'\e[35;1m' # 35 = Magenta, 1 = bold
cyan=$'\e[36;1m' # 36 = Cyan, 1 = bold
white=$'\e[37;1m' # 37 = White, 1 = bold
With those variables around, you can create your function as you wish:
colorLine() {
sed "/$1/s//$blue&$reset/“ file.txt
}
Where you set those variables depends on where you define your function. For myself, I'd probably make a script rather than a function, with full-scale argument parsing, and go from there. YMMV
Take a look at List of ANSI color escape sequences to get a more comprehensive list of colours (and other effects — including background and foreground colours) and the escape sequence used to generate it.
With GNU sed and Kubuntu 16.04.
foo="#.*"
sed 's/'"$foo"'/\x1b[31m&\x1b[0m/' file
I'd trick grep to do it for me this way:
function colorLine() {
GREP_COLORS="mt=31" grep --color=always --context=$(wc -l <file.txt) --no-filename "$1" file.txt
}
Split-out of the trick:
GREP_COLORS="mt=31": SGR substring for matching non-empty text in any matching line. Here will generate \e[31m red before matched string, and reset to default color after matched string.
--color=always: always colorise even in non interactive shell
context=$(wc -l <file.txt): output as much context lines as number of lines in the file (so all lines).
--no-filename: do not print the file name
An awk version
black='\033[30;1m'
red='\033[31;1m'
green='\033[32;1m'
yellow='\033[33;1m'
blue='\033[34;1m'
magenta='\033[35;1m'
cyan='\033[36;1m'
white='\033[37;1m'
color=$cyan
colorLine() { awk -v s="$1" -v c=$color '$0~s {$0=c$0"\033[0m"}1' file.txt; }
colorLine "#.*"
You can add file as a variable as vell:
color=$cyan
file="file.txt"
colorLine() { awk -v s="$1" -v c=$color '$0~s {$0=c$0"\033[0m"}1' $file; }
colorLine "#.*"
In awk \e is printed as \033
A more dynamic version:
colorLine() {
temp=$2;
col=${!temp};
awk -v s="$1" -v c=$col '$0~s {$0=c$0"\033[0m"}1' $3; }
colorLine "#.*" green file.txt
Then you have colorLine pattern color file

Removing control / special characters from log file

I have a log file captured by tclsh which captures all the backspace characters (ctrl-H, shows up as "^H") and color-setting sequences (eg. ^[[32m .... ^[[0m ). What is an efficient way to remove them?
^[...m
This one is easy since, I can just do "sed -i /^[.*m//g" to remove them
^H
Right now I have "sed -i s/.^H//", which "applies" a backspace, but I have to keep looping this until there are no more backspaces.
while [ logfile == `grep -l ^H logfile` ]; do sed -i s/.^H// logfile ; done;
"sed -i s/.^H//g" doesn't work because it would match consecutive backspaces. This process takes 11 mins for my log file with ~6k lines, which is too long.
Any better ways to remove the backspace?
You could always write a simple pipeline command to implement the backspace stripping, something like this:
#include <stdio.h>
#include <stdlib.h>
#define BUFFERSIZE 10240
int main(int argc, char* argv[])
{
int c ;
int buf[BUFFERSIZE] ;
int pos = 0 ;
while((c = getchar()) != EOF)
{
switch (c)
{
case '\b':
{
if (pos > 0)
pos-- ;
break ;
}
case '\n':
{
int i ;
for (i = 0; i < pos; ++i)
putchar(buf[i]) ;
putchar('\n') ;
pos = 0 ;
break ;
}
default:
{
buf[pos++] = c ;
break ;
}
}
}
return 0 ;
}
I've only given the code a minimal test and you may need to adjust the buffer sze depending on how big your lines our. It might be an idea to assert that pos is < BUFERSSIZE after pos++ just to be safe!
Alternatively you could maybe implement something similar with the Tcl code that captures the log file in the first place; but without knowing how that works it's a bit hard to say.
Your could try:
sed -i s/[^^H]^H//g
This might or might not work in one go, but should at least be faster than one at a time as you seem to be doing now.
Did you know that “sed” doesn't just do substitutions? The commands of a sed script have to be on separate lines though (or at least they do on the version of sed I've got on this machine).
sed -i bak 's/^[[^^]]*m//g
: again
s/[^^H]^H//g
t again' logfile
The : sets up a label (again in this case) and t branches to a label if any substitutions have been performed (since the start/last branch). Wrapping those round a suitable s gets the substitution applied until it can't any more.
Just to put it out here, I ended up doing this. It's not a pretty solution and not as flexible as Jackson's answer but does what I need in my particular case. I basically use the inner loop to generate the match string for sed.
# "Applies" up to 10 consecutive backspaces
for i in {10..1}; do
match=""
for j in `seq 1 $i`; do
match=".${match}^H"
done;
# Can't put quotes around s//g or else backspaces are evaluated
sed -i s/${match}//g ${file-to-process}
done;

Reading java .properties file from bash

I am thinking of using sed for reading .properties file, but was wondering if there is a smarter way to do that from bash script?
This would probably be the easiest way: grep + cut
# Usage: get_property FILE KEY
function get_property
{
grep "^$2=" "$1" | cut -d'=' -f2
}
The solutions mentioned above will work for the basics. I don't think they cover multi-line values though. Here is an awk program that will parse Java properties from stdin and produce shell environment variables to stdout:
BEGIN {
FS="=";
print "# BEGIN";
n="";
v="";
c=0; # Not a line continuation.
}
/^\#/ { # The line is a comment. Breaks line continuation.
c=0;
next;
}
/\\$/ && (c==0) && (NF>=2) { # Name value pair with a line continuation...
e=index($0,"=");
n=substr($0,1,e-1);
v=substr($0,e+1,length($0) - e - 1); # Trim off the backslash.
c=1; # Line continuation mode.
next;
}
/^[^\\]+\\$/ && (c==1) { # Line continuation. Accumulate the value.
v= "" v substr($0,1,length($0)-1);
next;
}
((c==1) || (NF>=2)) && !/^[^\\]+\\$/ { # End of line continuation, or a single line name/value pair
if (c==0) { # Single line name/value pair
e=index($0,"=");
n=substr($0,1,e-1);
v=substr($0,e+1,length($0) - e);
} else { # Line continuation mode - last line of the value.
c=0; # Turn off line continuation mode.
v= "" v $0;
}
# Make sure the name is a legal shell variable name
gsub(/[^A-Za-z0-9_]/,"_",n);
# Remove newlines from the value.
gsub(/[\n\r]/,"",v);
print n "=\"" v "\"";
n = "";
v = "";
}
END {
print "# END";
}
As you can see, multi-line values make things more complex. To see the values of the properties in shell, just source in the output:
cat myproperties.properties | awk -f readproperties.awk > temp.sh
source temp.sh
The variables will have '_' in the place of '.', so the property some.property will be some_property in shell.
If you have ANT properties files that have property interpolation (e.g. '${foo.bar}') then I recommend using Groovy with AntBuilder.
Here is my wiki page on this very topic.
I wrote a script to solve the problem and put it on my github.
See properties-parser
One option is to write a simple Java program to do it for you - then run the Java program in your script. That might seem silly if you're just reading properties from a single properties file. However, it becomes very useful when you're trying to get a configuration value from something like a Commons Configuration CompositeConfiguration backed by properties files. For a time, we went the route of implementing what we needed in our shell scripts to get the same behavior we were getting from CompositeConfiguration. Then we wisened up and realized we should just let CompositeConfiguration do the work for us! I don't expect this to be a popular answer, but hopefully you find it useful.
If you want to use sed to parse -any- .properties file, you may end up with a quite complex solution, since the format allows line breaks, unquoted strings, unicode, etc: http://en.wikipedia.org/wiki/.properties
One possible workaround would using java itself to preprocess the .properties file into something bash-friendly, then source it. E.g.:
.properties file:
line_a : "ABC"
line_b = Line\
With\
Breaks!
line_c = I'm unquoted :(
would be turned into:
line_a="ABC"
line_b=`echo -e "Line\nWith\nBreaks!"`
line_c="I'm unquoted :("
Of course, that would yield worse performance, but the implementation would be simpler/clearer.
In Perl:
while(<STDIN>) {
($prop,$val)=split(/[=: ]/, $_, 2);
# and do stuff for each prop/val
}
Not tested, and should be more tolerant of leading/trailing spaces, comments etc., but you get the idea. Whether you use Perl (or another language) over sed is really dependent upon what you want to do with the properties once you've parsed them out of the file.
Note that (as highlighted in the comments) Java properties files can have multiple forms of delimiters (although I've not seen anything used in practice other than colons). Hence the split uses a choice of characters to split upon.
Ultimately, you may be better off using the Config::Properties module in Perl, which is built to solve this specific problem.
I have some shell scripts that need to look up some .properties and use them as arguments to programs I didn't write. The heart of the script is a line like this:
dbUrlFile=$(grep database.url.file etc/zocalo.conf | sed -e "s/.*: //" -e "s/#.*//")
Effectively, that's grep for the key and filter out the stuff before the colon and after any hash.
if you want to use "shell", the best tool to parse files and have proper programming control is (g)awk. Use sed only simple substitution.
I have sometimes just sourced the properties file into the bash script. This will lead to environment variables being set in the script with the names and contents from the file. Maybe that is enough for you, too. If you have to do some "real" parsing, this is not the way to go, of course.
Hmm, I just run into the same problem today. This is poor man's solution, admittedly more straightforward than clever;)
decl=`ruby -ne 'puts chomp.sub(/=(.*)/,%q{="\1";}).gsub(".","_")' my.properties`
eval $decl
then, a property 'my.java.prop' can be accessed as $my_java_prop.
This can be done with sed or whatever, but I finally went with ruby for its 'irb' which was handy for experimenting.
It's quite limited (dots should be replaced only before '=',no comment handling), but could be a starting point.
#Daniel, I tried to source it, but Bash didn't like dots in variable names.
I have had some success with
PROPERTIES_FILE=project.properties
function source_property {
local name=$1
eval "$name=\"$(sed -n '/^'"$name"'=/,/^[A-Z]\+_*[A-Z]*=/p' $PROPERTIES_FILE|sed -e 's/^'"$name"'=//g' -e 's/"/\\"/g'|head -n -1)\""
}
source_property 'SOME_PROPERTY'
This is a solution that properly parses quotes and terminates at a space when not given quotes. It is safe: no eval is used.
I use this code in my .bashrc and .zshrc for importing variables from shell scripts:
# Usage: _getvar VARIABLE_NAME [sourcefile...]
# Echos the value that would be assigned to VARIABLE_NAME
_getvar() {
local VAR="$1"
shift
awk -v Q="'" -v QQ='"' -v VAR="$VAR" '
function loc(text) { return index($0, text) }
function unquote(d) { $0 = substr($0, eq+2) d; print substr($0, 1, loc(d)-1) }
{ sub(/^[ \t]+/, ""); eq = loc("=") }
substr($0, 1, eq-1) != VAR { next } # assignment is not for VAR: skip
loc("=" QQ) == eq { unquote(QQ); exit }
loc("=" Q) == eq { unquote( Q); exit }
{ print substr($1, eq + 1); exit }
' "$#"
}
This saves the desired variable name and then shifts the argument array so the rest can be passed as files to awk.
Because it's so hard to call shell variables and refer to quote characters inside awk, I'm defining them as awk variables on the command line. Q is a single quote (apostrophe) character, QQ is a double quote, and VAR is that first argument we saved earlier.
For further convenience, there are two helper functions. The first returns the location of the given text in the current line, and the second prints the content between the first two quotes in the line using quote character d (for "delimiter"). There's a stray d concatenated to the first substr as a safety against multi-line strings (see "Caveats" below).
While I wrote the code for POSIX shell syntax parsing, that appears to only differ from your format by whether there is white space around the asignment. You can add that functionality to the above code by adding sub(/[ \t]*=[ \t]*/, "="); before the sub(…) on awk's line 4 (note: line 1 is blank).
The fourth line strips off leading white space and saves the location of the first equals sign. Please verify that your awk supports \t as tab, this is not guaranteed on ancient UNIX systems.
The substr line compares the text before the equals sign to VAR. If that doesn't match, the line is assigning a different variable, so we skip it and move to the next line.
Now we know we've got the requested variable assignment, so it's just a matter of unraveling the quotes. We do this by searching for the first location of =" (line 6) or =' (line 7) or no quotes (line 8). Each of those lines prints the assigned value.
Caveats: If there is an escaped quote character, we'll return a value truncated to it. Detecting this is a bit nontrivial and I decided not to implement it. There's also a problem of multi-line quotes, which get truncated at the first line break (this is the purpose of the "stray d" mentioned above). Most solutions on this page suffer from these issues.
In order to let Java do the tricky parsing, here's a solution using jrunscript to print the keys and values in a bash read-friendy (key, tab character, value, null character) way:
#!/usr/bin/env bash
jrunscript -e '
p = new java.util.Properties();
p.load(java.lang.System.in);
p.forEach(function(k,v) { out.format("%s\t%s\000", k, v); });
' < /tmp/test.properties \
| while IFS=$'\t' read -d $'\0' -r key value; do
key=${key//./_}
printf -v "$key" %s "$value"
printf '=> %s = "%s"\n' "$key" "$value"
done
I found printf -v in this answer by #david-foerster.
To quote jrunscript: Warning: Nashorn engine is planned to be removed from a future JDK release

Resources