I'm currently using the following to capture everything that goes to the terminal and throw it into a log file
exec 4<&1 5<&2 1>&2>&>(tee -a $LOG_FILE)
however, I don't want color escape codes/clutter going into the log file. so i have something like this that sorta works
exec 4<&1 5<&2 1>&2>&>(
while read -u 0; do
#to terminal
echo "$REPLY"
#to log file (color removed)
echo "$REPLY" | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' >> $LOG_FILE
done
unset REPLY #tidy
)
except read waits for carriage return which isn't ideal for some portions of the script (e.g. echo -n "..." or printf without \n).
Follow-up to Jonathan Leffler's answer:
Given the example script test.sh:
#!/bin/bash
LOG_FILE="./test.log"
echo -n >$LOG_FILE
exec 4<&1 5<&2 1>&2>&>(tee -a >(sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' > $LOG_FILE))
##### ##### #####
# Main
echo "starting execution"
printf "\n\n"
echo "color test:"
echo -e "\033[0;31mhello \033[0;32mworld\033[0m!"
printf "\n\n"
echo -e "\033[0;36mEnvironment:\033[0m\n foo: cat\n bar: dog\n your wife: hot\n fix: A/C"
echo -n "Before we get started. Is the above information correct? "
read YES
echo -e "\n[READ] $YES" >> $LOG_FILE
YES=$(echo "$YES" | sed 's/^\s*//;s/\s*$//')
test ! "$(echo "$YES" | grep -iE '^y(es)?$')" && echo -e "\nExiting... :(" && exit
printf "\n\n"
#...some hundreds of lines of code later...
echo "Done!"
##### ##### #####
# End
exec 1<&4 4>&- 2<&5 5>&-
echo "Log File: $LOG_FILE"
The output to the terminal is as expected and there is no color escape codes/clutter in the log file as desired. However upon examining test.log, I do not see the [READ] ... (see line 21 of test.sh).
The log file [of my actual bash script] contains the line Log File: ... at the end of it even after closing the 4 and 5 fds. I was able to resolve the issue by putting a sleep 1 before the second exec - I assume there's a race condition or fd shenanigans to blame for it. Unfortunately for you guys, I am not able to reproduce this issue with test.sh but I'd be interested in any speculation anyone may have.
Consider using the pee program discussed in Is it possible to distribute stdin over parallel processes. It would allow you to send the log data through your sed script, while continuing to send the colours to the actual output.
One major advantage of this is that it would remove the 'execute sed once per line of log output'; that is really diabolical for performance (in terms of number of processes executed, if nothing else).
I know it's not a perfect solution, but cat -v will make non visible chars like \x1B to be converted into visible form like ^[[1;34m. The output will be messy, but it will be ascii text at least.
I use to do stuff like this by setting TERM=dumb before running my command. That pretty much removed any control characters except for tab, CR, and LF. I have no idea if this works for your situation, but it's worth a try. The problem is that you won't see color encodings on your terminal either since it's a dumb terminal.
You can also try either vis or cat (especially the -v parameter) and see if these do something for you. You'd simply put them in your pipeline like this:
exec 4<&1 5<&2 1>&2>&>(tee -a | cat -v | $LOG_FILE)
By the way, almost all terminal programs have an option to capture the input, and most clean it up for you. What platform are you on, and what type of terminal program are you using?
You could attempt to use the -n option for read. It reads in n characters instead of waiting for a new line. You could set it to one. This would increase the number of iteration the code runs, but it would not wait for newlines.
From the man:
-n NCHARS read returns after reading NCHARS characters rather than waiting for a complete line of input.
Note: I have not tested this
You can use ANSIFilter to strip or transform console output with ANSI escape sequences.
See http://www.andre-simon.de/zip/download.html#ansifilter
Might not screen -L or the script commands be viable options instead of this exec loop?
Related
EDIT: Corrected process/thread terminology
My shell script has a foreground process that reads user input and a background process that prints messages. I would like to print these messages on the line above the input prompt rather than interrupting the input. Here's a canned example:
sleep 5 && echo -e "\nINFO: Helpful Status Update!" &
echo -n "> "
read input
When I execute it and type "input" a bunch of times, I get something like this:
> input input input inp
INFO: Helpful Status Update!
ut input
But I would like to see something like this:
INFO: Helpful Status Update!
> input input input input input
The solution need not be portable (I'm using bash on linux), though I would like to avoid ncurses if possible.
EDIT: According to #Nick, previous lines are inaccessible for historical reasons. However, my situation only requires modifying the current line. Here's a proof of concept:
# Make named pipe
mkfifo pipe
# Spawn background process
while true; do
sleep 2
echo -en "\033[1K\rINFO: Helpful Status Update!\n> `cat pipe`"
done &
# Start foreground user input
echo -n "> "
pid=-1
collected=""
IFS=""
while true; do
read -n 1 c
collected="$collected$c"
# Named pipes block writes, so must do background process
echo -n "$collected" >> pipe &
# Kill last loop's (potentially) still-blocking pipe write
if kill -0 $pid &> /dev/null; then
kill $pid &> /dev/null
fi
pid=$!
done
This produces mostly the correct behavior, but lacks CLI niceties like backspace and arrow navigation. These could be hacked in, but I'm still having trouble believing that a standard approach hasn't already been developed.
The original ANSI codes still work in bash terminal on Linux (and MacOS), so you can use \033[F where \033 is the ESCape character. You can generate this in bash terminal by control-V followed by the ESCape character. You should see ^[ appear. Then type [F. If you test the following script:
echo "original line 1"
echo "^[[Fupdated line 1"
echo "line 2"
echo "line 3"
You should see output:
updated line 1
line 2
line 3
EDIT:
I forgot to add that using this in your script will cause the cursor to return to the beginning of the line, so further input will overwrite what you have typed already. You could use control-R on the keyboard to cause bash to re-type the current line and return the cursor to the end of the line.
Is the output of a Bash command stored in any register? E.g. something similar to $? capturing the output instead of the exit status.
I could assign the output to a variable with:
output=$(command)
but that's more typing...
You can use $(!!)
to recompute (not re-use) the output of the last command.
The !! on its own executes the last command.
$ echo pierre
pierre
$ echo my name is $(!!)
echo my name is $(echo pierre)
my name is pierre
The answer is no. Bash doesn't allocate any output to any parameter or any block on its memory. Also, you are only allowed to access Bash by its allowed interface operations. Bash's private data is not accessible unless you hack it.
Very Simple Solution
One that I've used for years.
Script (add to your .bashrc or .bash_profile)
# capture the output of a command so it can be retrieved with ret
cap () { tee /tmp/capture.out; }
# return the output of the most recent command that was captured by cap
ret () { cat /tmp/capture.out; }
Usage
$ find . -name 'filename' | cap
/path/to/filename
$ ret
/path/to/filename
I tend to add | cap to the end of all of my commands. This way when I find I want to do text processing on the output of a slow running command I can always retrieve it with ret.
If you are on mac, and don't mind storing your output in the clipboard instead of writing to a variable, you can use pbcopy and pbpaste as a workaround.
For example, instead of doing this to find a file and diff its contents with another file:
$ find app -name 'one.php'
/var/bar/app/one.php
$ diff /var/bar/app/one.php /var/bar/two.php
You could do this:
$ find app -name 'one.php' | pbcopy
$ diff $(pbpaste) /var/bar/two.php
The string /var/bar/app/one.php is in the clipboard when you run the first command.
By the way, pb in pbcopy and pbpaste stand for pasteboard, a synonym for clipboard.
One way of doing that is by using trap DEBUG:
f() { bash -c "$BASH_COMMAND" >& /tmp/out.log; }
trap 'f' DEBUG
Now most recently executed command's stdout and stderr will be available in /tmp/out.log
Only downside is that it will execute a command twice: once to redirect output and error to /tmp/out.log and once normally. Probably there is some way to prevent this behavior as well.
Inspired by anubhava's answer, which I think is not actually acceptable as it runs each command twice.
save_output() {
exec 1>&3
{ [ -f /tmp/current ] && mv /tmp/current /tmp/last; }
exec > >(tee /tmp/current)
}
exec 3>&1
trap save_output DEBUG
This way the output of last command is in /tmp/last and the command is not called twice.
Yeah, why type extra lines each time; agreed.
You can redirect the returned from a command to input by pipeline, but redirecting printed output to input (1>&0) is nope, at least not for multiple line outputs.
Also you won't want to write a function again and again in each file for the same. So let's try something else.
A simple workaround would be to use printf function to store values in a variable.
printf -v myoutput "`cmd`"
such as
printf -v var "`echo ok;
echo fine;
echo thankyou`"
echo "$var" # don't forget the backquotes and quotes in either command.
Another customizable general solution (I myself use) for running the desired command only once and getting multi-line printed output of the command in an array variable line-by-line.
If you are not exporting the files anywhere and intend to use it locally only, you can have Terminal set-up the function declaration. You have to add the function in ~/.bashrc file or in ~/.profile file. In second case, you need to enable Run command as login shell from Edit>Preferences>yourProfile>Command.
Make a simple function, say:
get_prev() # preferably pass the commands in quotes. Single commands might still work without.
{
# option 1: create an executable with the command(s) and run it
#echo $* > /tmp/exe
#bash /tmp/exe > /tmp/out
# option 2: if your command is single command (no-pipe, no semi-colons), still it may not run correct in some exceptions.
#echo `"$*"` > /tmp/out
# option 3: (I actually used below)
eval "$*" > /tmp/out # or simply "$*" > /tmp/out
# return the command(s) outputs line by line
IFS=$(echo -en "\n\b")
arr=()
exec 3</tmp/out
while read -u 3 -r line
do
arr+=($line)
echo $line
done
exec 3<&-
}
So what we did in option 1 was print the whole command to a temporary file /tmp/exe and run it and save the output to another file /tmp/out and then read the contents of the /tmp/out file line-by-line to an array.
Similar in options 2 and 3, except that the commands were exectuted as such, without writing to an executable to be run.
In main script:
#run your command:
cmd="echo hey ya; echo hey hi; printf `expr 10 + 10`'\n' ; printf $((10 + 20))'\n'"
get_prev $cmd
#or simply
get_prev "echo hey ya; echo hey hi; printf `expr 10 + 10`'\n' ; printf $((10 + 20))'\n'"
Now, bash saves the variable even outside previous scope, so the arr variable created in get_prev function is accessible even outside the function in the main script:
#get previous command outputs in arr
for((i=0; i<${#arr[#]}; i++))
do
echo ${arr[i]}
done
#if you're sure that your output won't have escape sequences you bother about, you may simply print the array
printf "${arr[*]}\n"
Edit:
I use the following code in my implementation:
get_prev()
{
usage()
{
echo "Usage: alphabet [ -h | --help ]
[ -s | --sep SEP ]
[ -v | --var VAR ] \"command\""
}
ARGS=$(getopt -a -n alphabet -o hs:v: --long help,sep:,var: -- "$#")
if [ $? -ne 0 ]; then usage; return 2; fi
eval set -- $ARGS
local var="arr"
IFS=$(echo -en '\n\b')
for arg in $*
do
case $arg in
-h|--help)
usage
echo " -h, --help : opens this help"
echo " -s, --sep : specify the separator, newline by default"
echo " -v, --var : variable name to put result into, arr by default"
echo " command : command to execute. Enclose in quotes if multiple lines or pipelines are used."
shift
return 0
;;
-s|--sep)
shift
IFS=$(echo -en $1)
shift
;;
-v|--var)
shift
var=$1
shift
;;
-|--)
shift
;;
*)
cmd=$option
;;
esac
done
if [ ${#} -eq 0 ]; then usage; return 1; fi
ERROR=$( { eval "$*" > /tmp/out; } 2>&1 )
if [ $ERROR ]; then echo $ERROR; return 1; fi
local a=()
exec 3</tmp/out
while read -u 3 -r line
do
a+=($line)
done
exec 3<&-
eval $var=\(\${a[#]}\)
print_arr $var # comment this to suppress output
}
print()
{
eval echo \${$1[#]}
}
print_arr()
{
eval printf "%s\\\n" "\${$1[#]}"
}
Ive been using this to print space-separated outputs of multiple/pipelined/both commands as line separated:
get_prev -s " " -v myarr "cmd1 | cmd2; cmd3 | cmd4"
For example:
get_prev -s ' ' -v myarr whereis python # or "whereis python"
# can also be achieved (in this case) by
whereis python | tr ' ' '\n'
Now tr command is useful at other places as well, such as
echo $PATH | tr ':' '\n'
But for multiple/piped commands... you know now. :)
-Himanshu
Like konsolebox said, you'd have to hack into bash itself. Here is a quite good example on how one might achieve this. The stderred repository (actually meant for coloring stdout) gives instructions on how to build it.
I gave it a try: Defining some new file descriptor inside .bashrc like
exec 41>/tmp/my_console_log
(number is arbitrary) and modify stderred.c accordingly so that content also gets written to fd 41. It kind of worked, but contains loads of NUL bytes, weird formattings and is basically binary data, not readable. Maybe someone with good understandings of C could try that out.
If so, everything needed to get the last printed line is tail -n 1 [logfile].
Not sure exactly what you're needing this for, so this answer may not be relevant. You can always save the output of a command: netstat >> output.txt, but I don't think that's what you're looking for.
There are of course programming options though; you could simply get a program to read the text file above after that command is run and associate it with a variable, and in Ruby, my language of choice, you can create a variable out of command output using 'backticks':
output = `ls` #(this is a comment) create variable out of command
if output.include? "Downloads" #if statement to see if command includes 'Downloads' folder
print "there appears to be a folder named downloads in this directory."
else
print "there is no directory called downloads in this file."
end
Stick this in a .rb file and run it: ruby file.rb and it will create a variable out of the command and allow you to manipulate it.
If you don't want to recompute the previous command you can create a macro that scans the current terminal buffer, tries to guess the -supposed- output of the last command, copies it to the clipboard and finally types it to the terminal.
It can be used for simple commands that return a single line of output (tested on Ubuntu 18.04 with gnome-terminal).
Install the following tools: xdootool, xclip , ruby
In gnome-terminal go to Preferences -> Shortcuts -> Select all and set it to Ctrl+shift+a.
Create the following ruby script:
cat >${HOME}/parse.rb <<EOF
#!/usr/bin/ruby
stdin = STDIN.read
d = stdin.split(/\n/)
e = d.reverse
f = e.drop_while { |item| item == "" }
g = f.drop_while { |item| item.start_with? "${USER}#" }
h = g[0]
print h
EOF
In the keyboard settings add the following keyboard shortcut:
bash -c '/bin/sleep 0.3 ; xdotool key ctrl+shift+a ; xdotool key ctrl+shift+c ; ( (xclip -out | ${HOME}/parse.rb ) > /tmp/clipboard ) ; (cat /tmp/clipboard | xclip -sel clip ) ; xdotool key ctrl+shift+v '
The above shortcut:
copies the current terminal buffer to the clipboard
extracts the output of the last command (only one line)
types it into the current terminal
I have an idea that I don't have time to try to implement immediately.
But what if you do something like the following:
$ MY_HISTORY_FILE = `get_temp_filename`
$ MY_HISTORY_FILE=$MY_HISTORY_FILE bash -i 2>&1 | tee $MY_HISTORY_FILE
$ some_command
$ cat $MY_HISTORY_FILE
$ # ^You'll want to filter that down in practice!
There might be issues with IO buffering. Also the file might get too huge. One would have to come up with a solution to these problems.
I think using script command might help. Something like,
script -c bash -qf fifo_pid
Using bash features to set after parsing.
Demo for non-interactive commands only: http://asciinema.org/a/395092
For also supporting interactive commands, you'd have to hack the script binary from util-linux to ignore any screen-redrawing console codes, and run it from bashrc to save your login session's output to a file.
You can use -exec to run a command on the output of a command. So it will be a reuse of the output as an example given with a find command below:
find . -name anything.out -exec rm {} \;
you are saying here -> find a file called anything.out in the current folder, if found, remove it. If it is not found, the remaining after -exec will be skipped.
I have some script that produces output with colors and I need to remove the ANSI codes.
#!/bin/bash
exec > >(tee log) # redirect the output to a file but keep it on stdout
exec 2>&1
./somescript
The output is (in log file):
java (pid 12321) is running...#[60G[#[0;32m OK #[0;39m]
I didn't know how to put the ESC character here, so I put # in its place.
I changed the script into:
#!/bin/bash
exec > >(tee log) # redirect the output to a file but keep it on stdout
exec 2>&1
./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"
But now it gives me (in log file):
java (pid 12321) is running...#[60G[ OK ]
How can I also remove this '#[60G?
Maybe there is a way to completely disable coloring for the entire script?
According to Wikipedia, the [m|K] in the sed command you're using is specifically designed to handle m (the color command) and K (the "erase part of line" command). Your script is trying to set absolute cursor position to 60 (^[[60G) to get all the OKs in a line, which your sed line doesn't cover.
(Properly, [m|K] should probably be (m|K) or [mK], because you're not trying to match a pipe character. But that's not important right now.)
If you switch that final match in your command to [mGK] or (m|G|K), you should be able to catch that extra control sequence.
./somescript | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2};?)?)?[mGK]//g"
IMHO, most of these answers try too hard to restrict what is inside the escape code. As a result, they end up missing common codes like [38;5;60m (foreground ANSI color 60 from 256-color mode).
They also require the -r option which enables GNU extensions. These are not required; they just make the regex read better.
Here is a simpler answer that handles the 256-color escapes and works on systems with non-GNU sed:
./somescript | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g'
This will catch anything that starts with [, has any number of decimals and semicolons, and ends with a letter. This should catch any of the common ANSI escape sequences.
For funsies, here's a larger and more general (but minimally tested) solution for all conceivable ANSI escape sequences:
./somescript | sed 's/\x1B[#A-Z\\\]^_]\|\x1B\[[0-9:;<=>?]*[-!"#$%&'"'"'()*+,.\/]*[][\\#A-Z^_`a-z{|}~]//g'
(and if you have #edi9999's SI problem, add | sed "s/\x0f//g" to the end; this works for any control char by replacing 0f with the hex of the undesired char)
I couldn't get decent results from any of the other answers, but the following worked for me:
somescript | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g"
If I only removed the control char "^[", it left the rest of the color data, e.g., "33m". Including the color code and "m" did the trick. I'm puzzled with s/\x1B//g doesn't work because \x1B[31m certainly works with echo.
I came across ansi2txt tool from colorized-logs package in Debian. The tool drops ANSI control codes from STDIN.
Usage example:
./somescript | ansi2txt
Source code http://github.com/kilobyte/colorized-logs
For Mac OSX or BSD use
./somescript | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'
The regular expression below will miss some ANSI Escape Codes sequences, as well as 3 digit colors. Example and Fix on regex101.com.
Use this instead:
./somescript | sed -r 's/\x1B\[(;?[0-9]{1,3})+[mGK]//g'
I also had the problem that sometimes, the SI character appeared.
It happened for example with this input : echo "$(tput setaf 1)foo$(tput sgr0) bar"
Here's a way to also strip the SI character (shift in) (0x0f)
./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | sed "s/\x0f//g"
Much simpler function in pure Bash to filter-out common ANSI codes from a text stream:
# Strips common ANSI codes from a text stream
shopt -s extglob # Enable Bash Extended Globbing expressions
ansi_filter() {
local line
local IFS=
while read -r line || [[ "$line" ]]; do
printf '%s\n' "${line//$'\e'[\[(]*([0-9;])[#-n]/}"
done
}
See:
linuxjournal.com: Extended Globbing
gnu.org: Bash Parameter Expansion
I had a similar problem. All solutions I found did work well for the color codes but did not remove the characters added by "$(tput sgr0)" (resetting attributes).
Taking, for example, the solution in the comment by davemyron the length of the resulting string in the example below is 9, not 6:
#!/usr/bin/env bash
string="$(tput setaf 9)foobar$(tput sgr0)"
string_sed="$( sed -r "s/\x1B\[[0-9;]*[JKmsu]//g" <<< "${string}" )"
echo ${#string_sed}
In order to work properly, the regex had to be extend to also match the sequence added by sgr0 ("\E(B"):
string_sed="$( sed -r "s/\x1B(\[[0-9;]*[JKmsu]|\(B)//g" <<< "${string}" )"
Not sure what's in ./somescript but if escape sequences are not hardcoded you can set the terminal type to avoid them
TERM=dumb ./somescript
For example, if you try
TERM=dumb tput sgr0 | xxd
you'll see it produces no output while
tput sgr0 | xxd
00000000: 1b28 421b 5b6d .(B.[m
does (for xterm-256color).
Hmm, not sure if this will work for you, but 'tr' will 'strip' (delete) control codes - try:
./somescript | tr -d '[:cntrl:]'
There's also a dedicated tool to handle ANSI escape sequences: ansifilter. Use the default --text output format to strip all ANSI escape sequences (note: not just coloring).
ref: https://stackoverflow.com/a/6534712
Here's a pure Bash solution.
Save as strip-escape-codes.sh, make executable and then run <command-producing-colorful-output> | ./strip-escape-codes.sh.
Note that this strips all ANSI escape codes/sequences. If you want to strip colors only, replace [a-zA-Z] with "m".
Bash >= 4.0:
#!/usr/bin/env bash
# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
local _input="$1" _i _char _escape=0
local -n _output="$2"; _output=""
for (( _i=0; _i < ${#_input}; _i++ )); do
_char="${_input:_i:1}"
if (( ${_escape} == 1 )); then
if [[ "${_char}" == [a-zA-Z] ]]; then
_escape=0
fi
continue
fi
if [[ "${_char}" == $'\e' ]]; then
_escape=1
continue
fi
_output+="${_char}"
done
}
while read -r line; do
strip_escape_codes "${line}" line_stripped
echo "${line_stripped}"
done
Bash < 4.0:
#!/usr/bin/env bash
# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
local input="${1//\"/\\\"}" output="" i char escape=0
for (( i=0; i < ${#input}; ++i )); do # process all characters of input string
char="${input:i:1}" # get current character from input string
if (( ${escape} == 1 )); then # if we're currently within an escape sequence, check if
if [[ "${char}" == [a-zA-Z] ]]; then # end is reached, i.e. if current character is a letter
escape=0 # end reached, we're no longer within an escape sequence
fi
continue # skip current character, i.e. do not add to ouput
fi
if [[ "${char}" == $'\e' ]]; then # if current character is '\e', we've reached the start
escape=1 # of an escape sequence -> set flag
continue # skip current character, i.e. do not add to ouput
fi
output+="${char}" # add current character to output
done
eval "$2=\"${output}\"" # assign output to target variable
}
while read -r line; do
strip_escape_codes "${line}" line_stripped
echo "${line_stripped}"
done
#jeff-bowman's solution helped me getting rid of SOME of the color codes.
I added another small portion to the regex in order to remove some more:
sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # Original. Removed Red ([31;40m[1m[error][0m)
sed -r "s/\x1B\[([0-9];)?([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # With an addition, removed yellow and green ([1;33;40m[1m[warning][0m and [1;32;40m[1m[ok][0m)
^^^^^^^^^
remove Yellow and Green (and maybe more colors)
The controversial idea would be to reconfigure terminal settings for this process environment to let the process know that terminal does not support colors.
Something like TERM=xterm-mono ./somescript comes to my mind. YMMV with your specific OS and ability of your script to understand terminal color settings.
I had some issues with colorized output which the other solutions here didn't process correctly, so I built this perl one liner. It looks for escape \e followed by opening bracket \[ followed by one or color codes \d+ separated by semicolons, ending on m.
perl -ple 's/\e\[\d+(;\d+)*m//g'
It seems to work really well for colorized compiler output.
I came across this question/answers trying to do something similar as the OP. I found some other useful resources and came up with a log script based on those. Posting here in case it can help others.
Digging into the links helps understand some of the redirection which I won't try and explain because I'm just starting to understand it myself.
Usage will render the colorized output to the console, while stripping the color codes out of the text going to the log file. It will also include stderr in the logfile for any commands that don't work.
Edit: adding more usage at bottom to show how to log in different ways
#!/bin/bash
set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
. $DIR/dev.conf
. $DIR/colors.cfg
filename=$(basename ${BASH_SOURCE[0]})
# remove extension
# filename=`echo $filename | grep -oP '.*?(?=\.)'`
filename=`echo $filename | awk -F\. '{print $1}'`
log=$DIR/logs/$filename-$target
if [ -f $log ]; then
cp $log "$log.bak"
fi
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>$log 2>&1
# log message
log(){
local m="$#"
echo -e "*** ${m} ***" >&3
echo "=================================================================================" >&3
local r="$#"
echo "================================================================================="
echo -e "*** $r ***" | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g"
echo "================================================================================="
}
echo "=================================================================================" >&3
log "${Cyan}The ${Yellow}${COMPOSE_PROJECT_NAME} ${filename} ${Cyan}script has been executed${NC}"
log $(ls) #log $(<command>)
log "${Green}Apply tag to image $source with version $version${NC}"
# log $(exec docker tag $source $target 3>&2) #prints error only to console
# log $(docker tag $source $target 2>&1) #prints error to both but doesn't exit on fail
log $(docker tag $source $target 2>&1) && exit $? #prints error to both AND exits on fail
# docker tag $source $target 2>&1 | tee $log # prints gibberish to log
echo $? # prints 0 because log function was successful
log "${Purple}Push $target to acr${NC}"
Here are the other links that helped:
Can I use sed to manipulate a variable in bash?
https://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
https://unix.stackexchange.com/questions/42728/what-does-31-12-23-do-in-a-script
https://serverfault.com/questions/103501/how-can-i-fully-log-all-bash-scripts-actions
https://www.gnu.org/software/bash/manual/bash.html#Redirections
I used perl as I have to do this frequently on many files. This will go through all files with filename*.txt and will remove any formatting. This works for my use case and may be useful for someone else too so just thought of posting here. replace whatever your file name is in place of filename*.txt or you can put file names separated by spaces in setting the FILENAME variable below.
$ FILENAME=$(ls filename*.txt) ; for file in $(echo $FILENAME); do echo $file; cat $file | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > $file-new; mv $file-new $file; done
my contribution:
./somescript | sed -r "s/\\x1B[\\x5d\[]([0-9]{1,3}(;[0-9]{1,3})?(;[0-9]{1,3})?)?[mGK]?//g"
This works for me:
./somescript | cat
I need to analyze a log for specific output and if a certain keyword is matched in the output, I need to issue an echo -e "\a".
How would I filter a script so that this would occur? I pass it through ack[-grep] as well so I'd like to put the alarm notification in prior to the colorization probably.
tail -f file | while read line; do
echo $line | grep -qF keyword && echo -e \\a >&2; echo $line; done
This spawns more processes than sed, but puts the control character on the terminal via stderr so that it is less likely to be modified by downstream processes. Also, it is entirely possible for an implementation of sed to buffer its input and not generate any output until tail generates 8K or so of output, so you might get the alert hours or days after the data occurs.
I would use sed to replace the key word with key word \a. That should work so long as your downstream steps don't mangle the \a.
I have a shell script in which in the first line I ask the user to input how many minutes they want the script to run for:
#!/usr/bin/ksh
echo "How long do you want the script to run for in minutes?:\c"
read scriptduration
loopcnt=0
interval=1
date2=$(date +%H:%M%S)
(( intervalsec = $interval * 1 ))
totalmin=${1:-$scriptduration}
(( loopmax = ${totalmin} * 60 ))
ofile=/home2/s499929/test.log
echo "$date2 total runtime is $totalmin minutes at 2 sec intervals"
while(( $loopmax > $loopcnt ))
do
date1=$(date +%H:%M:%S)
pid=`/usr/local/bin/lsof | grep 16752 | grep LISTEN |awk '{print $2}'` > /dev/null 2>&1
count=$(netstat -an|grep 16752|grep ESTABLISHED|wc -l| sed "s/ //g")
process=$(ps -ef | grep $pid | wc -l | sed "s/ //g")
port=$(netstat -an | grep 16752 | grep LISTEN | wc -l| sed "s/ //g")
echo "$date1 activeTCPcount:$count activePID:$pid activePIDcount=$process listen=$port" >> ${ofile}
sleep $intervalsec
(( loopcnt = loopcnt + 1 ))
done
It works great if I kick it off an input the values manually. But if I want to run this for 3 hours I need to kick off the script to run in the background.
I have tried just running ./scriptname & and I get this:
$ How long do you want the test to run for in minutes:360
ksh: 360: not found.
[2] + Stopped (SIGTTIN) ./test.sh &
And the script dies. Is this possible, any suggestions on how I can accept this one input and then run in the background?? Thanks!!!
You could do something like this:
test.sh arg1 arg2 &
Just refer to arg1 and arg2 as $1 and $2, respectively, in the bash script. ($0 is the name of the script)
So,
test.sh 360 &
will pass 360 as the first argument to the bash or ksh script which can be referred to as $1 in the script.
So the first few lines of your script would now be:
#!/usr/bin/ksh
scriptduration=$1
loopcnt=0
...
...
With bash you can start the script in the foreground and after you finished with the user input, interrupt it by hitting Ctrl-Z.
Then type
$ bg %
and the script will continue to run in the background.
Why You're Getting What You're Getting
When you run the script in the background, it can't take any user input. In fact, the program will freeze if it expects user input until its put back in the foreground. However, output has to go somewhere. Thus, the output goes to the screen (even though the program is running in the background. Thus, you see the prompt.
The prompt you see your program displaying is meaningless because you can't input at the prompt. Instead, you type in 360 and your shell is interpreting it as a command you want because you're not putting it in the program, you're putting it in the command prompt.
You want your program to be in the foreground for the input, but run in the background. You can't do both at once.
Solutions To Your Dilemma
You can have two programs. The first takes the input, and the second runs the actual program in the background.
Something like this:
#! /bin/ksh
read time?"How long in seconds do you want to run the job? "
my_actual_job.ksh $time &
In fact, you could even have a mechanism to run the job in the background if the time is over a certain limit, but otherwise run the job in the foreground.
#! /bin/ksh
readonly MAX_FOREGROUND_TIME=30
read time?"How long in seconds do you want to run the job? "
if [ $time -gt $MAX_FOREGROUND_TIME ]
then
my_actual_job.ksh $time &
else
my_actual_job.ksh $time
fi
Also remember if your job is in the background, it cannot print to the screen. You can redirect the output elsewhere, but if you don't, it'll print to the screen at inopportune times. For example, you could be in VI editing a file, and suddenly have the output appear smack in the middle of your VI session.
I believe there's an easy way to tell if your job is in the background, but I can't remember it offhand. You could find your current process ID by looking at $$, then looking at the output of jobs -p and see if that process ID is in the list. However, I'm sure someone will come up with an easy way to tell.
It is also possible that a program could throw itself into the background via the bg $$ command.
Some Hints
If you're running Kornshell, you might consider taking advantage of many of Kornshell's special features:
print: The print command is more flexible and robust than echo. Take a look at the manpage for Kornshell and see all of its features.
read: You notice that you can use the read var?"prompt" form of the read command.
readonly: Use readonly to declare constants. That way, you don't accidentally change the value of that variable later. Besides, it's good programming technique.
typeset: Take a look at typeset in the ksh manpage. The typeset command can help you declare particular variables as floating point vs. real, and can automatically do things like zero fill, right or left justify, etc.
Some things not specific to Kornshell:
The awk and sed commands can also do what grep does, so there's no reason to filter something through grep and then through awk or sed.
You can combine greps by using the -e parameter. grep foo | grep bar is the same as grep -e foo -e bar.
Hope this helps.
I've tested this with ksh and it worked. The trick is to let the script call itself with the time to wait as parameter:
if [ -z "$1" ]; then
echo "How long do you want the test to run for in minutes:\c"
read scriptduration
echo "running task in background"
$0 $scriptduration &
exit 0
else
scriptduration=$1
fi
loopcnt=0
interval=1
# ... and so on
So are you using bash or ksh? In bash, you can do this:
{ echo 360 | ./test.sh ; } &
It could work for ksh also.