I have some script that produces output with colors and I need to remove the ANSI codes.
#!/bin/bash
exec > >(tee log) # redirect the output to a file but keep it on stdout
exec 2>&1
./somescript
The output is (in log file):
java (pid 12321) is running...#[60G[#[0;32m OK #[0;39m]
I didn't know how to put the ESC character here, so I put # in its place.
I changed the script into:
#!/bin/bash
exec > >(tee log) # redirect the output to a file but keep it on stdout
exec 2>&1
./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"
But now it gives me (in log file):
java (pid 12321) is running...#[60G[ OK ]
How can I also remove this '#[60G?
Maybe there is a way to completely disable coloring for the entire script?
According to Wikipedia, the [m|K] in the sed command you're using is specifically designed to handle m (the color command) and K (the "erase part of line" command). Your script is trying to set absolute cursor position to 60 (^[[60G) to get all the OKs in a line, which your sed line doesn't cover.
(Properly, [m|K] should probably be (m|K) or [mK], because you're not trying to match a pipe character. But that's not important right now.)
If you switch that final match in your command to [mGK] or (m|G|K), you should be able to catch that extra control sequence.
./somescript | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2};?)?)?[mGK]//g"
IMHO, most of these answers try too hard to restrict what is inside the escape code. As a result, they end up missing common codes like [38;5;60m (foreground ANSI color 60 from 256-color mode).
They also require the -r option which enables GNU extensions. These are not required; they just make the regex read better.
Here is a simpler answer that handles the 256-color escapes and works on systems with non-GNU sed:
./somescript | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g'
This will catch anything that starts with [, has any number of decimals and semicolons, and ends with a letter. This should catch any of the common ANSI escape sequences.
For funsies, here's a larger and more general (but minimally tested) solution for all conceivable ANSI escape sequences:
./somescript | sed 's/\x1B[#A-Z\\\]^_]\|\x1B\[[0-9:;<=>?]*[-!"#$%&'"'"'()*+,.\/]*[][\\#A-Z^_`a-z{|}~]//g'
(and if you have #edi9999's SI problem, add | sed "s/\x0f//g" to the end; this works for any control char by replacing 0f with the hex of the undesired char)
I couldn't get decent results from any of the other answers, but the following worked for me:
somescript | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g"
If I only removed the control char "^[", it left the rest of the color data, e.g., "33m". Including the color code and "m" did the trick. I'm puzzled with s/\x1B//g doesn't work because \x1B[31m certainly works with echo.
I came across ansi2txt tool from colorized-logs package in Debian. The tool drops ANSI control codes from STDIN.
Usage example:
./somescript | ansi2txt
Source code http://github.com/kilobyte/colorized-logs
For Mac OSX or BSD use
./somescript | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'
The regular expression below will miss some ANSI Escape Codes sequences, as well as 3 digit colors. Example and Fix on regex101.com.
Use this instead:
./somescript | sed -r 's/\x1B\[(;?[0-9]{1,3})+[mGK]//g'
I also had the problem that sometimes, the SI character appeared.
It happened for example with this input : echo "$(tput setaf 1)foo$(tput sgr0) bar"
Here's a way to also strip the SI character (shift in) (0x0f)
./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | sed "s/\x0f//g"
Much simpler function in pure Bash to filter-out common ANSI codes from a text stream:
# Strips common ANSI codes from a text stream
shopt -s extglob # Enable Bash Extended Globbing expressions
ansi_filter() {
local line
local IFS=
while read -r line || [[ "$line" ]]; do
printf '%s\n' "${line//$'\e'[\[(]*([0-9;])[#-n]/}"
done
}
See:
linuxjournal.com: Extended Globbing
gnu.org: Bash Parameter Expansion
I had a similar problem. All solutions I found did work well for the color codes but did not remove the characters added by "$(tput sgr0)" (resetting attributes).
Taking, for example, the solution in the comment by davemyron the length of the resulting string in the example below is 9, not 6:
#!/usr/bin/env bash
string="$(tput setaf 9)foobar$(tput sgr0)"
string_sed="$( sed -r "s/\x1B\[[0-9;]*[JKmsu]//g" <<< "${string}" )"
echo ${#string_sed}
In order to work properly, the regex had to be extend to also match the sequence added by sgr0 ("\E(B"):
string_sed="$( sed -r "s/\x1B(\[[0-9;]*[JKmsu]|\(B)//g" <<< "${string}" )"
Not sure what's in ./somescript but if escape sequences are not hardcoded you can set the terminal type to avoid them
TERM=dumb ./somescript
For example, if you try
TERM=dumb tput sgr0 | xxd
you'll see it produces no output while
tput sgr0 | xxd
00000000: 1b28 421b 5b6d .(B.[m
does (for xterm-256color).
Hmm, not sure if this will work for you, but 'tr' will 'strip' (delete) control codes - try:
./somescript | tr -d '[:cntrl:]'
There's also a dedicated tool to handle ANSI escape sequences: ansifilter. Use the default --text output format to strip all ANSI escape sequences (note: not just coloring).
ref: https://stackoverflow.com/a/6534712
Here's a pure Bash solution.
Save as strip-escape-codes.sh, make executable and then run <command-producing-colorful-output> | ./strip-escape-codes.sh.
Note that this strips all ANSI escape codes/sequences. If you want to strip colors only, replace [a-zA-Z] with "m".
Bash >= 4.0:
#!/usr/bin/env bash
# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
local _input="$1" _i _char _escape=0
local -n _output="$2"; _output=""
for (( _i=0; _i < ${#_input}; _i++ )); do
_char="${_input:_i:1}"
if (( ${_escape} == 1 )); then
if [[ "${_char}" == [a-zA-Z] ]]; then
_escape=0
fi
continue
fi
if [[ "${_char}" == $'\e' ]]; then
_escape=1
continue
fi
_output+="${_char}"
done
}
while read -r line; do
strip_escape_codes "${line}" line_stripped
echo "${line_stripped}"
done
Bash < 4.0:
#!/usr/bin/env bash
# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
local input="${1//\"/\\\"}" output="" i char escape=0
for (( i=0; i < ${#input}; ++i )); do # process all characters of input string
char="${input:i:1}" # get current character from input string
if (( ${escape} == 1 )); then # if we're currently within an escape sequence, check if
if [[ "${char}" == [a-zA-Z] ]]; then # end is reached, i.e. if current character is a letter
escape=0 # end reached, we're no longer within an escape sequence
fi
continue # skip current character, i.e. do not add to ouput
fi
if [[ "${char}" == $'\e' ]]; then # if current character is '\e', we've reached the start
escape=1 # of an escape sequence -> set flag
continue # skip current character, i.e. do not add to ouput
fi
output+="${char}" # add current character to output
done
eval "$2=\"${output}\"" # assign output to target variable
}
while read -r line; do
strip_escape_codes "${line}" line_stripped
echo "${line_stripped}"
done
#jeff-bowman's solution helped me getting rid of SOME of the color codes.
I added another small portion to the regex in order to remove some more:
sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # Original. Removed Red ([31;40m[1m[error][0m)
sed -r "s/\x1B\[([0-9];)?([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # With an addition, removed yellow and green ([1;33;40m[1m[warning][0m and [1;32;40m[1m[ok][0m)
^^^^^^^^^
remove Yellow and Green (and maybe more colors)
The controversial idea would be to reconfigure terminal settings for this process environment to let the process know that terminal does not support colors.
Something like TERM=xterm-mono ./somescript comes to my mind. YMMV with your specific OS and ability of your script to understand terminal color settings.
I had some issues with colorized output which the other solutions here didn't process correctly, so I built this perl one liner. It looks for escape \e followed by opening bracket \[ followed by one or color codes \d+ separated by semicolons, ending on m.
perl -ple 's/\e\[\d+(;\d+)*m//g'
It seems to work really well for colorized compiler output.
I came across this question/answers trying to do something similar as the OP. I found some other useful resources and came up with a log script based on those. Posting here in case it can help others.
Digging into the links helps understand some of the redirection which I won't try and explain because I'm just starting to understand it myself.
Usage will render the colorized output to the console, while stripping the color codes out of the text going to the log file. It will also include stderr in the logfile for any commands that don't work.
Edit: adding more usage at bottom to show how to log in different ways
#!/bin/bash
set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
. $DIR/dev.conf
. $DIR/colors.cfg
filename=$(basename ${BASH_SOURCE[0]})
# remove extension
# filename=`echo $filename | grep -oP '.*?(?=\.)'`
filename=`echo $filename | awk -F\. '{print $1}'`
log=$DIR/logs/$filename-$target
if [ -f $log ]; then
cp $log "$log.bak"
fi
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>$log 2>&1
# log message
log(){
local m="$#"
echo -e "*** ${m} ***" >&3
echo "=================================================================================" >&3
local r="$#"
echo "================================================================================="
echo -e "*** $r ***" | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g"
echo "================================================================================="
}
echo "=================================================================================" >&3
log "${Cyan}The ${Yellow}${COMPOSE_PROJECT_NAME} ${filename} ${Cyan}script has been executed${NC}"
log $(ls) #log $(<command>)
log "${Green}Apply tag to image $source with version $version${NC}"
# log $(exec docker tag $source $target 3>&2) #prints error only to console
# log $(docker tag $source $target 2>&1) #prints error to both but doesn't exit on fail
log $(docker tag $source $target 2>&1) && exit $? #prints error to both AND exits on fail
# docker tag $source $target 2>&1 | tee $log # prints gibberish to log
echo $? # prints 0 because log function was successful
log "${Purple}Push $target to acr${NC}"
Here are the other links that helped:
Can I use sed to manipulate a variable in bash?
https://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
https://unix.stackexchange.com/questions/42728/what-does-31-12-23-do-in-a-script
https://serverfault.com/questions/103501/how-can-i-fully-log-all-bash-scripts-actions
https://www.gnu.org/software/bash/manual/bash.html#Redirections
I used perl as I have to do this frequently on many files. This will go through all files with filename*.txt and will remove any formatting. This works for my use case and may be useful for someone else too so just thought of posting here. replace whatever your file name is in place of filename*.txt or you can put file names separated by spaces in setting the FILENAME variable below.
$ FILENAME=$(ls filename*.txt) ; for file in $(echo $FILENAME); do echo $file; cat $file | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > $file-new; mv $file-new $file; done
my contribution:
./somescript | sed -r "s/\\x1B[\\x5d\[]([0-9]{1,3}(;[0-9]{1,3})?(;[0-9]{1,3})?)?[mGK]?//g"
This works for me:
./somescript | cat
Related
I want to read a file and save it in variable, but I need to keep the variable and not just print out the file.
How can I do this? I have written this script but it isn't quite what I needed:
#!/bin/sh
while read LINE
do
echo $LINE
done <$1
echo 11111-----------
echo $LINE
In my script, I can give the file name as a parameter, so, if the file contains "aaaa", for example, it would print out this:
aaaa
11111-----
But this just prints out the file onto the screen, and I want to save it into a variable!
Is there an easy way to do this?
In cross-platform, lowest-common-denominator sh you use:
#!/bin/sh
value=`cat config.txt`
echo "$value"
In bash or zsh, to read a whole file into a variable without invoking cat:
#!/bin/bash
value=$(<config.txt)
echo "$value"
Invoking cat in bash or zsh to slurp a file would be considered a Useless Use of Cat.
Note that it is not necessary to quote the command substitution to preserve newlines.
See: Bash Hacker's Wiki - Command substitution - Specialities.
If you want to read the whole file into a variable:
#!/bin/bash
value=`cat sources.xml`
echo $value
If you want to read it line-by-line:
while read line; do
echo $line
done < file.txt
Two important pitfalls
which were ignored by other answers so far:
Trailing newline removal from command expansion
NUL character removal
Trailing newline removal from command expansion
This is a problem for the:
value="$(cat config.txt)"
type solutions, but not for read based solutions.
Command expansion removes trailing newlines:
S="$(printf "a\n")"
printf "$S" | od -tx1
Outputs:
0000000 61
0000001
This breaks the naive method of reading from files:
FILE="$(mktemp)"
printf "a\n\n" > "$FILE"
S="$(<"$FILE")"
printf "$S" | od -tx1
rm "$FILE"
POSIX workaround: append an extra char to the command expansion and remove it later:
S="$(cat $FILE; printf a)"
S="${S%a}"
printf "$S" | od -tx1
Outputs:
0000000 61 0a 0a
0000003
Almost POSIX workaround: ASCII encode. See below.
NUL character removal
There is no sane Bash way to store NUL characters in variables.
This affects both expansion and read solutions, and I don't know any good workaround for it.
Example:
printf "a\0b" | od -tx1
S="$(printf "a\0b")"
printf "$S" | od -tx1
Outputs:
0000000 61 00 62
0000003
0000000 61 62
0000002
Ha, our NUL is gone!
Workarounds:
ASCII encode. See below.
use bash extension $"" literals:
S=$"a\0b"
printf "$S" | od -tx1
Only works for literals, so not useful for reading from files.
Workaround for the pitfalls
Store an uuencode base64 encoded version of the file in the variable, and decode before every usage:
FILE="$(mktemp)"
printf "a\0\n" > "$FILE"
S="$(uuencode -m "$FILE" /dev/stdout)"
uudecode -o /dev/stdout <(printf "$S") | od -tx1
rm "$FILE"
Output:
0000000 61 00 0a
0000003
uuencode and udecode are POSIX 7 but not in Ubuntu 12.04 by default (sharutils package)... I don't see a POSIX 7 alternative for the bash process <() substitution extension except writing to another file...
Of course, this is slow and inconvenient, so I guess the real answer is: don't use Bash if the input file may contain NUL characters.
this works for me:
v=$(cat <file_path>)
echo $v
With bash you may use read like this:
#!/usr/bin/env bash
{ IFS= read -rd '' value <config.txt;} 2>/dev/null
printf '%s' "$value"
Notice that:
The last newline is preserved.
The stderr is silenced to /dev/null by redirecting the whole commands block, but the return status of the read command is preserved, if one needed to handle read error conditions.
As Ciro Santilli notes using command substitutions will drop trailing newlines. Their workaround adding trailing characters is great, but after using it for quite some time I decided I needed a solution that didn't use command substitution at all.
My approach now uses read along with the printf builtin's -v flag in order to read the contents of stdin directly into a variable.
# Reads stdin into a variable, accounting for trailing newlines. Avoids
# needing a subshell or command substitution.
# Note that NUL bytes are still unsupported, as Bash variables don't allow NULs.
# See https://stackoverflow.com/a/22607352/113632
read_input() {
# Use unusual variable names to avoid colliding with a variable name
# the user might pass in (notably "contents")
: "${1:?Must provide a variable to read into}"
if [[ "$1" == '_line' || "$1" == '_contents' ]]; then
echo "Cannot store contents to $1, use a different name." >&2
return 1
fi
local _line _contents=()
while IFS='' read -r _line; do
_contents+=("$_line"$'\n')
done
# include $_line once more to capture any content after the last newline
printf -v "$1" '%s' "${_contents[#]}" "$_line"
}
This supports inputs with or without trailing newlines.
Example usage:
$ read_input file_contents < /tmp/file
# $file_contents now contains the contents of /tmp/file
All the given solutions are quite slow, so:
mapfile -d '' content </etc/passwd # Read file into an array
content="${content[*]%$'\n'}" # Remove trailing newline
Would be nice to optimise it even more but I can't think of much
Update: Found a faster way
read -rd '' content </etc/passwd
This will return exit code of 1 so if you need it
to be always 0:
read -rd '' content </etc/passwd || :
I use:
NGINX_PID=`cat -s "/sdcard/server/nginx/logs/nginx.pid" 2>/dev/null`
if [ "$NGINX_PID" = "" ]; then
echo "..."
exit
fi
You can access 1 line at a time by for loop
#!/bin/bash -eu
#This script prints contents of /etc/passwd line by line
FILENAME='/etc/passwd'
I=0
for LN in $(cat $FILENAME)
do
echo "Line number $((I++)) --> $LN"
done
Copy the entire content to File (say line.sh ) ; Execute
chmod +x line.sh
./line.sh
I have a big problem doing a script: basically, I read a line from files.
All lines are made of 3 to 8 characters contiguous (no space).
Then I used sed to replace those lines inside a pattern (aka "var" in my minimal script below)
var="iao"
for m in `more meshing/junction_names.txt`
do
echo $m
echo -n $m | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
echo $var |sed "s/a/b/"
echo $var |sed "s/a/$m/"
done
Now these are the first 3 record of my output (they are all the same anyway).
I am using linux. According kate, all files are encoded UTF-8. Very weird huh? Any idea why that is is welcome.
J_LEAK
\x4a\x5f\x4c\x45\x41\x4b\x0d
ibo
oJ_LEAK
JO_1
\x4a\x4f\x5f\x31\x0d
ibo
oJO_1
JPL2_F
\x4a\x50\x4c\x32\x5f\x46\x0d
ibo
oJPL2_F
JF_PL2
Your input file contains DOS carriage returns (or possibly, the absurd attempt to read it with more introduces them). The hex dump shows this clearly; every value ends with \x0d which translates to a control code which causes the terminal to jump the cursor back to the beginning of the line.
This is a massive FAQ and you can find many examples of how to troubleshoot this basic problem, including in the bash tag wiki.
Tangentially, you should always quote strings unless you specifically require the shell to perform wildcard expansion and whitespace tokenization on the value; and Bash has built-ins to avoid the inelegant and somewhat error-prone echo | sed. Finally, don't read lines with for.
var="iao"
tr -d '\015' <meshing/junction_names.txt |
while read -r m; do # don't use a for loop
echo "$m" # quote!
echo -n "$m" | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
echo "${var/a/b}" # quote; use Bash built-in substitution mechanism
echo "${var/a/$m}"
done
Perhaps you want to remove the carriage returns once and for all, and then just use while read .... done <fixed-file instead of the tr pipeline.
I'm currently using the following to capture everything that goes to the terminal and throw it into a log file
exec 4<&1 5<&2 1>&2>&>(tee -a $LOG_FILE)
however, I don't want color escape codes/clutter going into the log file. so i have something like this that sorta works
exec 4<&1 5<&2 1>&2>&>(
while read -u 0; do
#to terminal
echo "$REPLY"
#to log file (color removed)
echo "$REPLY" | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' >> $LOG_FILE
done
unset REPLY #tidy
)
except read waits for carriage return which isn't ideal for some portions of the script (e.g. echo -n "..." or printf without \n).
Follow-up to Jonathan Leffler's answer:
Given the example script test.sh:
#!/bin/bash
LOG_FILE="./test.log"
echo -n >$LOG_FILE
exec 4<&1 5<&2 1>&2>&>(tee -a >(sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' > $LOG_FILE))
##### ##### #####
# Main
echo "starting execution"
printf "\n\n"
echo "color test:"
echo -e "\033[0;31mhello \033[0;32mworld\033[0m!"
printf "\n\n"
echo -e "\033[0;36mEnvironment:\033[0m\n foo: cat\n bar: dog\n your wife: hot\n fix: A/C"
echo -n "Before we get started. Is the above information correct? "
read YES
echo -e "\n[READ] $YES" >> $LOG_FILE
YES=$(echo "$YES" | sed 's/^\s*//;s/\s*$//')
test ! "$(echo "$YES" | grep -iE '^y(es)?$')" && echo -e "\nExiting... :(" && exit
printf "\n\n"
#...some hundreds of lines of code later...
echo "Done!"
##### ##### #####
# End
exec 1<&4 4>&- 2<&5 5>&-
echo "Log File: $LOG_FILE"
The output to the terminal is as expected and there is no color escape codes/clutter in the log file as desired. However upon examining test.log, I do not see the [READ] ... (see line 21 of test.sh).
The log file [of my actual bash script] contains the line Log File: ... at the end of it even after closing the 4 and 5 fds. I was able to resolve the issue by putting a sleep 1 before the second exec - I assume there's a race condition or fd shenanigans to blame for it. Unfortunately for you guys, I am not able to reproduce this issue with test.sh but I'd be interested in any speculation anyone may have.
Consider using the pee program discussed in Is it possible to distribute stdin over parallel processes. It would allow you to send the log data through your sed script, while continuing to send the colours to the actual output.
One major advantage of this is that it would remove the 'execute sed once per line of log output'; that is really diabolical for performance (in terms of number of processes executed, if nothing else).
I know it's not a perfect solution, but cat -v will make non visible chars like \x1B to be converted into visible form like ^[[1;34m. The output will be messy, but it will be ascii text at least.
I use to do stuff like this by setting TERM=dumb before running my command. That pretty much removed any control characters except for tab, CR, and LF. I have no idea if this works for your situation, but it's worth a try. The problem is that you won't see color encodings on your terminal either since it's a dumb terminal.
You can also try either vis or cat (especially the -v parameter) and see if these do something for you. You'd simply put them in your pipeline like this:
exec 4<&1 5<&2 1>&2>&>(tee -a | cat -v | $LOG_FILE)
By the way, almost all terminal programs have an option to capture the input, and most clean it up for you. What platform are you on, and what type of terminal program are you using?
You could attempt to use the -n option for read. It reads in n characters instead of waiting for a new line. You could set it to one. This would increase the number of iteration the code runs, but it would not wait for newlines.
From the man:
-n NCHARS read returns after reading NCHARS characters rather than waiting for a complete line of input.
Note: I have not tested this
You can use ANSIFilter to strip or transform console output with ANSI escape sequences.
See http://www.andre-simon.de/zip/download.html#ansifilter
Might not screen -L or the script commands be viable options instead of this exec loop?
I want to read a file and save it in variable, but I need to keep the variable and not just print out the file.
How can I do this? I have written this script but it isn't quite what I needed:
#!/bin/sh
while read LINE
do
echo $LINE
done <$1
echo 11111-----------
echo $LINE
In my script, I can give the file name as a parameter, so, if the file contains "aaaa", for example, it would print out this:
aaaa
11111-----
But this just prints out the file onto the screen, and I want to save it into a variable!
Is there an easy way to do this?
In cross-platform, lowest-common-denominator sh you use:
#!/bin/sh
value=`cat config.txt`
echo "$value"
In bash or zsh, to read a whole file into a variable without invoking cat:
#!/bin/bash
value=$(<config.txt)
echo "$value"
Invoking cat in bash or zsh to slurp a file would be considered a Useless Use of Cat.
Note that it is not necessary to quote the command substitution to preserve newlines.
See: Bash Hacker's Wiki - Command substitution - Specialities.
If you want to read the whole file into a variable:
#!/bin/bash
value=`cat sources.xml`
echo $value
If you want to read it line-by-line:
while read line; do
echo $line
done < file.txt
Two important pitfalls
which were ignored by other answers so far:
Trailing newline removal from command expansion
NUL character removal
Trailing newline removal from command expansion
This is a problem for the:
value="$(cat config.txt)"
type solutions, but not for read based solutions.
Command expansion removes trailing newlines:
S="$(printf "a\n")"
printf "$S" | od -tx1
Outputs:
0000000 61
0000001
This breaks the naive method of reading from files:
FILE="$(mktemp)"
printf "a\n\n" > "$FILE"
S="$(<"$FILE")"
printf "$S" | od -tx1
rm "$FILE"
POSIX workaround: append an extra char to the command expansion and remove it later:
S="$(cat $FILE; printf a)"
S="${S%a}"
printf "$S" | od -tx1
Outputs:
0000000 61 0a 0a
0000003
Almost POSIX workaround: ASCII encode. See below.
NUL character removal
There is no sane Bash way to store NUL characters in variables.
This affects both expansion and read solutions, and I don't know any good workaround for it.
Example:
printf "a\0b" | od -tx1
S="$(printf "a\0b")"
printf "$S" | od -tx1
Outputs:
0000000 61 00 62
0000003
0000000 61 62
0000002
Ha, our NUL is gone!
Workarounds:
ASCII encode. See below.
use bash extension $"" literals:
S=$"a\0b"
printf "$S" | od -tx1
Only works for literals, so not useful for reading from files.
Workaround for the pitfalls
Store an uuencode base64 encoded version of the file in the variable, and decode before every usage:
FILE="$(mktemp)"
printf "a\0\n" > "$FILE"
S="$(uuencode -m "$FILE" /dev/stdout)"
uudecode -o /dev/stdout <(printf "$S") | od -tx1
rm "$FILE"
Output:
0000000 61 00 0a
0000003
uuencode and udecode are POSIX 7 but not in Ubuntu 12.04 by default (sharutils package)... I don't see a POSIX 7 alternative for the bash process <() substitution extension except writing to another file...
Of course, this is slow and inconvenient, so I guess the real answer is: don't use Bash if the input file may contain NUL characters.
this works for me:
v=$(cat <file_path>)
echo $v
With bash you may use read like this:
#!/usr/bin/env bash
{ IFS= read -rd '' value <config.txt;} 2>/dev/null
printf '%s' "$value"
Notice that:
The last newline is preserved.
The stderr is silenced to /dev/null by redirecting the whole commands block, but the return status of the read command is preserved, if one needed to handle read error conditions.
As Ciro Santilli notes using command substitutions will drop trailing newlines. Their workaround adding trailing characters is great, but after using it for quite some time I decided I needed a solution that didn't use command substitution at all.
My approach now uses read along with the printf builtin's -v flag in order to read the contents of stdin directly into a variable.
# Reads stdin into a variable, accounting for trailing newlines. Avoids
# needing a subshell or command substitution.
# Note that NUL bytes are still unsupported, as Bash variables don't allow NULs.
# See https://stackoverflow.com/a/22607352/113632
read_input() {
# Use unusual variable names to avoid colliding with a variable name
# the user might pass in (notably "contents")
: "${1:?Must provide a variable to read into}"
if [[ "$1" == '_line' || "$1" == '_contents' ]]; then
echo "Cannot store contents to $1, use a different name." >&2
return 1
fi
local _line _contents=()
while IFS='' read -r _line; do
_contents+=("$_line"$'\n')
done
# include $_line once more to capture any content after the last newline
printf -v "$1" '%s' "${_contents[#]}" "$_line"
}
This supports inputs with or without trailing newlines.
Example usage:
$ read_input file_contents < /tmp/file
# $file_contents now contains the contents of /tmp/file
All the given solutions are quite slow, so:
mapfile -d '' content </etc/passwd # Read file into an array
content="${content[*]%$'\n'}" # Remove trailing newline
Would be nice to optimise it even more but I can't think of much
Update: Found a faster way
read -rd '' content </etc/passwd
This will return exit code of 1 so if you need it
to be always 0:
read -rd '' content </etc/passwd || :
I use:
NGINX_PID=`cat -s "/sdcard/server/nginx/logs/nginx.pid" 2>/dev/null`
if [ "$NGINX_PID" = "" ]; then
echo "..."
exit
fi
You can access 1 line at a time by for loop
#!/bin/bash -eu
#This script prints contents of /etc/passwd line by line
FILENAME='/etc/passwd'
I=0
for LN in $(cat $FILENAME)
do
echo "Line number $((I++)) --> $LN"
done
Copy the entire content to File (say line.sh ) ; Execute
chmod +x line.sh
./line.sh
I have need to grep an entire directory for a string, and I get about 50 results. I would like to colour each second line, either text colour or background colour. Best would be a script that I can pipe the output of any command to, and so that it spits out the same (albeit coloured) output.
Not very pretty but does the trick:
(save this to foo.bash and do grep whatever wherever | ./foo.bash)
#!/bin/bash
while read line
do
echo -e "\e[1;31m$line"
read line
echo -e "\e[1;32m$line"
done
echo -en "\e[0m"
Here you can find the list of color codes in bash.
Perl is installed on many systems. You could have it alternate for you:
grep -r whatever somedir/ | perl -pe '$_ = "\033[1;29m$_\033[0m" if($. % 2)'
In Perl $. can be substituted with $INPUT_LINE_NUMBER if you prefer readability.
This is to delineate wrapped lines I presume?
This shell script uses a background color from the 256 color palette,
so as not to interfere with other highlighting that grep --color might do.
#!/bin/sh
c=0
while read line; do
[ $(($c%2)) -eq 1 ] && printf "\033[48;5;60m"
printf "%s\033[0m\n" "$line"
c=$(($c+1))
done
This has the caveat that backslashes etc. within the line will be mangled,
so treat this as pseudo code for reimplementation
and here is the same in python;
import sys
for line_number,line in enumerate(sys.stdin.readlines()):
print '%s[1;3%dm%s%s[0m' % (chr(27),(line_number % 2+1),line,chr(27)),