cat file.json gives me what I want to capture inside of $JSON:
{
key: "value\nwith\nnewline\nchars"
}
I can't do JSON=$(cat file.json) though because then the newline characters are translated and I get after echo $JSON or echo -e $JSON.
{
key: "value
with
newline
chars"
}.
How can I preserve the newline characters inside of $JSON?
Capture using command substitution doesn't perform the translation you're worried about here, but using echo (or misusing printf by substituting into the format string rather than a separate parameter) will.
To emit a variable with backslash sequences intact, use:
printf '%s\n' "$JSON"
This avoids behavior that echo can have (either explicitly with bash's noncompliant extension for echo -e, or implicitly when the xpg_echo flag is enabled in bash, or as default out-of-the-box behavior with other, POSIX+XSI-compatible /bin/sh implementations) wherein escape sequences are replaced by echo, even if the variable passed as an argument had a multi-character backslash sequence.
Related
I've noticed that to print the value of the $IFS variable in the shell I have to do something like:
$ printf '%s\nYour IFS is: %q' 'Hello' "$IFS"
Hello
Your IFS is: $' \t\n'
My question is why do I need to pass the IFS in that special way? For example, why (or why wouldn't) it be possible to do:
$ echo $IFS -- some parameter that prints special characters?
$ printf "$IFS" or $ printf '$IFS' -- why wouldn't either of these work?
Why $ printf "%q" $IFS and $ printf "%q" '$IFS' don't show this properly but $ printf "%q" "$IFS" does?
$ echo $IFS -- some parameter that prints special characters?
echo doesn't have such a parameter
$ printf "$IFS" or $ printf '$IFS' -- why wouldn't either of these work?
The first does interpolation of the string just like echo does and prints the IFS string just like it is - which by default is a bunch of whitespaces.
The second does not do interpolation and obviously prints $IFS.
printf "%q" $IFS
The variable has already been expanded into whitespaces that are eaten up by the shell so nothing is passed as a second parameter to printf and therefore %q has no input to work with.
printf "%q" '$IFS'
The string $IFS is passed as a parameter to %q which just adds escape characters to it.
You are running into three basic problems:
Unquoted variable references (as in echo $IFS and printf "%q" $IFS) undergo word splitting. Basically, any whitespace in the variable's value is treated as a separator between "words" (which get passed as arguments to the command). But "whitespace" is defined as the characters in $IFS (that's what it's for!). Therefore, the entire variable's value gets treated as just spacing, not actual content, and effectively vanishes!
This is one of the many examples of why you should put double-quotes around variable references.
Single-quoted strings (as in printf '$IFS' and printf "%q" '$IFS') don't undergo variable expansion at all. In these cases, $IFS is just a literal string, not a variable reference.
Finally, the default characters in $IFS aren't particularly visible on screen. printf "$IFS" actually does print them correctly (unless you put "%" or "\" in IFS for some reason), but you can't really see them. By default, the first character in $IFS is a space, so when that prints the cursor moves to the second column, but nothing visible appears. The second character is a tab, so the cursor moves even further over, but again nothing is actually visible. Then the last character is a newline, so the cursor moves to the beginning of the next line... and again, nothing visible appears.
This is why printf %q is needed -- the %q format converts the not-really-visible characters in $IFS into a visible, readable representation.
If you don't quote the variable, word-splitting is done after the variable is expanded. Word splitting treats all the characters in IFS as word delimiters; multiple of them in a row between words are collapsed into a single space, and if the output is entirely delimiters, nothing at all is output.
You need to use quotes around the variable to output it literally.
Then you need to use the %q format operator to output it as escape sequences so you can see what the individual characters are. Otherwise you wouldn't be able to tell that the second character is a TAB, you would just see a bunch of spaces on the screen. And the newline would just go to the next line.
I am writing a shell script for which I write a header that has 30 (growing) column names. Right now, I have a echo statement that works and looks like this
echo "Colum_Name1, Column_Name2,Column_Name30"
While this works the readability sucks for me. if i want to add a column, its a bit of a nightmare to look at the screen and understand whether it is already in there. of course, I search my way out. Is it possible to do something like this with echo or printf and get the CSV in one line?
echo " Column_Name1,
Column_Name2,
Column_Name30"
and get the output as
Column_Name1,Column_Name2,Column_Name30
You can add backslash as the line continuation:
echo " Column_Name1,"\
"Column_Name2,"\
"Column_Name30"
From the bash manual:
The backslash character ‘\’ may be used to remove any special meaning
for the next character read and for line continuation.
Decouple the definition of the header and printing it, and use an array to store the column names.
headers=(
Column_Name1
Column_Name2
Column_Name30
)
(IFS=","; printf '%s\n' "${headers[*]}")
The elements of the array are joined by the first character of IFS when ${headers[*]} is expanded. The subshell is used so you don't have to worry about restoring the previous value of IFS.
Convenience solution, using paste:
If you don't mind the (probably negligible) overhead of invoking an external utility (paste) to build your string, you can combine it with a (literal, in this case) here-doc:
paste -s -d, - <<'EOF'
Column_Name1
Column_Name2
Column_Name30
EOF
yields
Column_Name1,Column_Name2,Column_Name30
The above acts like a single-quoted string, due to the opening delimiter, 'EOF', being quoted.
Omit the enclosing '...' to treat the string like a double-quoted string, i.e., with expansions being performed (allowing the inclusion of variable references, command substitutions, and arithmetic expansions).
If you take care to use actual leading tabs (\t) in your here-doc (multiple spaces do not work), you can even introduce indentation, by prepending - to the opening delimiter:
# !! Only works with actual *tabs* as the leading whitespace.
paste -s -d, - <<-'EOF'
Column_Name1
Column_Name2
Column_Name30
EOF
More efficient solution, using line continuation:
POSIX-compatible shells support line continuation even inside double-quoted strings, "..." (but not inside single-quoted ones, '...').
That means that any \<newline> sequence inside a double-quoted string is removed:
echo "\
Column_Name1,\
Column_Name2,\
Column_Name3\
"
Given that a here-document with an unquoted opening delimiter is treated like a double-quoted string, you can do the following:
cat <<EOF
Column_Name1,\
Column_Name2,\
Column_Name30
EOF
Note:
Using <<-EOF with to-be-stripped leading tabs (\t) for readability is not an option here, because the line continuations will still include them.
To take advantage of line continuation, it is invariably the interpolating (expanding) here-doc variety that must be used; therefore, you may need to \-escape $ instances to ensure their literal use.
Both commands again yield the desired single-line string:
Column_Name1,Column_Name2,Column_Name30
echo "foo bar" | (IFS=" "; xargs -n 1 echo)
yields
foo
bar
I would like to use the bash here doc syntax to build a long string. I would like the heredoc to ignore newlines/spaces/tabs even when I use newlines for code clarity.
I thought this would work:
#!/bin/bash
#http://unix.stackexchange.com/questions/20035/how-to-add-newlines-into-variables-in-bash-script
IFS= read -r -d '' NS_LOG<<-EOF
*=error|warn|prefix_node|prefix_func
:PointToPointNetDevice
:ClockTest
:ClockPerfect
:TcpTestSuite
:TcpRxBuffer
:TcpTxBuffer
:TcpHeader=*
:TcpL4Protocol
:TraceHelper:PointToPointHelper
EOF
echo $NS_LOG
export NS_LOG
but somewhere bash appends spaces between lines and instead of having the desired
*=error|warn|prefix_node|prefix_func:PointToPointNetDevice:ClockTest:ClockPerfect:Clock
I have when running $ ./launch_myscript.sh:
*=error|warn|prefix_node|prefix_func :PointToPointNetDevice :ClockTest :ClockPerfect :Clock etc...
My bash --version:
GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
I just saw in the recommended posts this one Bash: Why is echo adding extra space?. How can I prevent NS_LOG from being considered as several arguments? Ultimately the goal is to export that variable.
Your read command is very explicitly treating newlines as data: By clearing IFS and passing -d '', you tell read not to treat whitespace characters as special; since they're not special, they go into the output variable like everything else. However, you can take them out later:
IFS= read -r -d '' NS_LOG <<'EOF'
...content...
EOF
NS_LOG=${NS_LOG//[[:space:]]/} ## replace all whitespace with the empty string
printf '%s\n' "$NS_LOG" ## the quotes are important!
See this snippet run, and its output, at http://ideone.com/fWhzBB.
Notes:
<<'EOF' prevents expansions from occurring within the heredoc itself; with <<EOF, $foo, $(foo), etc. would be special.
<<- only trims leading tab characters, not any other form of whitespace; it's typically safer to do without.
echo $foo string-splits and glob-expands the contents of $foo, passing each word created by this process as a separate argument; echo then places spaces between each argument. echo "$foo" ensures that the entire expansion is treated as a single word. See BashPitfalls #14.
Using echo with nontrivial or unknown data is advised against in the relevant portion of the POSIX specification; printf is the preferred substitute. POSIX echo is explicitly allowed to behave in undefined ways when content contains backslash literals, and the BSD- and AT&T-derived forms of the command are explicitly incompatible, both with each other and with the common GNU implementation (providing an -e flag, which the POSIX spec requires to simply print -e on its output).
things aren't that difficult. here is another solution:
var="`tr -d '[:space:] <<EOF'
your
text
with
lot of spaces
EOF`"
result:
$ echo "$var"
yourtextwithlotofspaces
When trying to use ANSI color escapes from a shell script, I was completely stuck, as the escape sequences (\e) were printed verbatim to the output.
#!/bin/sh
GREEN="\e[32m"
RED="\e[31m"
CLEAR="\e[0m"
printf "${GREEN}test passed${CLEAR}\n"
printf "${RED}test failed${CLEAR}\n"
Produces
\e[32mtest passed\e[0m
\e[31mtest failed\e[0m
\e is not recognized by POSIX sh (as mentioned by honzasp), but \033 is.
GREEN='\033[32m'
CLEAR='\033[0m'
printf "${GREEN}testpassed${CLEAR}\n"
Generally, it's safer to not expand parameters inside the first argument to printf (consider, for example FOO="hello %s"; printf "$FOO bar \n" baz;). However, this requires you to embed an actual escape character in your parameters, rather than a string that printf interprets as an escape character.
GREEN=$(printf '\033[32m')
CLEAR=$(printf '\033[0m')
printf '%stest passed%s' "$GREEN" "$CLEAR"
The solution is to use #!/bin/bash instead of #!/bin/sh in the first line, because raw sh's printf doesn't understand the escapes.
Say I have this command:
printf $text | perl program.pl
How do I guarantee that everything in the $text variable is literally? For example, if $text contains hello"\n, how do I make sure that's exactly what gets passed to program.pl, without the newline or quotation mark (or any conceivable character) being interpreted as a special character?
Quotes!
printf '%s' "$text" | ...
Don't ever expand variables unquoted if you care about preserving their contents precisely. Also, don't ever pass a dynamic string as a format variable when you want it to be treated as literal data.
If you want backslash sequences to be interpreted -- for instance, the two-character sequence \n to be changed to a single newline -- and your shell is bash, use printf '%b' "$text" instead. If you want byte-for-byte accuracy, %s is the Right Thing (and works on any POSIX-compliant shell). If you want escaping for interpretation by another shell (which would be appropriate if, say, you were passing content as part of a ssh command line), then the appropriate format string (for bash only) is %q.