I have the following use case. I need to read from an YAML file using yq v4 PEM keys. It's then important to keep the trailing newlines otherwise a future reading of those PEM keys would miserably fail.
I haven't found a way in Bash to read a PEM from an Yaml file and store it in a variable keeping the trailing newlines.
Naturally if I would use $() Bash would remove the trailing new lines.
Do you have any other idea?
I seriously doubt that you genuinely need to do this (see comments on the question), but using a process substitution to feed input to the read command (configured to expect end-of-input to be signified by a NUL rather than a newline) will work:
IFS='' read -r -d '' input < <(yq ... && printf '\0')
Be sure you check stored contents with echo "$input" or declare -p input, not echo $input. (That's true in the command-substitution case too).
Related
A script save.sh uses 'cp' and outputs its cp errors to an errors file. Mostly these errors are due to the origin filesystem being EXT4 and the destination filesystem being NTFS or FAT and doesnt accept some specia characters.
Another script onerrors.sh reads the error file so as to best manage files that could not be copied : it copies them toward a crafted filename file that's OK for FAT and NTFS = where "bad" characters have been replaced with '_'.
That works fine for allmost all of the errors, but not 100%.
In this error file, the special characters in the filenames seem to be multiple times escaped :
simple quotes ' appear as '\''.
\n (real newline in filenames!) appear as '$'\n'' (7 glyphs !)
I want to unescape these so as to get the filename.
I convert quotes back to ' with line=${line//\'\\\'\'/\'}. That's OK.
But how can i convert the escaped newline back to a real unescaped \n in $line variable = how can i replace the '$'\n'' to unescaped \n in variable ?
The issue is not in recognising the pattern but in inserting a real newline. I've not been able to do it using same variable expansion syntax.
What other tool is advised or other way of doing it ?
The question is:
how can i replace the '$'\n'' to unescaped \n in variable ?
That's simple:
var="def'$'\n''abc"
echo "${var//\'$\'\\n\'\'/$'\n'}"
I think I remember, that using ANSI C quoting inside variable expansion happened to be buggy in some version of bash. Use a temporary variable in such cases.
What other tool is advised or other way of doing it ?
For string replacement in shell, the most popular tools are sed (which the name literally comes from "String EDitor") and awk. Writing a parser is better done in full-blown programming languages, like Python, C, C++ and similar.
The only way to decode cp output correctly, is to see cp source code, see how it quotes the filenames, and decode it in the same way. Note that it may change between cp flavors and versions, so such a tool may need to query cp version and will be not portable.
Note that parsing cp output is a very very very very bad idea. cp output is in no way standardized, may change anytime and is meant for humans to read. Instead, strongly consider rewriting save.sh to copy file by file and in case of cp returning non-zero exit status, write the filename yourself in an "errors file" as a zero separated stream.
# save.sh
find .... -print0 |
while IFS= read -d '' -r file; do
if ! cp "$file" "$dst"; then
printf "%s\0" "$file" > errorsfile
fi
done
# onerrors.sh
while IFS= read -d '' -r file; do
echo "do something with $file"
done < errorsfile
I have file a.txt with following content
aaa
bbb
When I execute following script:
while read line
do
echo $line
done < a.txt > b.txt
generated b.txt contains following
aaa
bbb
It is seen that the leading spaces of lines have got removed. How can I preserve leading spaces?
This is covered in the Bash FAQ entry on reading data line-by-line.
The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared:
# Exact lines, no trimming
while IFS= read -r line; do
printf '%s\n' "$line"
done < "$file"
As Charles Duffy correctly points out (and I'd missed by focusing on the IFS issue); if you want to see the spaces in your output you also need to quote the variable when you use it or the shell will, once again, drop the whitespace.
Notes about some of the other differences in that quoted snippet as compared to your original code.
The use of the -r argument to read is covered in a single sentence at the top of the previously linked page.
The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the -r option with read.
As to using printf instead of echo there the behavior of echo is, somewhat unfortunately, not portably consistent across all environments and the differences can be awkward to deal with. printf on the other hand is consistent and can be used entirely robustly.
There are several problems here:
Unless IFS is cleared, read strips leading and trailing whitespace.
echo $line string-splits and glob-expands the contents of $line, breaking it up into individual words, and passing those words as individual arguments to the echo command. Thus, even with IFS cleared at read time, echo $line would still discard leading and trailing whitespace, and change runs of whitespace between words into a single space character each. Additionally, a line containing only the character * would be expanded to contain a list of filenames.
echo "$line" is a significant improvement, but still won't correctly handle values such as -n, which it treats as an echo argument itself. printf '%s\n' "$line" would fix this fully.
read without -r treats backslashes as continuation characters rather than literal content, such that they won't be included in the values produced unless doubled-up to escape themselves.
Thus:
while IFS= read -r line; do
printf '%s\n' "$line"
done
Problem
I'm trying to use a variable when calling a cURL command but it's including the the literal $line instead of the actual value.
while read line; do
curl "https://x.com/v1/PhoneNumbers/$line?Type=carrier" -u "x:x"
done < "${1:-/dev/stdin}"
Context
I'm passing a list of numbers to the script trying to read them line by line.
From the comments we know that you're getting the following error:
curl: (3) Illegal characters found in URL
If formatted this way:
while IFS="$IFS"$'\r' read line; do
curl "https://x.com/v1/PhoneNumbers/$line?Type=carrier" -u "x:x"
done < "${1:-/dev/stdin}"
your command should work.
The problem is that your appending \r at the end of your input lines (so that every line of your input ends with a \r\n sequence). By default read does not strip trailing r. If we want read to trim those characters we have to add this character to the IFS environmental variable for read like this: IFS="$IFS"$'\r')" read ....
Here's a great comment from Charles Duffy:
Personally, I'd suggest IFS=$' \t\n\r', not referring to the old $IFS value -- why make your code's behavior contextually dependent?
Another valuable comment; this time from chepner:
Granted, a valid line probably isn't going to contain a \r, but conceptually, you don't want to treat a carriage return as whitespace; you just want to strip the \r that is part of the \r\n line ending. Instead of modifying IFS, read the line normally, then strip it with line=${line%$'\r'} before calling curl.
Related:
curl: (3) Illegal characters found in URL : ${...%?} doesn't work
Why is a shell script giving syntax errors when the same code works elsewhere?
You try to address a variable and a text right after it, you should use ${VAR} syntax:
curl "https://x.com/v1/PhoneNumbers/${line}?Type=carrier" -u "x:x" done < "${1:-/dev/stdin}"
The reason is bash tries to expand the parameter $line?Type .. instead of $line.
For example
[root# ~]# line=1234
[root# ~]# echo $line567
[root# ~]# echo ${line}567
1234567
I have file a.txt with following content
aaa
bbb
When I execute following script:
while read line
do
echo $line
done < a.txt > b.txt
generated b.txt contains following
aaa
bbb
It is seen that the leading spaces of lines have got removed. How can I preserve leading spaces?
This is covered in the Bash FAQ entry on reading data line-by-line.
The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared:
# Exact lines, no trimming
while IFS= read -r line; do
printf '%s\n' "$line"
done < "$file"
As Charles Duffy correctly points out (and I'd missed by focusing on the IFS issue); if you want to see the spaces in your output you also need to quote the variable when you use it or the shell will, once again, drop the whitespace.
Notes about some of the other differences in that quoted snippet as compared to your original code.
The use of the -r argument to read is covered in a single sentence at the top of the previously linked page.
The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the -r option with read.
As to using printf instead of echo there the behavior of echo is, somewhat unfortunately, not portably consistent across all environments and the differences can be awkward to deal with. printf on the other hand is consistent and can be used entirely robustly.
There are several problems here:
Unless IFS is cleared, read strips leading and trailing whitespace.
echo $line string-splits and glob-expands the contents of $line, breaking it up into individual words, and passing those words as individual arguments to the echo command. Thus, even with IFS cleared at read time, echo $line would still discard leading and trailing whitespace, and change runs of whitespace between words into a single space character each. Additionally, a line containing only the character * would be expanded to contain a list of filenames.
echo "$line" is a significant improvement, but still won't correctly handle values such as -n, which it treats as an echo argument itself. printf '%s\n' "$line" would fix this fully.
read without -r treats backslashes as continuation characters rather than literal content, such that they won't be included in the values produced unless doubled-up to escape themselves.
Thus:
while IFS= read -r line; do
printf '%s\n' "$line"
done
I try to read /dev/stdin using command read. But part of lines are not reads
What I do wrong?
Lines are reads to symbol ^#
#!/bin/bash
echo "Content-Type: text/plain"
echo
IFS=$'\n'
while read -d $'\n'
do
echo $REPLY
done
POST http://79.135.220.173/post.png
RESPONSE http://79.135.220.173/response.png
^# is a legible representation of the ASCII null character.
C uses the null character as a string terminator, and bash is implemented in C.
Most likely bash's read function reads the entire line into a buffer which is assigned to $REPLY. When that buffer is passed to the built-in echo command, it treats the null character as a line terminator.
For the code in your question, you could use the cat command rather than using bash's read to read a line at a time. (cat, at least the GNU coreutils version, doesn't have a problem with null characters):
#!/bin/bash
echo "Content-Type: text/plain"
echo
cat
The final cat command simply copies stdin to stdout.
If you need to do some processing on the input before printing it, you might consider using Perl rather than bash for this script; Perl strings, unlike C strings, can contain embedded null characters.
Or can you arrange for the input not to contain any null characters?
read will read from stdin (or whatever descriptor you specify with -u) until it encounters a newline character (or whatever you specified with -d), splitting what is read on the characters in IFS and assigning each token to the variables you specify, or to REPLY if none were specified. The final variable always gets all remaining input that was read.
So, since you did not specify -d then REPLY should contain, for you, all input up to the first newline character. This will happen once per loop, so your script should read and echo every line that is sent to it and never terminate.