I have a variable with some lines in it and I would like to pad it with a number of newlines defined in another variable. However it seems that the subshell may be stripping the trailing newlines. I cannot just use '\n' with echo -e as the lines may already contain escaped chars which need to be printed as is.
I have found I can print an arbitrary number of newlines using this.
n=5
yes '' | sed -n "1,${n}p;${n}q"
But if I run this in a subshell to store it in the variable, the subshell appears to strip the trailing newlines.
I can approximate the functionality but it's clumsy and due to the way I am using it I would much rather be able to just call echo "$var" or even use $var itself for things like string concatenation. This approximation runs into the same issue with subshells as soon as the last (filler) line of the variable is removed.
This is my approximation
n=5
var="test"
#I could also just set n=6
cmd="1,$((n+1))p;$((n+1))q"
var="$var$(yes '' | sed -n $cmd; echo .)"
#Now I can use it with
echo "$var" | head -n -1
Essentially I need a good way of appending a number of newlines to a variable which can then be printed with echo.
I would like to keep this POSIX compliant if at all possible but at this stage a bash solution would also be acceptable. I am also using this as part of a tool for which I have set a challenge of minimizing line and character count while maintaining readability. But I can work that out once I have a workable solution
Command substitutions with either $( ) or backticks will trim trailing newlines. So don't use them; use the shell's built-in string manipulation:
n=5
var="test"
while [ "$n" -gt 0 ]; do
var="$var
"
n=$((n-1))
done
Note that there must be nothing after the var="$var (before the newline), and nothing before the " on the next line (no indentation!).
A sequence of n newlines:
printf -v spaces "%*s" $n ""
newlines=${spaces// /$'\n'}
Related
I am trying to create a bash script that uses the sed command to replace a pattern by a variable that contains a string or put a space if there is nothing in the variable. I cannot find the good way to write it and make it work. Here is the part where I have issues:
a_flag=$(echo $a | wc -w)
if [[ $a_flag = 0 ]]; then
sed -i -e 's/#b/\\hspace{2cm}/g' source.tex
else
sed -i -e "s/#b/$a/g" source.tex
fi
When running this, the condition is always false. I tried [] or (()) for the if statement but I just can't find a way to fix it.
You only need a single parameter expansion here, to replace the expansion of $a with \hspace{2cm} if the expansion is empty.
sed -i -e "s/#b/${a:-\\\\hspace{2cm}}/g" source.tex
You need a stack of \ because there are two rounds of escaping involved. First, the shell itself reduces each \\ to a single literal backslash. Then sed also reduces each pair of \\ to a single literal backslash.
Counting the number of occurrences of someething seems like a very roundabout way to approach this anyway.
case $a in
*[A-Za-z0-9_]*) x='\\hspace{2cm}';;
*) x=$a;;
esac
sed -i "s/#b/$x/g" source.tex
I want to prepend a string to all the files in a directory. What I want to do is something like:
echo string{$(ls some_dir)}
This won't work because ls separates words with spaces, and brace expansion requires commas. So I thought I'd use tr to replace the spaces with commas, like:
echo string{$(ls some_dir) | tr ' ' ','}
But that doesn't work either because the pipe takes precedence.
What's the correct way to do this? I know I could probably use a sane language like Python, but it's frustrating that Bash can't even do something as simple as that.
If you really want to interpolate the contents of a directory (which is what $(ls some_dir) would give you) then you can do
printf 'string%s ' some_dir/*
IRL, you probably want it to end with a newline.
{ printf 'string%s ' some_dir/*; echo; }
You can generalize this to the output of any glob or brace expansion:
printf 'foo%d\n' {11..22}
Edit
Based on your comment, you want to eliminate the "some_dir/" part, you can't merely do that with printf. You can either cd to the directory so the globs expand as desired, or use parameter expansion to clean up the leading directory name:
( cd some_dir && printf 'string%s ' *; echo )
or
{ cd some_dir && printf 'string%s ' * && cd - >/dev/null; echo; }
or
names=( some_dir/* ) names=( "${names[#]#some_dir/}" )
{ printf 'string%s ' "${names[#]}"; echo; }
One way to do it, which will deal gracefully with whitespace in filenames:
files=("$dir"/*); files=("${files[#]/#"$dir"\//"$prefix"}")
That will store the prefixed strings in the array $files; you could iterate over them using an array expansion:
for file in "${files[#]}"; do
# Something with file
done
or print them out using printf:
printf "%s\n" "${files[#]}"
The advantage of using the array expansion is that it does not involve word-splitting, so even if the elements have whitespace in them, the array expansion will contain each element as a single word.
Of course, bash can do it.
Let's go step by step.
1. fix an issue in your second example
This is your second example
echo string{$(ls some_dir) | tr ' ' ','}
You put pipe outside the command substitution, which is totally wrong.
I believe you want to pipe the stream from ls output to tr input, so it's obvious that the pipe is supposed to be put inside the command substitution, like this
echo string{$(ls some_dir | tr ' ' ',')}
2. output of ls is separated by newline rather than whitespace
so here we go
echo string{$(ls some_dir | tr '\n' ',')}
3. brace expansion is performed prior to command substitution
In the other word, after command substitution is expanded to f1,f2,f3,d1,, the brace expansion will not be performed any more.
So, no doubt, the command will print string{f1,f2,f3,d1,}.
The solution is letting bash evaluate it again.
eval echo string{$(ls some_dir | tr '\n' ',')}
OK, up to now, the result looks very good (try it yourself, you'll get it), it is very close to what you were looking for, except one tiny spot.
You may already noticed the comma at the end of the output I demonstrated above. The comma results in an unnecessary string appearing at the end of the final output.
So let's make it done.
4. remove the ending comma
eval echo string{$(echo -n "$(ls some_dir)" | tr '\n' ',')}
OK, this is it.
Oh... BTW., this is just an specific solution for your specific question. You may develop new variants of your question, and this specific solution may not fit your new question. If so, I suggest you run man bash, and read it from head to toe very very carefully, then you will become unstoppable.
I have a text file with a list of filenames. I would like to create a variable from a specific line number using AWK. I get the correct output using:
awk "NR==\$Line" /myPath/fileList.txt
I want to assign this output to a variable and from documentation I found I expected the following to work:
INFILE=$(awk "NR==\$Line" /myPath/fileList.txt)
or
INFILE=`awk "NR==\$Line" /myPath/fileList.txt`
However,
echo "\$INFILE"
is blank. I am new to bash scripting and would appreciate any pointers.
The output of the AWK command is assigned to the variable. To see the contents of the variable, do this:
echo "$INFILE"
You should use single quotes for your AWK command so you don't have to escape the literal dollar sign (the literal string should be quoted, see below if you want to substitute a shell variable instead):
awk 'NR == "$Line"' /myPath/fileList.txt
The $() form is much preferred over the backtick form (I don't understand why you have the backticks escaped, by the way). Also, you should habitually use lowercase or mixed case variable names to avoid name collision with shell or environment variables.
infile=$(awk 'NR == "$Line"' /myPath/fileList.txt)
echo "$infile"
If your intention is that the value of a variable named $Line should be substituted rather than the literal string "$Line" being used, then you should use AWK's -v variable passing feature:
infile=$(awk -v "line=$Line" 'NR == line' /myPath/fileList.txt)
Don't mask the dollar sign.
wrong:
echo "\$INFILE"
right:
echo $INFILE
echo ${INFILE}
echo "$INFILE"
echo "${INFILE}"
The ${x} - construct is useful, if you like to glue texts together.
echo $INFILE0
will look for a Variable INFILE0. If $INFILE is 5, it will not produce "50".
echo ${INFILE}0
This will produce 50, if INFILE is 5.
The apostrophes are useful if you variable contains whitespace and for more or less unpredictable text.
If your rownumber is a parameter:
#!/bin/bash
Line=$1
INFILE=$(awk "NR==$Line" ./demo.txt)
echo "$INFILE"
If INFILE contains multiple spaces or tabs, echo $INFILE would condense them to single spaces, while "$INFILE" preserves them.
I know it is possible to invert grep output with the -v flag. Is there a way to only output the non-matching part of the matched line? I ask because I would like to use the return code of grep (which sed won't have). Here's sort of what I've got:
tags=$(grep "^$PAT" >/dev/null 2>&1)
[ "$?" -eq 0 ] && echo $tags
You could use sed:
$ sed -n "/$PAT/s/$PAT//p" $file
The only problem is that it'll return an exit code of 0 as long as the pattern is good, even if the pattern can't be found.
Explanation
The -n parameter tells sed not to print out any lines. Sed's default is to print out all lines of the file. Let's look at each part of the sed program in between the slashes. Assume the program is /1/2/3/4/5:
/$PAT/: This says to look for all lines that matches pattern $PAT to run your substitution command. Otherwise, sed would operate on all lines, even if there is no substitution.
/s/: This says you will be doing a substitution
/$PAT/: This is the pattern you will be substituting. It's $PAT. So, you're searching for lines that contain $PAT and then you're going to substitute the pattern for something.
//: This is what you're substituting for $PAT. It is null. Therefore, you're deleting $PAT from the line.
/p: This final p says to print out the line.
Thus:
You tell sed not to print out the lines of the file as it processes them.
You're searching for all lines that contain $PAT.
On these lines, you're using the s command (substitution) to remove the pattern.
You're printing out the line once the pattern is removed from the line.
How about using a combination of grep, sed and $PIPESTATUS to get the correct exit-status?
$ echo Humans are not proud of their ancestors, and rarely invite
them round to dinner | grep dinner | sed -n "/dinner/s/dinner//p"
Humans are not proud of their ancestors, and rarely invite them round to
$ echo $PIPESTATUS[1]
0[1]
The members of the $PIPESTATUS array hold the exit status of each respective command executed in a pipe. $PIPESTATUS[0] holds the exit status of the first command in the pipe, $PIPESTATUS[1] the exit status of the second command, and so on.
Your $tags will never have a value because you send it to /dev/null. Besides from that little problem, there is no input to grep.
echo hello |grep "^he" -q ;
ret=$? ;
if [ $ret -eq 0 ];
then
echo there is he in hello;
fi
a successful return code is 0.
...here is 1 take at your 'problem':
pat="most of ";
data="The apples are ripe. I will use most of them for jam.";
echo $data |grep "$pat" -q;
ret=$?;
[ $ret -eq 0 ] && echo $data |sed "s/$pat//"
The apples are ripe. I will use them for jam.
... exact same thing?:
echo The apples are ripe. I will use most of them for jam. | sed ' s/most\ of\ //'
It seems to me you have confused the basic concepts. What are you trying to do anyway?
I am going to answer the title of the question directly instead of considering the detail of the question itself:
"grep a pattern and output non-matching part of line"
The title to this question is important to me because the pattern I am searching for contains characters that sed will assign special meaning to. I want to use grep because I can use -F or --fixed-strings to cause grep to interpret the pattern literally. Unfortunately, sed has no literal option, but both grep and bash have the ability to interpret patterns without considering any special characters.
Note: In my opinion, trying to backslash or escape special characters in a pattern appears complex in code and is unreliable because it is difficult to test. Using tools which are designed to search for literal text leaves me with a comfortable 'that will work' feeling without considering POSIX.
I used both grep and bash to produce the result because bash is slow and my use of fast grep creates a small output from a large input. This code searches for the literal twice, once during grep to quickly extract matching lines and once during =~ to remove the match itself from each line.
while IFS= read -r || [[ -n "$RESULT" ]]; do
if [[ "$REPLY" =~ (.*)("$LITERAL_PATTERN")(.*) ]]; then
printf '%s\n' "${BASH_REMATCH[1]}${BASH_REMATCH[3]}"
else
printf "NOT-REFOUND" # should never happen
exit 1
fi
done < <(grep -F "$LITERAL_PATTERN" < "$INPUT_FILE")
Explanation:
IFS= Reassigning the input field separator is a special prefix for a read statement. Assigning IFS to the empty string causes read to accept each line with all spaces and tabs literally until end of line (assuming IFS is default space-tab-newline).
-r Tells read to accept backslashes in the input stream literally instead of considering them as the start of an escape sequence.
$REPLY Is created by read to store characters from the input stream. The newline at the end of each line will NOT be in $REPLY.
|| [[ -n "$REPLY" ]] The logical or causes the while loop to accept input which is not newline terminated. This does not need to exist because grep always provides a trailing newline for every match. But, I habitually use this in my read loops because without it, characters between the last newline and the end of file will be ignored because that causes read to fail even though content is successfully read.
=~ (.*)("$LITERAL_PATTERN")(.*) ]] Is a standard bash regex test, but anything in quotes in taken as a literal. If I wanted =~ to consider the regex characters in contained in $PATTERN, then I would need to eliminate the double quotes.
"${BASH_REMATCH[#]}" Is created by [[ =~ ]] where [0] is the entire match and [N] is the contents of the match in the Nth set of parentheses.
Note: I do not like to reassign stdin to a while loop because it is easy to error and difficult to see what is happening later. I usually create a function for this type of operation which acts typically and expects file_name parameters or reassignment of stdin during the call.
I get:
$ echo -e "D"{a,b,c}".jpg\n"
Da.jpg
Db.jpg
Dc.jpg
Note: The extra spaces before Db and Dc on the 2nd and 3rd line of the output.
Why are these there?
Thanks,
Dan
Edit: Since my actual objective had spaces in it (which I should have written originally):
echo -e "Name"{,.}" "{-,}"extra"{,so}" 5v5 "{one,two,No\ four}{,!,\!\!}"\n"
Most solutions here didn't work for me (for loop, xarg, tr). Printf didn't work because of multiple braces expansions that I want to cantesian product.
I combined 3 solutions (mletterle's \b, Dennis Williamson's extra space, and Jim Dennis's using far less quotes) to get:
echo -e "" \\bName{,.}\ {-,}extra{,so}\ 5v5\ {one,two,No\ four}{,\!,\!\!}\\n
Thanks all who answered! I learned a lot from your responses!
Dan
use the more portable printf
$ printf "D%s.jpg\n" {a,b,c}
Da.jpg
Db.jpg
Dc.jpg
Because that's what brace expansion does. From man bash, under the heading Brace expansion:
Patterns to be brace expanded take the
form of an
optional preamble, followed by ... a series of comma-separated
strings ... followed by an optional
postscript. The preamble is prefixed
to each string contained within the
braces, and the postscript is then appended to each resulting
string, expanding left to right
For example, a{d,c,b}e expands into
‘ade ace abe’
So in your example, "D" is the preamble and ".jpg\n" is the postscript.
So, after brace expansion occurs, you're left with:
echo -e Da.jpg\n Db.jpg\n Dc.jpg\n
As hewgill points out, the shell then splits this into three tokens and passes them to echo; which outputs each token separated by a space. To get the output you want, you need to use one of the many suggestions here that don't re-inserted the unwanted space between tokens.
It's longer and probably not the neatest way to do this, but the following gives the output you're after:
for file in "D"{a,b,c}".jpg"
do
echo ${file}
done
echo always adds spaces between arguments. Try your command without \n and compare the results.
The easiest and cleanest solution is to add a backspace to the front of each line:
echo -e -n "\bD"{a,b,c}".jpg\n"
This produces the desired output.
You can get the desired effect by using xargs to separate the arguments spit by the first echo into a line each:
$ echo "D"{a,b,c}".jpg" | xargs -n1 echo
Da.jpg
Db.jpg
Dc.jpg
You can get a more consistent look by prepending a null:
$ echo -en "" "D"{a..c}".jpg\n"
Da.jpg
Db.jpg
Dc.jpg
Now they all have an extra space. Also, using -n eliminates the extra newline at the end. Also, you can use a range in your brace expansion.
Here is a solution using sed (that builds upon https://stackoverflow.com/a/2003856/8180143):
$ echo -en "" "D"{a..c}".jpg\n" | sed 's/ //'
Da.jpg
Db.jpg
Dc.jpg
This solution has the advantage of working with inputs having spaces, e.g.,
$ echo -en "" "D "{a..c}".jpg\n" | sed 's/ //'
D a.jpg
D b.jpg
D c.jpg