I have read here Difference between single and double quotes in Bash that single quotes will not allow any kind of interpolation. However I came across a situation where single quotes were actually the only way to interpolate my variable' value.
For the sake of an example I created a file named 'test_file' containing
1
2
3
4
5
6
7
8
9
10
Than in the same directory I created a script whose purpose was to extract parts of this file, I called it test.sh and this is its content:
#!/bin/bash
BEGIN=3
END=9
cat test_file | sed -n ''$BEGIN', '$END'p'
Now I tried different setups of the last line like:
cat test_file | sed -n '${BEGIN}, ${END}p'
cat test_file | sed -n '"${BEGIN}", "${END}"p'
cat test_file | sed -n '"$BEGIN", "$END"p'
But none of them worked.
My question is: why does it work that way?
Quotes of both flavors are a shortcut: they cause the enclosed characters to be treated as if they were each escaped by \. The difference between single and double quotes, then, is that double quotes do not escape $ (or following characters which form part of an expansion), allowing parameter and command substitutions to occur (with the results being treated as double-quoted strings as well).
Your first consists of an empty string '', followed by an unquoted expansion, then the literal string ,, the unquoted expansion, and a literal p.
Your second is a single literal sting.
Your third adds literal double quotes around the literal strings ${BEGIN} and ${END}.
The fourth is similar to the third, but drops the literal { and }.
The correct string is
sed -n "$BEGIN, ${END}p" # sed -n "3, 9p"
Related
This question already has answers here:
How to escape single quote in sed?
(9 answers)
Closed 2 years ago.
I want to replace a set of characters with ' using sed.
This post suggest:
With single quotes around the argument (sed 's/…/…/'), use '\'' to put a single quote in the replacement text.
So, I tried following:
echo 'abcd' | sed 's/[abcd]/\'/g'
But it simply ends up expecting more input:
anir#DESKTOP-4856511:~$ echo 'abcd' | sed 's/[abcd]/\'/g'
>
>
>
> ^C
When I copy pasted echo 'abcd' | sed 's/[abcd]/\'/g' in .sh file and ran, it gave me following error:
anir#DESKTOP-4856511:~/Mahesha999/delete$ ./trysed.sh
./trysed.sh: line 1: unexpected EOF while looking for matching `''
./trysed.sh: line 2: syntax error: unexpected end of file
What the right way to do this? Is it impossible to escape single quote inside single quoted string (and I have to use double quotes only as explained here)?
As the post says:
With single quotes around the argument (sed 's/…/…/'), use '\'' to put a single quote in the replacement text.
So, using your example, you would do:
echo 'abcd' | sed 's/[abcd]/'\''/g'
If you want to replace with just one single quote:
echo 'abcd' | sed 's/[abcd][abcd]*/'\''/g'
The shell does not allow single quotes inside a single quoted string. What the code above does is create three strings (which are not separated by anything):
single-quoted string: 's/[abcd]/'
unquoted string containing just an escaped single-quote: \'
single-quoted string: '/g'
The shell then expands them and because they are not separated, they effectively become joined into a single string.
sed sees: s/[abcd]/'/g
I am trying to write a bash script to insert the word Test to any line starting with AlarmName:
Example:
AlarmName: "Blah"
Becomes
AlarmName: "TestBlah"
The double quotes are optional and here is the command I have so far
's/\(AlarmName:\s*("?)\([^"]*\)"?\)/<WHAT GOES HERE>/g' blah.txt > blah2.txt
I am running into two issues, the first is I can't seem to get SED to match the optional quotes. And the second issue is I am not sure how to modify text within a capture group
This should do it:
sed -E 's/^AlarmName:\s*"?/&Test/' file
It's enough to match the prefix part AlarmName: " and replace it with the copy of self (the & part), with Test text appended.
Or, if you prefer a pure POSIX BRE:
sed 's/^AlarmName:[[:space:]]*"\{0,1\}/&Test/' file
You can do the following:
sed 's|^\(AlarmName\s*:\s*\)\("*\)\([^"]*\)\("*\)|\1\2Test\3\4|'
You are breaking the input into four sections (\1, \2, \3 and \4), where:
\1 represents the AlarmName: prefix,
\2 represents the optional quotation,
\3 represents the string to prepend Test to, and
\4 represents the closing brace.
Notice the \'s before each of the group start/end braces (you missed some of those in your attempt above)
I have written a script, which takes first and second parameter strings and the other parameters are files.The idea of the script is to replace the first parameter with the second in every line of every file .Here is my implementation ,however it does not change the content of the files ,but it prints correct information
first=$1
second=$2
shift 2
for i in $*; do
if [ -f $i]; then
sed -i -e 's/$first/$second/g' $i
fi
done
You used a single quote to enclose the sed command. Thus, the special meaning of the dollar sign (parameter expansion) is ignored and it is treated as a simple character.
Check out bash manual:
Enclosing characters in single quotes preserves the literal value of each character within the quotes.
... Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !.
You should replace them with double quotes:
sed -i -e "s/$first/$second/g" $i
Your script doesn't change the files because you are simply just printing to stdout, not to the file. They way you did it, you would need a new variable to store the new content word by word and then echo it to the original file with redirection (>).
But you can do this simply with sed, like this:
sed -i '' 's/original/new/g' file(s)
Explanation:
sed is a stream editor
-i '' means it will edit the current file and won't create any backup
s/original/new/g means substitute original word or regexp with new word or regexp. The g means global = substitute all occurencies, not just the first for every line
file(s) are all the files in which to perform the substitution. Can be * for all files in the working directory.
printf '%s' 'abc' | sed 's/./\\&/g' #1, \a\b\c
printf '%s' "`printf '%s' 'abc' | sed 's/./\\&/g'`" #2, &&&
The expression inside the second backticks returns \a\b\c, and we have printf '%s' "\a\b\c", so it should print \a\b\c.
My question is: why does the second script print &&& ?
note:
I can get the second script work (prints \a\b\c) by prepending each backslash with another backslash, but I don't know why it's needed.
One related question:
why does this single quoted string get interpreted when it's inside of a command substitution
This is a good example to show difference between back-tick and $(cmd) command substitutions.
When the old-style backquoted form of substitution is used, backslash
retains its literal meaning except when followed by "$", "`", or "\".
The first backticks not preceded by a backslash terminates the command
substitution. When using the "$(COMMAND)" form, all characters between
the parentheses make up the command; none are treated specially.
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html
So take a look your example, I used echo instead of printf:
kent$ echo 'abc' | sed 's/./\\&/g'
\a\b\c
kent$ echo -E "`echo 'abc' | sed 's/./\\&/g'`"
&&&
kent$ echo -E "$(echo 'abc' | sed 's/./\\&/g')"
\a\b\c
You can see, the back-tick command substitution made your \\ as single \, thus together with the followed & it became \& (literal &)
Note that I used echo -E in order to disable the interpretation of backslash escapes so that the \a\b\c could be printed out.
Because on the second line :
you are saying:
printf '%s' 'abc' -> 'abc'
Then replace:
'abc'| sed 's/./\\&g' -> &&&
The s mean substitute
. mean one character
\\& by a char &
g mean multiple occurrence on the line
So you are saying:
Replace in abc each char by & multiple time on the same line
Explanation of \\\& :
Two backslashes become a single backslash in the shell which then in sed escapes the forward slash which is the middle delimiter.
\\& -> \& (which makes the forward & a regular character instead of a delimiter)
Three of them: The first two become one in the shell which then escape the third one in sed
\\\& -> \\&
Finally! don't forget that you command is under backquote:
The reason you have to escape it "twice" is because you're entering this command in an environment (such as a shell script) that interprets the double-quoted string once. It then gets interpreted again by the subshell.
From:
Why does sed require 3 backslashes for a regular backslash?
I have a file with a lot of lines, two of them are:
videoId: 'S2Rgr6yuuXQ'
var vid_seq=1;
in a shell script, I have two variables,
for id, the value is always 11 characters/numbers
id='fsafsferii2'
id_seq=80
I want to modify these two lines with id and id_seq
videoId: 'fsafsferii2'
var vid_seq=80;
I used
sed -i 's/\(videoId: \).*\\1'${id}'/\2' file
but there are errors, what is wrong with my script?
thanks
The grep command won't "replace" text, it is for "global regular expression print". But sed will.
sed -i'' '/^videoId: /s/: .*/: '"$id"'/;/^var vid_seq=/s/=.*/='"$id_seq"';/'
I'm not a big fan of inserting variables into sed scripts this way, but sed is simple, and provides no mechanism for actually using actual variables on its own. If you're going to do this, include some format checking for the two variables to make sure they contain the data you want them to contain, before you run this sed script. An accidental / in a variable would cause the sed script to fail.
UPDATE per comments:
Here's a successful test:
$ id=fsafsferii2
$ id_seq=80
$ cat inp686
videoId: 'S2Rgr6yuuXQ'
var vid_seq=1;
$ sed '/^videoId: /s/: .*/: '"$id"'/;/^var vid_seq=/s/=.*/='"$id_seq"';/' < inp686
videoId: fsafsferii2
var vid_seq=80;
$
Of course, you'll need to do some quote magic to get the single quotes into your videoId, but I'm sure you can figure that out yourself.
UPDATE 2
According to sed's man page, the substitute command is in the form:
[2addr]s/regular expression/replacement/flags
The [2addr] means you can specify up to two "addresses", which can be line numbers or regular expressions to match. So the s (substitute) command can take a line, a range, a match, or a span between matches. In our case, we're just using a single match to identify what lines we want to execute the substitution on.
The script above is made up of two sed commands, separated by a semicolon.
/^videoId: / -- Match lines that start with the word videoId:...
s/: .*/: '"$id"'/; -- Substitute all text from the colon to the end of the line with whatever is in the $id environment variable.
/^var vid_seq=/ -- Match lines that ... meh, as above.
s/=.*/='"$id_seq"';/ -- Substitute all text from the equals sign on with $id_seq.
Note that the '"$id"' construct means that we are exiting the single quotes, then immediately entering double quotes for the expansion of the variable ... then exiting the double quotes and going back into a new set of single quotes. Sed scripts are safest inside single quotes because of the frequent use of characters that might be interpreted by a shell.
Note also that because sed's substitute command uses a forward slash as a delimiter, the $id and $id_seq variables may not contain a slash. If they might, you can switch to a different delimiter.
What is wrong with:
sed -i 's/\(videoId: \).*\\1'${id}'/\2' file
Missing the third delimiter (/). Valid syntax is s/regex/replace/
Incorrect regex pattern (let's assume ${id} has been substituted)
\(videoId: \).*\\1fsafsferii2
is telling it to match a string that looks like this:
videoId: anything\1fsafsferii2
(\\ in regex matches literal backslash, so \\1 would match a literal backslash followed by 1 instead of 1st sub-expression)
Replace the matched string with \2
But since there is only one set of parentheses, \2 is actually empty.
Also, since the regex pattern in 2. doesn't match anything, nothing is replaced.
This should work (GNU sed)
sed -i 's/\(videoId: \).*/\1 \x27'${id}'\x27/
s/\(var vid_seq=\).*/\1'${id_seq}'\;/' file
Note:
\x27 is the hexadecimal representation of single quote (to prevent clashing with the other single quote)
\; for literal semicolon. If ; is not escaped, it's interpreted to terminate the s command in sed.