I'm building a shell script with echo. I have something like:
echo "sed -i \"\\|charlie\.url\\s*=\\s*.*|c\\charlie.url = ${CHARLIE_URL}\" foo.conf" >> bar.sh
i.e. replace the line in foo.conf containing the current charlie.url (not necesarily at the begining, 'cause the line could be commented) for a new line with a new url.
I would expect the output to bar.sh to be
sed -i "\|charlie\.url\s*=\s*.*|c\charlie.url = ${CHARLIE_URL}" foo.conf
Nevertheless, the c\\charlie is interpreted as c \c harlie, instead of
c\ charlie, which generates the following output:
sed -i "\|charlie\.url\s*=\s*.*|c
I have found that I could prevent this by using single instead of doubles quotes, but in that case ${CHARLIE_URL} (which I do need to expand) does not get expanded.
How should my echo argument look like?
I'm using dash (#!/bin/sh under Ubuntu), but I could also use bash or zsh.
Instead of echo, you can try cat :
cat << EOF >> bar.sh
sed -i "\|charlie\.url\s*=\s*.*|c\charlie.url = ${CHARLIE_URL}" foo.conf
EOF
#!/bin/bash
# bash will expand content within "" and stuff like $URL gets expanded
# also double-backslash reduces to a single; bash wouldn't expand stuff within ''
URL="https..."
echo double: "c\\charlie.url, URL $URL"
echo single: 'c\\charlie.url, URL $URL'
# if you need to output a \\
echo "\\\\"
The command line is using your first slash to quote your second one. If CHARLIE_URL=foo, your echo is actually outputting
sed -i "\|charlie\.url\s*=\s*.*|c\charlie.url = foo" foo.conf
Try using single quotes, but close/open them around the variable.
echo 'sed -i "\\|charlie\.url\\s*=\\s*.*|c\\charlie.url = '"${CHARLIE_URL}"'" foo.conf'
This produces
sed -i "\\|charlie\.url\\s*=\\s*.*|c\\charlie.url = foo" foo.conf
You could also, as mentioned, quote the slash that quotes the slash, and then quote the slash being quoted by the other slash, so that the subsequent iterations boil them down to what you want... but that's generally a mess, and leads to leaning-toothpick syndrome.
Someone made a good here-doc suggestion already, so I won't repeat that, but you can also use a "here string" which I generally prefer.
cat <<< "sed -i '\|charlie\.url\s*=\s*.*|c\charlie.url = ${CHARLIE_URL}' foo.conf"
I switched the double-quotes in your script for singles, assuming what you want is to output the literal value of $CHARLIE_URL into your script. If you want the script to use the variable, with whatever value is assigned at runtime, then quote that too -
cat <<< "sed -i '\|charlie\.url\s*=\s*.*|c\charlie.url = \${CHARLIE_URL}' foo.conf"
Related
I am trying to find and replace a specific text content using the sed command and to run it via a shell script.
Below is the sample script that I am using:
fp=/asd/filename.txt
fd="sed -i -E 's ($2).* $2:$3 g' ${fp}"
eval $fd
and executing the same by passing the arguments:
./test.sh update asd asdfgh
But if the argument string contains $ , it breaks the commands and it is replacing with wrong values, like
./test.sh update asd $apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.
How can I make sure that the values inside the variables are not expanded because of the $?
Updated
sh file test.sh
set -xv
fp="/asd/filename.txt"
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
text file filename.txt
hello:world
Outputs
1)
./test.sh update hello WORLD
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:WORLD/g' /asd/filename.txt
2)
./test.sh update hello '$apr1$hosgaxyv$D0KXp5dCyZ2BUYCS9BmHu1'
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:'\''$'\''apr1'\''$'\''hosgaxyv'\''$'\''D0KXp5dCyZ2BUYCS9BmHu1/g' /asd/filename.txt
In both the case , its not replacing the content
You don't need eval here at all:
fp=/asd/filename.txt
sed -i -E "s/(${2//'$'/'\$'}).*/\1:${3//'$'/'\$'}/g" "$fp"
The whole sed command is in double quotes so variables can expand.
I've replaced the blank as the s separator with / (doesn't really matter in the example).
I've used \1 to reference the first capture group instead of repeating the variable in the substitution.
Most importantly, I've used ${2//'$'/'\$'} instead of $2 (and similar for $3). This escapes every $ sign as \$; this is required because of the double quoting, or the $ get eaten by the shell before sed gets to see them.
When you call your script, you must escape any $ in the input, or the shell tries to expand them as variable names:
./test.sh update asd '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.'
Put the command-line arguments that are filenames in single quotes:
./test.sh update 'asd' '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1'
must protect all the script arguments with quotes if having space and special shell char, and escape it if it's a dollar $, and -Ei instead of -iE even better drop it first for test, may add it later if being really sure
I admit i won't understant your regex so let's just get in the gist of solution, no need eval;
fp=/asd/filename.txt
sed -Ei "s/($2).*/$2:$3/g" $fp
./test.sh update asd '\$apr1\$HnIF6bOt\$9m3NzAwr.aG1Yp.t.bpIS1.'
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
Someone with better experience then me regarding bashscript that could take a look at this code and explain why no string replacements is taking place. If I run the same command (text input instead of variables) in the terminal, then it works.
#!/bin/bash
echo "important to escape every \"/\" character"
read -p "Specify the old string you want to replace? (from) " FROM
read -p "Specify the new string you want to use instead? (to) " TO
cp ../backup/mysql/dump.sql ../backup/mysql/dump.sql.backup.$(date +"%Y-%m-%d-%H-%M-%S") \
&& sed -i 's/$FROM/$TO/g' ../backup/mysql/dump.sql
Use double quotes for variables expansion instead of single quotes: "s/$FROM/$TO/g"
You can even have the cp implicitly within the sed command itself using -i(inplace) option.
sed -i.backup.$(date +"%Y-%m-%d-%H-%M-%S") "s/$FROM/$TO/g" ../backup/mysql/dump.sql
This will create the backup file as well.
Running this statement in OS X Terminal
for i in `ls -v *.mkv`; do echo $i; done
will successfully print out all the file names in the directory in name order with each file name on its own line.
Source: This StackOverFlow answer
However, if I run this statement in OS X Terminal
for i in 'ls -v *.mkv'; do echo $i; done
the output is "ls -v fileName1.mkv fileName2.mkv", etc. with all the file names concatenated into one long line (as opposed to each being printed on its own line).
My questions are:
What's the difference between ` and ' in bash?
Why is that difference responsible for the completely different output?
What keyboard combination produces `? (Keyboard combination)
1) Text between backticks is executed and replaced by the output of the enclosed command, so:
echo `echo 42`
Will expand to:
echo 42
This is called Command Substitution and can also be achieved using the syntax $(command). In your case, the following:
for i in `ls -v *.mkv`; do ...
Is replaced by something like (if your directory contains 3 files named a.mkv, b.mkv and c.mkv):
for i in a.mkv b.mkv c.mkv; do ...
Text between quotes or double quotes are just plain Bash strings with characters like space scaped inside them (there are other ways to quote strings in Bash and are described here):
echo "This is just a plain and simple String"
echo 'and this is another string'
A difference between using ' and " is that strings enclosed between " can interpolate variables, for example:
variable=42
echo "Your value is $variable"
Or:
variable=42
echo "Your value is ${variable}"
Prints:
Your value is 42
2) Wildcard expressions like *.mkv are replaced by the expanded filenames in a process known as Globbing. Globbing is activated using wildcards in most of the commands without enclosing the expression inside a string:
echo *.mkv
Will print:
a.mkv b.mkv c.mkv
Meanwhile:
echo "*.mkv"
prints:
*.mkv
The i variable in your for loop takes the value "ls -v *.mkv" but the echo command inside the loop body takes $i without quotes, so Bash applied globbing there, you end up with the following:
for i in 'ls -v *.mkv'; do
# echo $i
#
# which is expanded to:
# echo ls -v *.mkv (no quotes)
#
# and the globbing process transform the above into:
echo ls -v a.mkv b.mkv c.mkv
Which is just a one-line string with the file names after the globbing is applied.
3) It depends on your keyboard layout.
One trick to keep the character around is to use the program ascii, search for the character 96 (Hex 60), copy it and keep it on your clipboard (you can use parcellite or any other clipboard manager that suits your needs).
Update: As suggested by #triplee, you should check useless use of ls as this is considered a bash pitfall and there are better ways to achieve what you're trying to do.
'expression', will output the exact string in expression.
`expression`, will execute the content of the expression and echo outputs it.
For example:
x="ls"
echo "$x" --> $x
echo `$x` --> file1 file2 ... (the content of your current dir)
Backticks mean "run the thing between the backticks as a command, and then act as if I had typed the output of that command here instead". The single quotes mean, as others have said, just a literal string. So in the first case, what happens is this:
bash runs ls -v *.mkv as a command, which outputs something like:
fileName1.mkv
fileName2.mkv
bash then substitutes this back into where the backtick-surrounded command was, i.e. it effectively makes your for statement into this:
for i in fileName1.mkv fileName2.mkv; do echo $i; done
That has two "tokens": "fileName1.mkv" and "fileName2.mkv", so the loop runs its body (echo $i) twice, once for each:
echo fileName1.mkv
echo fileName2.mkv
By default, the echo command will output a newline after it finishes echoing what you told it to echo, so you'll get the output you expect, of each filename on its own line.
When you use single quotes instead of backticks, however, the stuff in between the single quotes doesn't get evaluated; i.e. bash doesn't see it as a command (or as anything special at all; the single quotes are telling bash, "this text is not special; do not try to evaluate it or do anything to it"). So that means what you're running is this:
for i in 'ls -v *.mkv'; do echo $i; done
Which has only one token, the literal string "ls -v *.mkv", so the loop body runs only once:
echo ls -v *.mkv
...but just before bash runs that echo, it expands the "*.mkv".
I glossed over this above, but when you do something like ls *.mkv, it's not actually ls doing the conversion of *.mkv into a list of all the .mkv filenames; it's bash that does that. ls never sees the *.mkv; by the time ls runs, bash has replaced it with "fileName1.mkv fileName2.mkv ...".
Similarly for echo: before running this line, bash expands the *.mkv, so what actually runs is:
echo ls -v fileName1.mkv fileName2.mkv
which outputs this text:
ls -v fileName1.mkv fileName2.mkv
(* Footnote: there's another thing I've glossed over, and that's spaces in filenames. The output of the ls between the backticks is a list of filenames, one per line. The trouble is, bash sees any whitespace -- both spaces and newlines -- as separators, so if your filenames are:
file 1.mkv
file 2.mkv
your loop will run four times ("file", "1.mkv", "file", "2.mkv"). The other form of the loop that someone mentioned, for i in *.mkv; do ... doesn't have this problem. Why? Because when bash is expanding the "*.mkv", it does a clever thing behind the scenes and treats each filename as a unit, as if you'd said "file 1.mkv" "file 2.mkv" in quotes. It can't do that in the case where you use ls because after it passes the expanded list of filenames to ls, bash has no way of knowing that what came back was a list of those same filenames. ls could have been any command.)
I would like to do that:
i="1"; echo -e '#!/usr/bin/env bash\nmyprogram -i "input_${i}.txt"'
and pipe it to a job scheduler.
However, this doesn't replace the variable i by its value. Instead, I obtain this:
#!/usr/bin/env bash
myprogram -i "input_${i}.txt"
I played a bit with option -e of echo and with single-/double-quote but could not make it work. For instance, I get this:
i="1"; echo -e "#!/usr/bin/env bash\nmyprogram -i \"input_${i}.txt\""
-bash: !/usr/bin/env: event not found
My bash version is 4.1.2.
Try this:
i="1"; echo -e '#!/usr/bin/env bash\nmyprogram -i '"\"input_${i}.txt\""
You can echo single- and double-quoted strings at the same time.
Try also escaping the exclamation mark:
\! should be okay, and will not be read as an "event" by bash.
i="1"; echo -e '#!/usr/bin/env bash\nmyprogram -i "input_'"${i}"'.txt"'
Basically, use single quotes until you need to interpolate, then close the single quotes, open the double quotes, add the interpolation, close the double quotes, reopen single quotes, and finish the string. In the shell, quotation marks don't delimit a word; they just change the interpretation of the part of a word falling between them.