This question already has answers here:
Difference between single and double quotes in Bash
(7 answers)
Closed 2 years ago.
I have a requirement where I need to replace line feed characters appearing as part of a data field in a CSV files. Fortunately all the unnecessary linefeeds are followed by an '_' character. So I decided to used sed to preprocess the file. The following command in sed works if run it interactively.
sed -i -e ':a;N;$!ba;s/\n_/_/g' file
But in server, the command will get executed with sh -c "<command>"
To test it in local, I ran the same command with sh and it is not working. The command looks as follows.
sh -c "sed -i -e ':a;N;$!ba;s/\n_/_/g' file"
Not sure what I'm missing. Please help.
Between double-quotes, $! expands to the PID of most recent background command (or the empty string if there is not one). Pass sed script as a positional parameter to sh to avoid dealing with quotation issues/escaping every special character:
sh -c 'sed -i -e "$1" file' _ ':a;N;$!ba;s/\n_/_/g'
See also: Difference between single and double quotes in Bash
Related
I am trying to add an argument to the flag batch start. This is the error it gives me. Any idea on how to fix this?
$ sed -i "s/batch_start.*/batch_start\ 1111/" /tmp/runfile
sed: -e expression #1, char 27: unterminated `s' command
The root problem is that the command is being sent over ssh; that means it's running through two levels of shell parsing, one on the local computer, then another on the remote computer. That means it goes through quote/escape parsing, application, and removal twice, so you need two "layers" of quoting/escaping.
The command in the last comment doesn't parse (mismatched quotes), but I can reproduce the error message with this command:
ssh remoteHost "sudo sed -i "s/batch_start.*/batch_start\ 1111/" /tmp/runfile"
This sort-of has two levels of quotes, but quotes don't nest, so it doesn't work. The local shell parses this as a double-quoted string "sudo sed -i ", then an unquoted section s/batch_start.*/batch_start\ 1111/ (which contains an escaped space, so it'll remove the escape), then another double-quoted section: " /tmp/runfile". Since there are no spaces between them, they all get passed to ssh as a single argument. You can see the post-parsing string by replacing ssh remoteHost with echo:
$ echo "sudo sed -i "s/batch_start.*/batch_start\ 1111/" /tmp/runfile"
sudo sed -i s/batch_start.*/batch_start 1111/ /tmp/runfile
...so that's the command the remote shell will execute. Since there's a space between s/batch_start.*/batch_start and 1111/, they get passed to sed as separate arguments, and it treats the first as the command to execute (which is missing a close /) and the second as a filename.
Solution: there are many ways to correct the quoting. You can use the echo trick to see what'll get sent to the remote shell. I tend to prefer single-quotes around the entire command, and then just quote normally inside that (as long as the inner command doesn't itself contain single-quotes). In this case, that means:
ssh remoteHost 'sudo sed -i "s/batch_start.*/batch_start 1111/" /tmp/runfile'
which executes this on the remote computer:
sudo sed -i "s/batch_start.*/batch_start 1111/" /tmp/runfile
(note that I removed the escape on the space.)
This question already has answers here:
Difference between single and double quotes in Bash
(7 answers)
Closed 2 years ago.
I'm trying to execute a list of commands through the command:
bash -l -c "commands"
However, when I define a variable an then try to use them, the variable is undefined (empty). To be clear:
bash -l -c "var=MyVariable; echo $var"
Bash expansion (as explained here) will expand the variable value inside the double quotes before actually executing the commands. To avoid so you can follow either of these options:
Option 1:
Avoid the expansion using single quotes
bash -l -c 'var=MyVariable; echo $var'
Option 2:
Avoid the expansion inside the double quotes by escaping the desired variables
bash -l -c "var=MyVariable; echo \$var"
The second option allows you to expand some variables and some others not. For example:
expandVar=MyVariable1
bash -l -c "var=MyVariable; echo $expandVar; echo \$var"
Bash expands variables inside double quotes. So in effect in your command $var is replaced by the current value of var before the command is executed. What you want can be accomplished by using single quotes:
bash -l -c 'var=MyVariable; echo $var'
Please note that it is rather unusual to invoke Bash as a login shell (-l) when passing a command string with -c, but then you may have your reasons.
I have just recently got back into learning bash. Currently working on a project of mine and when using sed I've run into an issue, I've tried looking around the web for help but haven't had any joy. I suspect as I may not be using the correct terminology so I can't find what I'm looking for. ANYHOW.
So in my script I'm trying to assign the output of date to a variable. Here's the line from my script.
origdate=$(date)
When I call it the output looks like this:
Wed Oct 5 19:40:45 BST 2016
Part of my script then generates a file and writes information to it, part of which I am trying to use sed to find lines and replace parts of it. This is the first I've been playing around with sed, I've used it successfully so far for my needs. However I'm getting stuck when I try this:
sed -i '/origdate=empty/c\'$origdate'' $sd/pingcheck-email-$job.txt
When I run the script and it gets to this line, this is the error I'm getting:
sed: can't read Oct: No such file or directory
sed: can't read 5: No such file or directory
sed: can't read 19:52:56: No such file or directory
sed: can't read BST: No such file or directory
sed: can't read 2016: No such file or directory
I suspect it's something to do with the spaces in the date (variable), my question is: how can I work around this? Can I get sed to 'ignore' the spaces? or should I just use cut to cut the field for the date, and set that to a variable and the same thing again to set the time to another variable?
Even if someone could kindly point me in the right direction that'd be great!
Thanks in advance!
double quote the variable
sed -i '/origdate=empty/c\'"$origdate"'' $sd/pingcheck-email-$job.txt
or alternatively, the whole script
sed -i "/origdate=empty/c\$origdate" $sd/pingcheck-email-$job.txt
The problem is not with sed but rather with how bash word splits on your date given your command.
Bash
In bash, word splitting is performed on the command line so that text is broken up into a list of arguments. To illustrate, I'm going to run a simple script that outputs the first argument only.
bash -c 'echo $1' ignored_0 foo bar
Think of bash -c 'echo $1' ignored_0 as the command (sed in your case) and foo bar as the arguments. In this case, foo bar is split into two arguments, foo and bar.
To pass foo bar in as the first parameter, you need to have the text in either single or double quotes. See the GNU manual on quoting.
bash -c 'echo $1' ignored_0 'foo bar'
bash -c 'echo $1' ignored_0 "foo bar"
Parameter expansion does not occur when the variable is inside a single quote.
var="foo bar"
bash -c 'echo $1' ignored_0 '$var'
bash -c 'echo $1' ignored_0 "$var"
NOTE: In the command `bash -c 'echo $1', I do not want $1 to expand before being passed as an argument to bash because that's part of the code I want to execute.
Parameter expansion occurs when variables are outside of quotes, but word splitting will apply after the parameter is expanded. From the bash man page in the Word Splitting section:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
From the GNU bash manual on Word Splitting:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
var="foo bar"
bash -c 'echo $1' ignored_0 $var
The last step in Shell Expansions in Quote Removal where unquoted quote characters are removed before being passed to commands. The following command shows that ''"" has no effect on the arguments passed.
bash -c 'echo $1' ignored_0 foo''""
Application
In your example, the trailing '' after $origdate is extraneous. The important part is that $origdate is not quoted so word splitting applies to the expanded variable.
When -e is not passed to the sed command, sed expects the expression to be in one argument, or word from bash. When you run your command, your expression is /origdate=empty/c\Wed and the rest of the date is considered to be files for the expression to be applied to.
The simple fix is to put double quotes around the string for which you want to prevent word splitting. I've modified the command so that anyone can run this example without having the files on their system.
In this example, the \ must be escaped so that it is not considered an escape character for $.
echo "origdate=empty" | sed "/origdate=empty/c\\$origdate"
You can also change the type of quotes you are using without affecting word splitting like so.
echo "origdate=empty" | sed '/origdate=empty/c\'"$origdate"
You need escape by double slash
\ / \%
This question already has answers here:
Expansion of variables inside single quotes in a command in Bash
(8 answers)
Closed 22 days ago.
I want to use sed with a variable. My script:
input_variable="test" &&
ssh root#192.168.7.2 'cd /path/to/file && sed -i "s/this is not/this is a $input_variable/g" text.txt'
My script is supposed to change this is not into this is a test
However it changes it to this is a and somehow ingnoring the variable input_variable
Anyone an idea?
You're not interpolating that variable because you are using single quotes. I recommend continuing to use single quotes except for around the variable where you should switch to double quotes:
input_variable="test" &&
ssh root#192.168.7.2 'cd /path/to/file && sed -i "s/this is not/this is a '"$input_variable"'/g" text.txt'
Environment variables will not be replace in single quoted strings. Did you try:
input_variable="test" &&
ssh root#192.168.7.2 "cd /path/to/file && sed -i \"s/this is not/this is a ${input_variable}/g\" text.txt"
This question already has answers here:
Escape a string for a sed replace pattern
(17 answers)
Closed 9 years ago.
I have a sed command in a bash script like this:
sed -i 's/db.url=.*/db.url='$URL'/' config.properties
and URL var is assigned as:
$URL=jdbc\:oracle\:thin\:#\/\/hostname\:12345\/XYZ
When I run this bash script on the host it exists on it work as intended, replacing the url with the one specified in URL. However, when I add a command in the bash script to do this on my other host like this:
ssh user#host02 <<EOF
sed -i 's/db.url=.*/db.url='$URL'/' config.properties
exit
EOF
I get this error:
sed: -e expression #1, char 47: unknown option to `s'
Anyone know what may be going on here?
You've properly quote the sed expression if it were running on the local host, but the string is then passed to the shell on the remote host, where the * is now unquoted and expanded as a glob. The simplest thing to do is to pipe the command to the remote shell via standard input, so you don't have to worry about quoting:
echo "sed -i 's/db.url=.*/db.url=$URL/' config.properties" | ssh user#host02 bash
With multiple commands, you may consider using a here document:
ssh user#host02 bash <<EOF
command1
sed -i 's/db.url=.*/db.url=$URL/' config.properities
command2
EOF
The solution was to use double single quotes (') around $URL like this: ''$URL''