Searching pattern and displaying with SED which have double code (") and variable need to pass.
Actual log is:
<confirmation ID="123456-109" status>
Want to Print 109
This is with actual sed command which worked :
sed -n 's%.*confirmation="123456-\(.*\)" status.*%\%p' /tmp/log
output: 109
But when I tried use this in script where passing 123456 as variable it is not working.
req_data=`sed -n 's%.*confirmation="$variable-\(.*\)" status.*%\%p' /tmp/log`
When I run in Script is is not giving me any output.
I am expecting output: 109
Execute a command and assign its output to variable res as res="$(command)".
You need to used sed -E to make parenthesis work for grouping (or in normal mode, you have to escape each parenthesis). (note that -E option of sed is not part of the posix standard)
Also, as stated in the comments, variables are not expanded/interpolated inside single quotes. If you are in a single quoted block, terminate that before the variable and start a different block after the variable.
Following should addresses all the above and should solve your problem.
req_data="$(sed -En 's%.*confirmation ID=\\"'$variable'-(.+)\\" status.*%\1%p' /tmp/log)"
Related
This is a simple question but i am unable to find it in tutorials. Could anybody please explain what this statement does when executed in a bash shell within a folder containing .sh scripts. I know -i does in place editing, i understand that it will run sed on all scripts within the current directory. And i know that it does some sort of substitution. But what does this \(.*\) mean?
sed -i 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
Thanks in advance.
You have an expression like:
sed -i 's/XXX=\(YYY\)/XXX=ZZZ/' file
This looks for a string XXX= in a file and captures what goes after. Then, it replaces this captured content with ZZZ. Since there is a captured group, it is accessed with \1. Finally, using the -i flag in sed makes the edition to be in-place.
For the replacement, it uses the following syntax described in Shell parameter expansion:
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.
Example:
$ d=5
$ echo ${d-3}
5
$ echo ${a-3}
3
So with ${MY_BASE_DIR-SOMETHING-\1} you are saying: print $MY_BAS_DIR. And if this variable is unset or null, print what is stored in \1.
All together, this is resetting MY_BASE_DIR to the value in the variable $MY_BASE_DIR unless this is not set; in such case, the value remains the same.
Note though that the variable won't be expanded unless you use double quotes.
Test:
$ d=5
$ cat a
d=23
blabla
$ sed "s/d=\(.*\)/d=${d-\1}/" a # double quotes -> value is replaced
d=5
blabla
$ sed 's/d=\(.*\)/d=${d-\1}/' a # single quotes -> variable is not expanded
d=${d-23}
blabla
Andd see how the value remains the same if $d is not set:
$ unset d
$ sed "s/d=\(.*\)/d=${d-\1}/" a
d=23
The scripts contain lines like this:
MY_BASE_DIR=/usr/local
The sed expression changes them to:
MY_BASE_DIR=${MY_BASE_DIR-/usr/local}
The effect is that /usr/local is not used as a fixed value, but only as the default value. You can override it by setting the environment variable MY_BASE_DIR.
For future reference, I would take a look at the ExplainShell website:
http://explainshell
that will give you a breakdown of the command structure etc. In this instance, let step through the details...Let's start with a simple example, let's assume that we were going to make the simple change - commenting out all lines by adding a "#" before each line. We can do this for all *.sh files in a directory with the ".sh" extension in the current directory:
sed 's/^/\#/' *.sh
i.e. Substitute beginning of line ^, with a # ...
Caveat: You did not specify the OS you are using. You may get different results with different versions of sed and OS...
ok, now we can drill into the substitution in the script. An example is probably easier to explain:
File: t.sh
MY_BASE_DIR="/important data/data/bin"
the command 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
will search for "MY_BASE_DIR" in each .sh file in the directory.
When it encounters the string "MY_BASE_DIR=.*", in the file, it expands it to be MY_BASE_DIR="/important data/data/bin", this is now replaced on the right side of the expression /MY_BASE_DIR=${MY_BASE_DIR-\1}/ which becomes
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
essentially what happens is that the substitute operation takes
MY_BASE_DIR="/important data/data/bin"
and inserts
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
now if we run the script with the variable MY_BASE_DIR set
export MY_BASE_DIR="/new/import/dir"
the scripts modified by the sed script referenced will now substitute /important data/data/bin with /new/import/dir...
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
I'm implementing a template renderer in shell script. The template variables are represented in a template by #VAR_NAME# and their values are defined in a separate shell script.
Sample code:
# template variables values
CONTACT_EMAIL="myemail"
CONTACT_NAME="myname"
VARS="CONTACT_EMAIL CONTACT_NAME"
TEMPLATE_FILEPATH="mytemplate.txt"
# template renderer
set -x
SEDARGS=
for VAR in $VARS; do
SEDARGS+=" -e \"s,#$VAR#,${!VAR},\""
done
sed -r $SEDARGS $TEMPLATE_FILEPATH
sed command executed by shell and printed by it because of "set -x":
+ sed -r -e '"s,#CONTACT_EMAIL#,myemail,"' -e '"s,#CONTACT_NAME#,myname,"' mytemplate.txt
sed output:
sed: -e expression #1, char 1: unknown command: `"'
I know the single quotes around each sed expression are causing this non-intuitive error message, but I do not know why they are added.
What is wrong?
You have embedded quotes inside your SEDARGS variable. These are NOT removed when the command is executed. To remove them, you need to call the interpreter again, which you can do using eval. For example:
eval sed -r $SEDARGS $TEMPLATE_FILEPATH
You may need to play around that some more (adding quotes, etc.).
The single quotes aren't part of the actual arguments. They get added by your shell for the output caused by set -x only.
Why would the shell do that? So that you can use that output to re-run exactly what was executed during the script execution. As you correctly noticed, they are needed to protect the " that came from SEDARGS content (i.e., the inner ones, in your script escaped as \").
So I have seen this question: Replace complete line getting number from variable
which is pretty similar, but in my case I am trying to use multiple variables: one for the line number [lineNo], one for the text to replace [transFormatted] and the file to which it should be looking in [OUTPUT_FILE] I've tried dozens of combinations to try to get it to recognize all these variables but nothing seems to work. It's unhappy no matter which way I try. What am I doing wrong?
sed -e '${lineNo}s/.*/${transFormatted}/' < $OUTPUT_FILE
Single quotes inhibit parameter expansion.
sed -e "${lineNo}s/.*/$transFormatted/" < "$OUTPUT_FILE"
You have to use double quotes for the variables to be expanded from the shell environment.
I'd like to use sed to do a replace, but not by searching for what to replace.
Allow me to explain. I have a variable set to a default value initially.
VARIABLE="DEFAULT"
I can do a sed to replace DEFAULT with what I want, but then I would have to put DEFAULT back when I was all done. This is becuase what gets stored to VARIABLE is unique to the user. I'd like to use sed to search for somthing else other than what to replace. For example, search for VARIABLE=" and " and replace whats between it. That way it just constantly updates and there is no need to reset VARIABLE.
This is how I do it currently:
I call the script and pass an argument
./script 123456789
Inside the script, this is what happens:
sed -i "s%DEFAULT%$1%" file_to_modify
This replaces
VARIABLE="DEFAULT"
with
VARIABLE="123456789"
It would be nice if I didn't have to search for "DEFAULT", because then I would not have to reset VARIABLE at end of script.
sed -r 's/VARIABLE="[^"]*"/VARIABLE="123456789"/' file_to_modify
Or, more generally:
sed -r 's/VARIABLE="[^"]*"/VARIABLE="'"$1"'"/' file_to_modify
Both of the above use a regular expression that looks for 'VARIABLE="anything-at-all"' and replaces it with, in the first example above 'VARIABLE="123456789"' or, in the second, 'VARIABLE="$1"' where "$1" is the first argument to your script. The key element is [^"]. It means any character other than double-quote. [^"]* means any number of characters other than double-quote. Thus, we replace whatever was in the double-quotes before, "[^"]*", with our new value "123456789" or, in the second case, "$1".
The second case is a bit tricky. We want to substitute $1 into the expression but the expression is itself in single quotes. Inside single-quotes, bash will not substitute for $1. So, the sed command is broken up into three parts:
# spaces added for exposition but don't try to use it this way
's/VARIABLE="[^"]*"/VARIABLE="' "$1" '"/'
The first part is in single quotes and bash passes it literally to sed. The second part is in double-quotes, so bash will subsitute in for the value of `$``. The third part is in single-quotes and gets passed to sed literally.
MORE: Here is a simple way to test this approach on the command line without depending on any files:
$ new=1234 ; echo 'VARIABLE="DEFAULT"' | sed -r 's/VARIABLE="[^"]*"/VARIABLE="'"$new"'"/'
VARIABLE="1234"
The first line above is the command run at the prompt ($). The second is the output from running the command..