I have file called "text_file1.txt" and the content in the file is
"subject= /C=US/O=AAA/OU=QA/OU=12345/OU=TESTAPP/"
Now what i want to achieve is to the content to be like below:
"subject= /C=US/O=AAA/$$$QA/###12345/###TESTAPP/"
when i execute the below piece of code:
#! /bin/ksh
OU1="QA"
OU2=12345
OU3="TESTAPP"
`sed -i "s/OU=$OU1/$$$\${OU1}/g" text_file1.txt`
`sed -i "s/OU=$OU2/###\${OU2}/g" text_file1.txt`
`sed -i "s/OU=$OU3/###\${OU3}/g" text_file1.txt`
content=`cat text_file1.txt`
echo "content:$content"
i get the output like this:
content:subject= /C=US/O=Wells Fargo/2865528655{OU1}/###12345/###TESTAPP/CN=03032015_CUST_2131_Unix_CLBLABB34C02.wellsfargo.com
only this command "sed -i "s/OU=$OU1/$$$\${OU1}/g" text_file1.txt" is not working as expected.Can anyone please suggest some idea on this?
Thanks in advance.
Two things play into this:
You have to escape $ (i.e., use \$) in doubly-quoted shell strings if you want a literal $, and
\ does not retain its literal meaning when it comes before a $ inside backticks (that is to say, inside backticks, \$ becomes just $).
When you write
`sed -i "s/OU=$OU1/$$$\${OU1}/g" text_file1.txt`
because the command is in backticks, you spawn a subshell with the command
sed -i "s/OU=$OU1/$$$${OU1}/g" text_file1.txt
Since $$$$ is inside a doubly-quoted string, variable expansion takes place, and it is expanded as two occurrences of $$ (the process ID of the shell that's doing the expansion). This means that the code sed sees is ultimately
s/OU=QA/1234512345{OU1}/g
...if the process ID of the spawned subshell is 12345.
In this particular case, you don't need the command substitution (the backticks), so you could write
sed -i "s/OU=$OU1/\$\$\$${OU1}/g" text_file1.txt
However, using shell variables in sed code is always a problem. Consider, if you will, what would happen if OU1 had the value /; e rm -Rf * # (hint: GNU sed has an e instruction that runs shell commands). For this reason, I would always prefer awk to do substitutions that involve shell variables:
cp text_file1.txt text_file1.txt~
awk -v OU1="$OU1" '{ gsub("OU=" OU1, "$$$" OU1) } 1' text_file1.txt~ > text_file1.txt
This avoids code injection problems by not treating OU1 as code.
If you have GNU awk 4.1 or later,
awk -v OU1="$OU1" -i inplace '{ gsub("OU=" OU1, "$$$" OU1) } 1' text_file1.txt
can do the whole thing without a (visible) temporary file.
Does this help as a start?
echo ''
OU1="QA"
echo "subject= /C=US/O=AAA/OU=${OU1}/OU=12345/OU=TESTAPP/" \
| sed -e "s|/OU=${OU1}/|/OU=\$\$\$${OU1}/|g"
The result is:
subject= /C=US/O=AAA/OU=$$$QA/OU=12345/OU=TESTAPP/
(You are mixing up the use of $ signs .)
You must be careful when putting $ inside double quotes.
sed -i "s/OU=$OU1/"'$$$'"${OU1}/g" text_file1.txt
Example:
$ OU1="QA"
$ echo 'OU=QA' | sed "s/OU=$OU1/"'$$$'"${OU1}/g"
$$$QA
Related
I want to replace the version in my code using git rev-parse HEAD with template string %VERSION% in a source file.
For simplicity I will use date as version command in this question.
Given test.txt
$ echo "This is test-%VERSION%." > test.txt
$ cat test.txt
This is test-%VERSION%.
Expect
This is test-Sat Dec 2 16:48:59 +07 2017.
These are failed try
$ echo "This is test-%VERSION%." > test.txt
$ sed -i 's/%VERSION/`date`/' test.txt && cat test.txt
This is test-`date`%.
$ echo "This is test-%VERSION%." > test.txt
$ DD=`date` sed -i 's/%VERSION/$DD/' test.txt && cat test.txt
This is test-$DD%.
$ echo "This is test-%VERSION%." > test.txt
$ DD=`date` sed -i "s/%VERSION/$DD/" test.txt && cat test.txt
This is test-%.
Do I really need to use xargs ?
You can embed $(...) within double-quotes, but not in single-quotes:
sed -i "s/%VERSION%/$(date)/" test.txt && cat test.txt
(Same as `...` but you shouldn't use that obsolete syntax, $(...) is better.)
Btw, for testing purposes, it's better to use sed without -i,
so the original file is not modified:
sed "s/%VERSION%/$(date)/" test.txt
As a side note, and this is a completely different discussion,
but worth mentioning here.
This may look like it should work but it doesn't, and you may wonder why:
DD=$(date) sed -i "s/%VERSION%/$DD/" test.txt && cat test.txt
Why it doesn't work?
Because the $DD embedded in the "..." is evaluated at the time the command is executed.
At that time the value of DD is not set to the output of $(date).
In the "..." it will have whatever value it had before executing the command.
For the sed process, the value DD with the output of $(date) is visible,
but sed doesn't use that, because why would it.
The "..." passed to sed is evaluated by the shell, not by sed.
Use double-quotes to do the substitution and avoid using the outdated `` construct but rather use the $(..) syntax for Command substitution
sed -i "s/%VERSION%/$(date)/" file
Also another way if you just want to use the single quotes, would be to wrap the substitution part in double-quotes and then single-quote on top of it, something like sed 's/%VERSION%/'"$(date)"'/' file which is less efficient than simply double-quoting the entire substitution string.
I have some variables:
$begin=10
$end=20
how to pass them to sed command.
sed -n '$begin,$endp' filename | grep word
sed -n '10,20p' filename | grep word
The reason this doesn't work is that single quotes in shell code prevent variable expansion. The good way is to use awk:
awk -v begin="$begin" -v end="$end" 'NR == begin, NR == end' filename
It is possible with sed if you use double quotes (in which shell variables are expanded):
sed -n "$begin,$end p" filename
However, this is subject to code injection vulnerabilities because sed cannot distinguish between code and data this way (unlike the awk code above). If a user manages to set, say, end="20 e rm -Rf /;", unpleasant things can happen.
I have a bash script which checks for a string pattern in file and delete entire line i same file but somehow its not deleting the line and no throwing any error .same command from command prompt deletes from file .
#array has patterns
for k in "${patternarr[#]}
do
sed -i '/$k/d' file.txt
done
sed version is >4
when this loop completes i want all lines matching string pattern in array to be deleted from file.txt
when i run sed -i '/pataern/d file.txt from command prompt then it works fine but not inside bash
Thanks in advance
Here:
sed -i '/$k/d' file.txt
The sed script is singly-quoted, which prevents shell variable expansion. It will (probably) work with
sed -i "/$k/d" file.txt
I say "probably" because what it will do depends on the contents of $k, which is just substituted into the sed code and interpreted as such. If $k contains slashes, it will break. If it comes from an untrustworthy source, you open yourself up to code injection (particularly with GNU sed, which can be made to execute shell commands).
Consider k=^/ s/^/rm -Rf \//e; #.
It is generally a bad idea to substitute shell variables into sed code (or any other code). A better way would be with GNU awk:
awk -i inplace -v pattern="$k" '!($0 ~ pattern)' file.txt
Or to just use grep -v and a temporary file.
first of all, you got an unclosed double quote around ${patternarr[#]} in your for statement.
Then your problem is that you use single quotes in the sed argument, making your shell not evaluate the $k within the quotes:
% declare -a patternarr=(foo bar fu foobar)
% for k in ${patternarr[#]}; do echo sed -i '/$k/d' file.txt; done
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
if you replace them with double quotes, here it goes:
% for k in ${patternarr[#]}; do echo sed -i "/$k/d" file.txt; done
sed -i /foo/d file.txt
sed -i /bar/d file.txt
sed -i /fu/d file.txt
sed -i /foobar/d file.txt
Any time you write a loop in shell just to manipulate text you have the wrong approach. This is probably closer to what you really should be doing (no surrounding loop required):
awk -v ks="${patternarr[#]}" 'BEGIN{gsub(/ /,")|(",ks); ks="("ks")} $0 !~ ks' file.txt
but there may be even better approaches still (e.g. only checking 1 field instead of the whole line, or using word boundaries, or string comparison or....) if you show us some sample input and expected output.
You need to use double quotes to interpolate shell variables inside the sed command, like:
for k in ${patternarr[#]}; do
sed -i "/$k/d" file.txt
done
Basically I want to copy several lines of code from a template file to a script file.
Is it even possible to use sed to copy a string full of symbols that interact with the script?
I used these lines:
$SWAP='sudo cat /home/kaarel/template'
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
The output is:
./line-adder.sh: line 11: =sudo cat /home/kaarel/template: No such file or directory
No, it is not possible to do this robustly with sed. Just use awk:
awk -v swap="$SWAP" '{sub(/#pointer/,swap)}1' script.sh > tmp && mv tmp script.sh
With recent versions of GNU awk there's a -i inplace flag for inplace editing if that's something you care about.
Good point about "&&". Here's the REALLY robust version that will work for absolutely any character in the search or replacement strings:
awk -v old="#pointer" -v new="$SWAP" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
e.g.:
$ echo "abc" | awk -v old="b" -v new="m&&n" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
am&&nc
There are two issues with the line:
$SWAP='sudo cat /home/kaarel/template'
The first is that, before executing the line, bash performs variable expansion and replaces $SWAP with the current value of SWAP. That is not what you wanted. You wanted bash to assign a value to SWAP.
The second issue is that the right-hand side is enclosed in single-quotes which protect the string from expansion. You didn't want to protect the string from expansion: you wanted to execute it. To execute it, you can use back-quotes which may look similar but act very differently.
Back-quotes, however, are an ancient form of asking for command execution. The more modern form is $(...) which eliminates some problems that back-quotes had.
Putting it all together, use:
SWAP=$(sudo cat /home/kaarel/template)
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
Be aware, though, that the sed command may have problems if there are any sed-active characters in the template file.
I'm using sed to replace a character in a file with a variable. This variable is basically reading the contents of a file or a webpage which contains multiple hash-like strings like below, which are randomly generated:
define('AUTH_KEY', 'CVo|BO;Qt1B|+GE}+h2/yU7h=5`/wRV{>%h.b_;s%S8-p|>qpf]|/Vf#`&[g~*:&');
define('SECURE_AUTH_KEY', '{G2-<^jWRd7]2,?]6hhM^*asg.2C.+k=gf33-m+ZK_{Mt|q*<ELF4|gPjyxtTh!)');
define('LOGGED_IN_KEY', 'jSNo9Z;5d]tzZoh-QQ`{M-&~y??$R({:*m`0={67=+mF?L.e+R{;)+4}qCAAHz=C');
define('NONCE_KEY', '19Vt4=%8j/Z-&~ni0S<]9)J^~sy9dh|h9M_RX2#K0]F9+.v+[BP1d&B&}-FTKIJ,');
define('AUTH_SALT', 'jr7f?T|#Cbo]XVAo}N^ilkvD>dC-rr]5{al64|?_Hz }JG$yEi:_aU )Olp YAD+');
define('SECURE_AUTH_SALT', 'hm#Z%O!X_mr?lM|>>~r-?F%wi R($}|9R[):4^NTsj+gS[qnv}7|+0<9e-$DJjju');
define('LOGGED_IN_SALT', 'tyPHBOCkXZh_4H;G|^.&|^#JPB/f;{}y_Orj!6AH?#wovx+KKtTZ A[HMS9SZJ|N');
define('NONCE_SALT', 'Eb-/t 5D-vPV9I--8F<[^lcherGv.g+|7p6;+xP|5g6P}tup1K.vuHAQ=uWZ#}H^');
The variable is defined like so:
KEY=$(cat keyfile)
I used the following sed syntax:
sed -i "s/stringtoreplace/$KEY/" /path/to/file
I've also tried different variations, using single-quotes, quotes around the variables
sed -i "s/stringtoreplace/"$KEY"/" /path/to/file
sed -i "s/stringtoreplace/"${KEY}"/" /path/to/file
sed -i "s/stringtoreplace/'$KEY'/" /path/to/file
I think I "brute-forced" every way possible to put quotes and I don't know how to escape randomly generated characters like those. I keep getting the following error:
unterminated s command
Any suggestions on how to replace the string with the weird-hash-like variable?
sed is an excellent tool for simple substitutions on a single line but for anything else you should use awk:
awk -v key="$KEY" '{sub(/stringtoreplace/,key)}1' file
That will work no matter what characters "$KEY" contains, except for "&".
One possibility is to use bash instead of sed. That makes the substitution easy, but you'll have to emulate the -i option.
Something like this:
TMPFILE=$(mktemp)
KEY=$(cat keyfile)
while IFS= read -r LINE; do
echo "${LINE//stringtoreplace/$KEY}"
done </path/to/file >$TMPFILE
mv $TMPFILE /path/to/file
Try this:
KEY=$(cat keyfile | sed -e 's/[]\/()$*.^|[]/\\&/g')
sed -i "s/stringtoreplace/$KEY/" /path/to/file
The problem is that the $KEY variable contains slashes, which are the delimiters for your s command.
sed can do more than search and replace:
sed '/stringtoreplace/{
d
r keyfile
}' /path/to/file
I'm assuming "stringtoreplace" occurs by itself on a line.