I have problem with this sed oneliner
sed -i -n "1h; 1!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}"
Obviously I need to expand variable key, val and deval in sed. So I need the " around sed command.
With this command I get
bash: !H: event not found
escaping the ! corrects it
sed -i -n "1h; 1\!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val(\"\]\]>\)/\1$deval\2/;ta p}"
With this I get
bash: sed -i -n "1h; 1\!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}" :bad substitution
So I guess the { is a problem. Trying to fix it like this
sed -i -n "1h; 1\!H; $\{g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val(\"\]\]>\)/\1$deval\2/;ta p}"
yields
sed: -e expression 1, char 6: unknown command: "\"
What is going on here? How can I make this work?
event not found is only a problem in interactive shells because histexpand is enabled by default. If you either run set +H first or put it in a script and run it from there, Bash will leave your !s alone.
${..} is variable substitution syntax (so a mangled value gives bad substitution). Let sed treat it as a block of commands to do on the final line by escaping the $, as in \${ .. }.
In full:
set +H
key="foo"
val="bar"
deval="puppies"
echo 'Name="foo" <![CDATA["bar"]]>' > file
sed -i -n "1h; 1!H; \${g; :a s/\(Name=\"$key\".*<!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}" file
cat file
Will print Name="foo" <![CDATA["puppies"]]>
You can use separate single-quoted strings:
sed -i -n '1h; 1!H; ${g; :a s/\(Name='"$key"'.*<\!\[CDATA\[\"\)'"$val"'\(\"\]\]>\)/\1'"$deval"'\2/;ta p}'
Related
I have a Bash script working fine locally, now I am trying to put it in Jenkinsfile to run as its pipeline:
stage('Update Cloudfront'){
steps {
sh '''
#!/bin/bash
YAML_FILE="path/to/values.yaml"
DATE="$(date '+%d-%m-%Y')"
wget https://www.cloudflare.com/ips-v4 && wget https://www.cloudflare.com/ips-v6
CLOUDFLARE_NEW=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -)
CLOUDFLARE_OLD=$(yq -r .controller.config.proxy-real-ip-cidr $YAML_FILE | sed -E 's/\,37\.16\.11\.30\/32//')
if [[ "$CLOUDFLARE_NEW" == "$CLOUDFLARE_OLD" ]]; then
echo "No need to do anything"
else
echo "Cloudflare IP ranges change detected, updating Nginx value file"
CLOUDFLARE_NEW=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -) yq e '.controller.config.proxy-real-ip-cidr = env(CLOUDFLARE_NEW)' -i $YAML_FILE
echo "Add third party IP range"
yq e '.controller.config.proxy-real-ip-cidr +=",1.2.3.4/32"' -i $YAML_FILE
fi
'''
}
}//end stage('Update Cloudfront')
Unfortunately it won't work:
WorkflowScript: 73: unexpected char: '\' # line 73, column 113.
cidr $YAML_FILE | sed -E \\"s/\,37\.16\.
^
I've tried to escape it with \\"s/\,37\.16\.11\.30\/32//\\" etc. but it doesn't work either. I've tried with double and single quotes with no luck.
You can avoid all the escaping by using a character class and different regex delimiters, like so:
sed -e 's#,37[.]16[.]11[.]30/32##'
In the event you do need to escape something though, simply doubling the backslash should do it:
sed -e 's/,37\\.16\\.11\\.30\\/32//'
Though, given the number of levels involved here, it might need double escaping:
sed -e 's/,37\\\\.16\\\\.11\\\\.30\\\\/32//'
I am using the sed command on Ubuntu to replace content.
This initial command comes from here.
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
However, as you can see, I have a slash in the replacement. The slash causes the command to throw:
sed: -e expression #1, char 9: unknown option to `s'
Moreover, my replacement is stored in a variable.
So the following will not work because of the slash:
sed -i "$ s/$/ $1/" "$DIR./result/doc.md"
As stated here and in duplicate, I should use another delimiter. If I try with #:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
It gives the error:
sed: -e expression #1, char 42: unterminated `s' command
My question is:
How can I use a variable in this command as well as other delimiter than / ?
Don't use sed here; perl and awk allow more robust approaches.
sed doesn't allow variables to be passed out-of-band from code, so they always need to be escaped. Use a language without that limitation, and you have code that always works, no matter what characters your data contains.
The Short Answer: Using perl
The below is taken from BashFAQ #21:
inplace_replace() {
local search=$1; shift; local replace=$1; shift
in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' "$#"
}
inplace_replace '#' "replacement" "$DIR/result/doc.md"
The Longer Answer: Using awk
...or, using awk to do a streaming replacement, and a shell function to make that file replacement instead:
# usage as in: echo "in should instead be out" | gsub_literal "in" "out"
gsub_literal() {
local search=$1 replace=$2
awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'
}
# usage as in: inplace_replace "in" "out" /path/to/file1 /path/to/file2 ...
inplace_replace() {
local search=$1 replace=$2 retval=0; shift; shift
for file; do
tempfile=$(mktemp "$file.XXXXXX") || { retval |= $?; continue; }
if gsub_literal "$search" "$replace" <"$file" >"$tempfile"; then
mv -- "$tempfile" "$file" || (( retval |= $? ))
else
rm -f -- "$tempfile" || (( retval |= $? ))
fi
done
}
TL;DR:
Try:
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Long version:
Let's start with your original code:
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
And let's compare it to the code you referenced:
sed -i '$ s/$/abc/' file.txt
We can see that they don't exactly match up. I see that you've correctly made this substitution:
file.txt --> "$DIR./result/doc.md"
That looks fine (although I do have my doubts about the . after $DIR ). However, the other substitution doesn't look great:
abc --> /replacement
You actually introduced another delimeter /. However, if we replace the delimiters with '#' we get this:
sed -i '$ s#$# /replacement#' "$DIR./result/doc.md"
I think that the above is perfectly valid in sed/bash. The $# will not be replaced by the shell because it is single quoted. The $DIR variable will be interpolated by the shell because it is double quoted.
Looking at one of your attempts:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
You will have problems due to the shell interpolation of $# in the double quotes. Let's correct that by replacing with single quotes (but leaving $1 unquoted):
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Notice the '"$1"'. I had to surround $1 with '' to basically unescape the surrounding single quotes. But then I surrounded the $1 with double quotes so we could protect the string from white spaces.
Use shell parameter expansion to add escapes to the slashes in the variable:
$ cat file
foo
bar
baz
$ set -- ' /repl'
$ sed "s/$/$1/" file
sed: 1: "s/$/ /repl/": bad flag in substitute command: 'r'
$ sed "s/$/${1//\//\\\/}/" file
foo /repl
bar /repl
baz /repl
That is a monstrosity of leaning toothpicks, but it serves to transform this:
sed "s/$/ /repl/"
into
sed "s/$/ \/repl/"
The same technique can be used for whatever you choose as the sed s/// delimiter.
I have a script like this:
#!/bin/bash
xx="-e \"s|a|b|g\""
sed -i $xx file
But sed breaks with message:
sed: -e expression #1, char 1: unknown command: `"'
Using set -x I can see that the command is being expanded to sed -i -e '"s|a|b|g"' file, so I guess the double quotes are why it is not working.
How to fix this?
I'm not sure exactly why you want to do what you're doing but I think that this might help:
$ cat file
a a
$ xx=( -e 's|a|b|g' -e 's|b|c|g' )
$ sed "${xx[#]}" file
c c
Use an array to store each argument to sed. Use "${xx[#]}" to safely expand the array, passing each element as a single argument.
You can build up the array like this:
$ xx=()
$ xx+=( -e 's|a| b c |g' )
$ xx+=( -e 's|c| d |g' )
$ sed "${xx[#]}" file
b d b d
You could try expanding the strings using eval, but it is not often recommended by bash aficionados.
#!/bin/bash
xx="-e 's|b|a|g'"
eval sed -i "$xx" file
You can see it getting expanded when using eval, below is the snippet from set +x
++ xx='-e '\''s|a|b|g'\'''
++ eval sed -i '-e '\''s|a|b|g'\''' file
+++ sed -i -e 's|a|b|g' file
To see it in action:-
$ cat file
line1/script
alaalaala++
line1/script
line2/script
line3/script
alaalaala--
line1/script
$ ./script.sh ; cat file
line1/script
blbblbblb++
line1/script
line2/script
line3/script
blbblbblb--
line1/script
I can print a range of lines from a file using this cmd:
sed -n 267975,1000000p < dump-gq1-sample > dump267975
but how to print to the end? I tried
sed -n 267975,$p < dump-gq1-sample > dump267975
and it gives me:
sed: 1: "267975,": expected context address
You're the victim of Shell Parameter Expansion
sed -n 267975,$p < dump-gq1-sample > dump267975
is received by sed as
sed -n 267975, < dump-gq1-sample > dump267975
because the p variable is undefined.
You should quote your parameter with single quotes '
sed -n '267975,$p' < dump-gq1-sample > dump267975
See https://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html For the full list of existing shell expansions.
the single-quote response does not work in cases where the start/end of range are passed to sed as a variable.
including a space between the end-symbol $ and p in single quotes works on my systems to prevent unintended expansion of $p
sed -n '267975,$ p' ${INPUT_FILE} > ${OUTPUT_FILE} #double quotes work here too
If you need to pass the initial line as a variable, double quotes are required for the expansion of $L1 so the space is generally a good idea:
L1=267975
sed -n "${L1},$ p" ${INPUT_FILE} > dump${L1}
I want to use sed to replace all ' with \' and all " with \". Example input:
"a" 'b'
Output:
\"a\" \'b\'
There's no ? character in your post, but I'll assume your question is "How do I do such a replacement?". I just made a quick test file with your input, and this command line seems to work:
sed -e 's#"#\\"#g' -e "s#'#\\\'#g"
Example:
$ cat input
"a" 'b'
$ sed -e 's#"#\\"#g' -e "s#'#\\\'#g" input
\"a\" \'b\'
While using sed is the portable solution, all this can be done using Bash's builtin string manipulation functions as well.
(
#set -xv
#str=$'"a" \'b\''
str='"a" '"'b'" # concatenate 'str1'"str2"
str="${str//\"/\\\"}"
str="${str//\'/\'}"
echo "$str"
)