I have a script like this:
#!/bin/bash
xx="-e \"s|a|b|g\""
sed -i $xx file
But sed breaks with message:
sed: -e expression #1, char 1: unknown command: `"'
Using set -x I can see that the command is being expanded to sed -i -e '"s|a|b|g"' file, so I guess the double quotes are why it is not working.
How to fix this?
I'm not sure exactly why you want to do what you're doing but I think that this might help:
$ cat file
a a
$ xx=( -e 's|a|b|g' -e 's|b|c|g' )
$ sed "${xx[#]}" file
c c
Use an array to store each argument to sed. Use "${xx[#]}" to safely expand the array, passing each element as a single argument.
You can build up the array like this:
$ xx=()
$ xx+=( -e 's|a| b c |g' )
$ xx+=( -e 's|c| d |g' )
$ sed "${xx[#]}" file
b d b d
You could try expanding the strings using eval, but it is not often recommended by bash aficionados.
#!/bin/bash
xx="-e 's|b|a|g'"
eval sed -i "$xx" file
You can see it getting expanded when using eval, below is the snippet from set +x
++ xx='-e '\''s|a|b|g'\'''
++ eval sed -i '-e '\''s|a|b|g'\''' file
+++ sed -i -e 's|a|b|g' file
To see it in action:-
$ cat file
line1/script
alaalaala++
line1/script
line2/script
line3/script
alaalaala--
line1/script
$ ./script.sh ; cat file
line1/script
blbblbblb++
line1/script
line2/script
line3/script
blbblbblb--
line1/script
Related
For the following MVCE:
echo "test_num: 0" > test.txt
test_num=$(grep 'test_num:' test.txt | cut -d ':' -f 2)
new_test_num=$((test_num + 1))
echo $test_num
echo $new_test_num
sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt
cat test.txt
echo "sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt"
sed -i "s/test_num: 0/test_num: 1/g" test.txt
cat test.txt
Which outputs
0 # parsed original number correctly
1 # increment the number
test_num: 0 # sed with interpolated variable, does not work
sed -i s/test_num: 0/test_num: 1/g test.txt # interpolated parameter looks right
test_num: 1 # ???
Why does sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt not produce the expected result when sed -i "s/test_num: 0/test_num: 1/g" test.txt works just fine in the above example?
As mentioned in the comment, there is a white space in ${test_num}. Therefore in your sed there should not be an empty space between the colon and your variable.
Also I guess you should surround your variable with curly bracket {} to increase readability.
sed "s/test_num:${test_num}/test_num: ${new_test_num}/g" test.txt
test_num: 1
If you just want the number in ${test_num}, you can try something like:
grep 'test_num:' test.txt | awk -F ': ' '{print $2}'
awk allows to specify delimiter with more than 1 character.
Instead of grep|cut you can also use sed.
#! /bin/bash
exec <<EOF
test_num: 0
EOF
grep 'test_num:' | cut -d ':' -f 2
exec <<EOF
test_num: 0
EOF
sed -n 's/^test_num: //p'
When using regexp replace in sed there is special meaning to $ .
Suggesting to rebuild your sed command segments as follow:
sed -i 's/test_num: '$test_num'/test_num: '$new_test_num'/g' test.txt
Other option, use echo command to expand variables in sed command.
sed_command=$(echo "s/test_num:${test_num}/test_num: ${new_test_num}/g")
sed -i "$sed_command" test.txt
I am using the sed command on Ubuntu to replace content.
This initial command comes from here.
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
However, as you can see, I have a slash in the replacement. The slash causes the command to throw:
sed: -e expression #1, char 9: unknown option to `s'
Moreover, my replacement is stored in a variable.
So the following will not work because of the slash:
sed -i "$ s/$/ $1/" "$DIR./result/doc.md"
As stated here and in duplicate, I should use another delimiter. If I try with #:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
It gives the error:
sed: -e expression #1, char 42: unterminated `s' command
My question is:
How can I use a variable in this command as well as other delimiter than / ?
Don't use sed here; perl and awk allow more robust approaches.
sed doesn't allow variables to be passed out-of-band from code, so they always need to be escaped. Use a language without that limitation, and you have code that always works, no matter what characters your data contains.
The Short Answer: Using perl
The below is taken from BashFAQ #21:
inplace_replace() {
local search=$1; shift; local replace=$1; shift
in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' "$#"
}
inplace_replace '#' "replacement" "$DIR/result/doc.md"
The Longer Answer: Using awk
...or, using awk to do a streaming replacement, and a shell function to make that file replacement instead:
# usage as in: echo "in should instead be out" | gsub_literal "in" "out"
gsub_literal() {
local search=$1 replace=$2
awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'
}
# usage as in: inplace_replace "in" "out" /path/to/file1 /path/to/file2 ...
inplace_replace() {
local search=$1 replace=$2 retval=0; shift; shift
for file; do
tempfile=$(mktemp "$file.XXXXXX") || { retval |= $?; continue; }
if gsub_literal "$search" "$replace" <"$file" >"$tempfile"; then
mv -- "$tempfile" "$file" || (( retval |= $? ))
else
rm -f -- "$tempfile" || (( retval |= $? ))
fi
done
}
TL;DR:
Try:
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Long version:
Let's start with your original code:
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
And let's compare it to the code you referenced:
sed -i '$ s/$/abc/' file.txt
We can see that they don't exactly match up. I see that you've correctly made this substitution:
file.txt --> "$DIR./result/doc.md"
That looks fine (although I do have my doubts about the . after $DIR ). However, the other substitution doesn't look great:
abc --> /replacement
You actually introduced another delimeter /. However, if we replace the delimiters with '#' we get this:
sed -i '$ s#$# /replacement#' "$DIR./result/doc.md"
I think that the above is perfectly valid in sed/bash. The $# will not be replaced by the shell because it is single quoted. The $DIR variable will be interpolated by the shell because it is double quoted.
Looking at one of your attempts:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
You will have problems due to the shell interpolation of $# in the double quotes. Let's correct that by replacing with single quotes (but leaving $1 unquoted):
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Notice the '"$1"'. I had to surround $1 with '' to basically unescape the surrounding single quotes. But then I surrounded the $1 with double quotes so we could protect the string from white spaces.
Use shell parameter expansion to add escapes to the slashes in the variable:
$ cat file
foo
bar
baz
$ set -- ' /repl'
$ sed "s/$/$1/" file
sed: 1: "s/$/ /repl/": bad flag in substitute command: 'r'
$ sed "s/$/${1//\//\\\/}/" file
foo /repl
bar /repl
baz /repl
That is a monstrosity of leaning toothpicks, but it serves to transform this:
sed "s/$/ /repl/"
into
sed "s/$/ \/repl/"
The same technique can be used for whatever you choose as the sed s/// delimiter.
I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}
sed -i "s|{FILES}|$FILES|" ./myFile.json
What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :
FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."
I can't think of a solution where xargs would help me.
The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.
For example, if myFile.json would contain:
{foo: 1, bar: "{FILES}"}
then the script should be:
#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF
You can generate the script with a single sed command:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json
Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.
Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash
Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin
Output:
{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}
Maybe perl would have fewer limitations?
perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json
It's a little gross, but you can do it all within shell...
while read l
do
if ! echo "$l" | grep -q '{DATA}'
then
echo "$l"
else
echo "$l" | sed 's/{DATA}.*$//'
echo "$FILES"
echo "$l" | sed 's/^.*{DATA}//'
fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json
Obviously I'd leave the final line commented until you were confident it worked...
Maybe just don't do it? Can you just :
echo "var f = " > myFile2.json
echo $FILES >> myFile2.json
And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)
Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:
foo.pl:
open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
s/world/$foo/;
print;
}
Command to run:
aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json
Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.
I have problem with this sed oneliner
sed -i -n "1h; 1!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}"
Obviously I need to expand variable key, val and deval in sed. So I need the " around sed command.
With this command I get
bash: !H: event not found
escaping the ! corrects it
sed -i -n "1h; 1\!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val(\"\]\]>\)/\1$deval\2/;ta p}"
With this I get
bash: sed -i -n "1h; 1\!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}" :bad substitution
So I guess the { is a problem. Trying to fix it like this
sed -i -n "1h; 1\!H; $\{g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val(\"\]\]>\)/\1$deval\2/;ta p}"
yields
sed: -e expression 1, char 6: unknown command: "\"
What is going on here? How can I make this work?
event not found is only a problem in interactive shells because histexpand is enabled by default. If you either run set +H first or put it in a script and run it from there, Bash will leave your !s alone.
${..} is variable substitution syntax (so a mangled value gives bad substitution). Let sed treat it as a block of commands to do on the final line by escaping the $, as in \${ .. }.
In full:
set +H
key="foo"
val="bar"
deval="puppies"
echo 'Name="foo" <![CDATA["bar"]]>' > file
sed -i -n "1h; 1!H; \${g; :a s/\(Name=\"$key\".*<!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}" file
cat file
Will print Name="foo" <![CDATA["puppies"]]>
You can use separate single-quoted strings:
sed -i -n '1h; 1!H; ${g; :a s/\(Name='"$key"'.*<\!\[CDATA\[\"\)'"$val"'\(\"\]\]>\)/\1'"$deval"'\2/;ta p}'
I want to use sed to replace all ' with \' and all " with \". Example input:
"a" 'b'
Output:
\"a\" \'b\'
There's no ? character in your post, but I'll assume your question is "How do I do such a replacement?". I just made a quick test file with your input, and this command line seems to work:
sed -e 's#"#\\"#g' -e "s#'#\\\'#g"
Example:
$ cat input
"a" 'b'
$ sed -e 's#"#\\"#g' -e "s#'#\\\'#g" input
\"a\" \'b\'
While using sed is the portable solution, all this can be done using Bash's builtin string manipulation functions as well.
(
#set -xv
#str=$'"a" \'b\''
str='"a" '"'b'" # concatenate 'str1'"str2"
str="${str//\"/\\\"}"
str="${str//\'/\'}"
echo "$str"
)