This question already has an answer here:
sed delete option change of delimiter
(1 answer)
Closed 8 years ago.
#!/bin/bash
# Let's say now, we are working in my $HOME directory
# Content of testfile (originally)
# 123456
# ABCDEF
# /home/superman
string="ABCDEF"
myfile="$HOME/testfile"
# test-1, this is okay
sed -i "/$string/d" $myfile
echo $string >> $myfile
# test-2, this fails
# ERROR (sed: -e expression #1, char 4: extra characters after command)
sed -i "/$PWD/d" $myfile
echo $PWD >> $myfile
# Not working either
sed -i ":$PWD:d" $myfile
echo $PWD >> $myfile
My question: How to handle the $PWD situation?
To use the alternate delimiters for addresses, you need to use backslash - \
sed "\:$PWD:d" < $myfile
Should work.
Of course for this exact example, grep -v is probably easier.
Related
I am using the sed command on Ubuntu to replace content.
This initial command comes from here.
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
However, as you can see, I have a slash in the replacement. The slash causes the command to throw:
sed: -e expression #1, char 9: unknown option to `s'
Moreover, my replacement is stored in a variable.
So the following will not work because of the slash:
sed -i "$ s/$/ $1/" "$DIR./result/doc.md"
As stated here and in duplicate, I should use another delimiter. If I try with #:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
It gives the error:
sed: -e expression #1, char 42: unterminated `s' command
My question is:
How can I use a variable in this command as well as other delimiter than / ?
Don't use sed here; perl and awk allow more robust approaches.
sed doesn't allow variables to be passed out-of-band from code, so they always need to be escaped. Use a language without that limitation, and you have code that always works, no matter what characters your data contains.
The Short Answer: Using perl
The below is taken from BashFAQ #21:
inplace_replace() {
local search=$1; shift; local replace=$1; shift
in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' "$#"
}
inplace_replace '#' "replacement" "$DIR/result/doc.md"
The Longer Answer: Using awk
...or, using awk to do a streaming replacement, and a shell function to make that file replacement instead:
# usage as in: echo "in should instead be out" | gsub_literal "in" "out"
gsub_literal() {
local search=$1 replace=$2
awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'
}
# usage as in: inplace_replace "in" "out" /path/to/file1 /path/to/file2 ...
inplace_replace() {
local search=$1 replace=$2 retval=0; shift; shift
for file; do
tempfile=$(mktemp "$file.XXXXXX") || { retval |= $?; continue; }
if gsub_literal "$search" "$replace" <"$file" >"$tempfile"; then
mv -- "$tempfile" "$file" || (( retval |= $? ))
else
rm -f -- "$tempfile" || (( retval |= $? ))
fi
done
}
TL;DR:
Try:
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Long version:
Let's start with your original code:
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
And let's compare it to the code you referenced:
sed -i '$ s/$/abc/' file.txt
We can see that they don't exactly match up. I see that you've correctly made this substitution:
file.txt --> "$DIR./result/doc.md"
That looks fine (although I do have my doubts about the . after $DIR ). However, the other substitution doesn't look great:
abc --> /replacement
You actually introduced another delimeter /. However, if we replace the delimiters with '#' we get this:
sed -i '$ s#$# /replacement#' "$DIR./result/doc.md"
I think that the above is perfectly valid in sed/bash. The $# will not be replaced by the shell because it is single quoted. The $DIR variable will be interpolated by the shell because it is double quoted.
Looking at one of your attempts:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
You will have problems due to the shell interpolation of $# in the double quotes. Let's correct that by replacing with single quotes (but leaving $1 unquoted):
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Notice the '"$1"'. I had to surround $1 with '' to basically unescape the surrounding single quotes. But then I surrounded the $1 with double quotes so we could protect the string from white spaces.
Use shell parameter expansion to add escapes to the slashes in the variable:
$ cat file
foo
bar
baz
$ set -- ' /repl'
$ sed "s/$/$1/" file
sed: 1: "s/$/ /repl/": bad flag in substitute command: 'r'
$ sed "s/$/${1//\//\\\/}/" file
foo /repl
bar /repl
baz /repl
That is a monstrosity of leaning toothpicks, but it serves to transform this:
sed "s/$/ /repl/"
into
sed "s/$/ \/repl/"
The same technique can be used for whatever you choose as the sed s/// delimiter.
I have a short bash script to replace a uuid in a line in a file:
#!/bin/sh
alpha="0-9A-F"
uuidPtn="[$alpha]{8}-[$alpha]{4}-[$alpha]{4}-[$alpha]{4}-[$alpha]{12}"
ProductCode="\"ProductCode\" = \"8:{0059DDB5-D384-46F9-BBFD-0004A8C39732}\""
newguid=`uuidgen`
newguid="${newguid^^}"
cmd="echo $ProductCode | sed -r s/$uuidPtn/$newguid/"
echo "$ProductCode"
eval "$cmd"
It produces almost correct output, but with the quotation marks omitted:
"ProductCode" = "8:{0059DDB5-D384-46F9-BBFD-0004A8C39732}"
ProductCode = 8:{A4B1D092-1C56-44F3-B096-34B67A5F39B1}
How can I include the quotation marks?
Glad you got it working! Here's another way, which does not involve eval (since eval is evil):
#!/bin/bash
alpha="0-9A-F"
uuidPtn="[$alpha]{8}-[$alpha]{4}-[$alpha]{4}-[$alpha]{4}-[$alpha]{12}"
ProductCode="\"ProductCode\" = \"8:{0059DDB5-D384-46F9-BBFD-0004A8C39732}\""
newguid=`uuidgen`
newguid="${newguid^^}"
#cmd="echo "$ProductCode" | sed -r s/$uuidPtn/$newguid/" ## Not this
echo "$ProductCode"
#eval "$cmd" ## Not this either
# v v whole pattern quoted
changedcode=$(sed -r "s/$uuidPtn/$newguid/" <<<"$ProductCode")
# ^^ command substitution ^
# here-strings for input ^^^^^^^^^^^^^^^^^
echo "$changedcode"
Output:
"ProductCode" = "8:{0059DDB5-D384-46F9-BBFD-0004A8C39732}"
"ProductCode" = "8:{6094CF73-E23E-4655-B4A8-DAA57BE7EF72}"
This is a sh version
#!/bin/sh
alpha="0-9A-F"
uuidPtn="[$alpha]{8}-[$alpha]{4}-[$alpha]{4}-[$alpha]{4}-[$alpha]{12}"
ProductCode="\"ProductCode\" = \"8:{0059DDB5-D384-46F9-BBFD-0004A8C39732}\""
newguid=`uuidgen`
newguid=$(echo "${newguid}" | tr a-z A-Z)
ChangedCode=$(echo "$ProductCode" | sed -r s/$uuidPtn/$newguid/)
echo "$ProductCode"
echo "$ChangedCode"
I solved my own problem by changing the cmd= line to this:
cmd='echo $ProductCode | sed -r "s/$uuidPtn/$newguid/"'
thanks for the eyes on though folks.
I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}
sed -i "s|{FILES}|$FILES|" ./myFile.json
What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :
FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."
I can't think of a solution where xargs would help me.
The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.
For example, if myFile.json would contain:
{foo: 1, bar: "{FILES}"}
then the script should be:
#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF
You can generate the script with a single sed command:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json
Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.
Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash
Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin
Output:
{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}
Maybe perl would have fewer limitations?
perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json
It's a little gross, but you can do it all within shell...
while read l
do
if ! echo "$l" | grep -q '{DATA}'
then
echo "$l"
else
echo "$l" | sed 's/{DATA}.*$//'
echo "$FILES"
echo "$l" | sed 's/^.*{DATA}//'
fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json
Obviously I'd leave the final line commented until you were confident it worked...
Maybe just don't do it? Can you just :
echo "var f = " > myFile2.json
echo $FILES >> myFile2.json
And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)
Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:
foo.pl:
open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
s/world/$foo/;
print;
}
Command to run:
aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json
Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.
Simple question.
When I use sed to add \r\n into the variable
it fails.
how to add \r\n?
dateRecent=$(sed 's| 年| 年'"\r\n"'|g' <<< $newsDate)
dateRecent=$(sed 's| 年| 年\r\n|g' <<< $newsDate)
sed: -e expression #1, char 146: unterminated `s' command
The whole code is here:
cp /var/www/html/INFOSEC/textonly/sc_chi/anti/recent.html /var/www/html/INFOSEC/textonly/sc_chi/anti/recent.test.html
echo "Please input Date in this format(eg.2011 年 7 月 8 日):"
read -e newsDate
echo "Please input Title:"
read -e title
echo "Please input Description:"
read -e desc
echo "Please input ID(eg.d071101):"
read -e id
echo "Please input reference website:"
read -e web
echo "Confirm? Have to do it all over again if wrong (Yes:y, No:n)"
read -e confirm
dateRecent=$newsDate
if [[ "$mail" == "y" ]]; then
dateRecent=$(sed -e 's/ 年/ 年\r\n/g' <<< $newsDate)
fi
#Add Phishing attack in recent.html
sed -i '0,/<li>/ { s/<li>/<li><a href="'"$web"'" target="_blank">'"$dateRecent"' - '"$title"'<\/a><\/li>\r\n <li>/ }' /var/www/html/INFOSEC/textonly/sc_chi/anti/recent.test.html
Ppl can't re-create. So it might depends on sed version.
New gnused should handle \r\n like some in comments reports.
Older gnused and other sed might need the original newline and return re-produced. Hence you can use echo to get it and neglect sed impplementions, but this brings another dependency on your shell.
# ksh is my suggested standard style:
sed "s/ 年/ 年`echo -e \\\r\\\n`/g"
# zsh is like ksh and you can omit the -e for echo
# Old bash?:
sed 's/ 年'"/ 年`echo \\\r\\\n`/g"
Windows is easy, just double quote GNUSed is assumed.
sed "s/ 年/ 年\r\n/g"
See how life is a pain under AIX...
I have a fairly simple bash shell scripting problem.
I want to sed a piece of text and then assign the result of the sed to a variable.
#!/bin/bash
MOD_DATE=echo $(date) | sed 's/\ /_/g'
echo $MOD_DATE // should show date with spaces replaced with underscores.
I have tried the above and it doesn't work. Can anyone point out what I'm doing wrong?
To collect the output in stdout into a variable, use a command substitution:
MOD_DATE=`echo $(date) | sed 's/\ /_/g'`
# ^ ^
or
MOD_DATE=$(echo $(date) | sed 's/\ /_/g')
# ^^ ^
Maybe this can help:
mod_date = "$(date +"%d_%m_%Y")"
echo "$mod_date"