I want to append bash variable (which has html tags) at the beginning of a file.
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
Below is the command I am using -
sed -i '1i'$INSERTTO file.html
But i am getting error -
sed: -e expression #1, char 177: unknown command: `<'
Do i need to encode the html tags in INSERTTO variable ?
First of all, use quotes:
INSERTTO='<h2>title</h2>
<li>sdfdsf</li>'
Then try this:
sed "1 i $INSERTTO" file.html
Start with some text in your file, e.g.
$ cat file
some text
Then you need your variable to contain an explicit '\n' character where the line break is, e.g. INSERTTO='<h2>title</h2>\n<li>sdfdsf</li>'. Then you can use the sed expression to place both lines as the beginning lines in the file, e.g.
$ INSERTTO='<h2>title</h2>\n<li>sdfdsf</li>'; sed "1i $INSERTTO" file
<h2>title</h2>
<li>sdfdsf</li>
some text
Now at present, what will be done has only been written to the terminal stdout. To modify the file in place, you will need to add the -i option for sed (or -i.bak to save a backup of the original file with the .bak extension. (however you prefer to do it)
I offer you an ed(1) solution.
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
printf '%s\n' 1i "$INSERTTO" . w | ed -s file.html
Assuming the file is not empty that will work. otherwise use 0a as the address and command instead of 1i
... Or use the cat(1) and mv(1)
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
Add the stdin flag and use a herestring works in bash but not in POSIX shells.
cat - file.html <<< "$INSERTTO"
You should see the output to stdout, redirect it to another file and move that file to the original file, something like this.
cat - file.html <<< "$INSERTTO" > tempfile && mv tempfile file.html
However if the html file is a symlink then it is now broken... a work around would be to use another cat.
cat - file.html <<< "$INSERTTO" > tempfile && cat tempfile > file.html && rm tempfile.
tempfile is just an example you should take a loot at How to create a tempfile in a secure manner
Related
I have these files
NotRequired.txt (having lines which need to be remove)
Need2CleanSED.txt (big file , need to clean)
Need2CleanGRP.txt (big file , need to clean)
content:
more NotRequired.txt
[abc-xyz_pqr-pe2_123]
[lon-abc-tkt_1202]
[wat-7600-1_414]
[indo-pak_isu-5_761]
I am reading above file and want to remove lines from Need2Clean???.txt, trying via SED and GREP but no success.
myFile="NotRequired.txt"
while IFS= read -r HKline
do
sed -i '/$HKline/d' Need2CleanSED.txt
done < "$myFile"
myFile="NotRequired.txt"
while IFS= read -r HKline
do
grep -vE \"$HKline\" Need2CleanGRP.txt > Need2CleanGRP.txt
done < "$myFile"
Looks as if the Variable and characters [] making some problem.
What you're doing is extremely inefficient and error prone. Just do this:
grep -vF -f NotRequired.txt Need2CleanGRP.txt > tmp &&
mv tmp Need2CleanGRP.txt
Thanks to grep -F the above treats each line of NotRequired.txt as a string rather than a regexp so you don't have to worry about escaping RE metachars like [ and you don't need to wrap it in a shell loop - that one command will remove all undesirable lines in one execution of grep.
Never do command file > file btw as the shell might decide to execute the > file first and so empty file before command gets a chance to read it! Always do command file > tmp && mv tmp file instead.
Your assumption is correct. The [...] construct looks for any characters in that set, so you have to preface ("escape") them with \. The easiest way is to do that in your original file:
sed -i -e 's:\[:\\[:' -e 's:\]:\\]:' "${myFile}"
If you don't like that, you can probably put the sed command in where you're directing the file in:
done < replace.txt|sed -e 's:\[:\\[:' -e 's:\]:\\]:'
Finally, you can use sed on each HKline variable:
HKline=$( echo $HKline | sed -e 's:\[:\\[:' -e 's:\]:\\]:' )
try gnu sed:
sed -Ez 's/\n/\|/g;s!\[!\\[!g;s!\]!\\]!g; s!(.*).!/\1/d!' NotRequired.txt| sed -Ef - Need2CleanSED.txt
Two sed process are chained into one by shell pipe
NotRequired.txt is 'slurped' by sed -z all at once and substituted its \n and [ meta-char with | and \[ respectively of which the 2nd process uses it as regex script for the input file, ie. Need2CleanSED.txt. 1st process output;
/\[abc-xyz_pqr-pe2_123\]|\[lon-abc-tkt_1202\]|\[wat-7600-1_414\]|\[indo-pak_isu-5_761\]/d
add -u ie. unbuffered, option to evade from batch process, sort of direct i/o
Suppose my file a.conf is as following
Include /1
Include /2
Include /3
I want to replace "Include /2" with a new line, I write the code in .sh file :
line="Include /2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf
But after running the sh file, It give me the following error
sed: -e expression #1, char 14: unknown option to `s'
If you are using a newer version of sed you can use -i to read from and write to the same file. Using -i you can specify a file extension so a backup will be made, incase something went wrong. Also you don't need to use the -e flag unless you are using multiple commands
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
I have just noticed that as the variables you are using are quoted strings you may want to use single quotes around your sed expression. Also your string contains a forward slash, to avoid any errors you can use a different delimiter in your sed command (the delimiter doesn't need to be a slash):
sed -i.bak 's|${line}|${rep}|g' /root/new_scripts/a.conf
You have to write the changes to a new file and then, move the new file over the old one. Like this:
line="Include 2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf > /root/new_scripts/a.conf-new
mv /root/new_scripts/a.conf-new /root/new_scripts/a.conf
The redirection (> /root/new_scripts/a.conf) wipes the contents of the file before sed can see it.
You need to pass the -i option to sed to edit the file in-place:
sed -i "s/${line}/${rep}/g" /root/new_scripts/a.conf
You can also ask sed to create a backup of the original file:
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
So, if you have to replace a substring in a file, you can use sed command like this, say we have a file as file.txt, so replacing a substring in it can be done like this
searchString="abc";
replaceString="def";
sed -i '' "s|$searchString|$replaceString|g" file.txt
This will all the occurrences of "abc" with "def" in file.txt. Also, this keeps a check for any / character present in the variables used, and with no backup file made.
I have a large number of words in a text file to replace.
This script is working up until the sed command where I get:
sed: 1: "*.js": invalid command code *
PS... Bash isn't one of my strong points - this doesn't need to be pretty or efficient
cd '/Users/xxxxxx/Sites/xxxxxx'
echo `pwd`;
for line in `cat myFile.txt`
do
export IFS=":"
i=0
list=()
for word in $line; do
list[$i]=$word
i=$[i+1]
done
echo ${list[0]}
echo ${list[1]}
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
done
You're running BSD sed (under OS X), therefore the -i flag requires an argument specifying what you want the suffix to be.
Also, no files match the glob *.js.
This looks like a simple typo:
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
Should be:
sed -i "s/${list[0]}/${list[1]}/g" *.js
(just like the echo lines above)
So myFile.txt contains a list of from:to substitutions, and you are looping over each of those. Why don't you create a sed script from this file instead?
cd '/Users/xxxxxx/Sites/xxxxxx'
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt |
# Output from first sed script is a sed script!
# It contains substitutions like this:
# s:from:to:
# s:other:substitute:
sed -f - -i~ *.js
Your sed might not like the -f - which means sed should read its script from standard input. If that is the case, perhaps you can create a temporary script like this instead;
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt >script.sed
sed -f script.sed -i~ *.js
Another approach, if you don't feel very confident with sed and think you are going to forget in a week what the meaning of that voodoo symbols is, could be using IFS in a more efficient way:
IFS=":"
cat myFile.txt | while read PATTERN REPLACEMENT # You feed the while loop with stdout lines and read fields separated by ":"
do
sed -i "s/${PATTERN}/${REPLACEMENT}/g"
done
The only pitfall I can see (it may be more) is that if whether PATTERN or REPLACEMENT contain a slash (/) they are going to destroy your sed expression.
You can change the sed separator with a non-printable character and you should be safe.
Anyway, if you know whats on your myFile.txt you can just use any.
I recently wrote a script that does a sed command, to replace all the occurrences of "string1" with "string2" in a file named "test.txt".
It looks like this:
sed -i 's/string1/string2/g' test.txt
The catch is, "string1" does not necessarily exist in test.txt.
I notice after executing a bunch of these sed commands, I get a number of empty files, left behind in the directory, with names that look like this:
"sed4l4DpD"
Does anyone know why this might be, and how I can correct it?
-i is the suffix given to the new/output file. Also, you need -e for the command.
Here's how you use it:
sed -i '2' -e 's/string1/string2/g' test.txt
This will create a file called test.txt2 that is the backup of test.txt
To replace the file (instead of creating a new copy - called an "in-place" substitution), change the -i value to '' (ie blank):
sed -i '' -e 's/string1/string2/g' test.txt
EDIT II
Here's actual command line output from a Mac (Snow Leopard) that show that my modified answer (removed space from between the -i and the suffix) is correct.
NOTE: On a linux server, there must be no space between it -i and the suffix.
> echo "this is a test" > test.txt
> cat test.txt
this is a test
> sed -i '2' -e 's/a/a good/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a good test
> cat test.txt2
this is a test
> sed -i '' -e 's/a/a really/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a really good test
I wasn't able to reproduce this with a quick test (using GNU sed 4.2.1) -- but strace did show sed creating a file called sedJd9Cuy and then renaming it to tmp (the file named on the command line).
It looks like something is going wrong after sed creates the temporary file and before it's able to rename it.
My best guess is that you've run out of room in the filesystem; you're able to create a new empty file, but unable to write to it.
What does df . say?
EDIT:
I still don't know what's causing the problem, but it shouldn't be too difficult to work around it.
Rather than
sed -i 's/string1/string2/g' test.txt
try something like this:
sed 's/string1/string2/g' test.txt > test.txt.$$ && mv -f test.txt.$$ test.txt
Something is going wrong with the way sed creates and then renames a text file to replace your original file. The above command uses sed as a simple input-output filter and creates and renames the temporary file separately.
So after much testing last night, it turns out that sed was creating these files when trying to operate on an empty string. The way i was getting the array of "$string1" arguments was through a grep command, which seems to be malformed. What I wanted from the grep was all lines containing something of the type "Text here '.'".
For example the string, "Text here 'ABC.DEF'" in a file, should have been caught by grep, then the ABC.DEF portion of the string, would be substituted by ABC_DEF. Unfortunately the grep I was using would catch lines of the type "Text here ''" (that is, nothing between the ''). When later on, the script attempted to perform a sed replacement using this empty string, the random file was created (probably because sed died).
Thanks for all your help in understanding how sed works.
Its better if you do it in this way:
cat large_file | sed 's/string1/string2/g' > file_filtred
sed 's/$/<br>/' mytext.txt
worked but output it all on the command line. I want it to just do the replacement WITHIN the specified file itself. Should I be using another tool?
If you have gnu sed, you can use the -i option, which will do the replacement in place.
sed -i 's/$/<br>/' mytext.txt
Otherwise you will have to redirect to another file and rename it over the old one.
sed 's/$/<br>/' mytext.txt > mytext.txt.new && mv mytext.txt.new mytext.txt
Just for completeness. On Mac OS X (which uses FreeBSD sed) you have to use an additional null-string "" for in-place file editing without backup:
sed -i "" 's/$/<br>/' mytext.txt
As an alternative to using sed for no-backup in-place file editing, you may use ed(1), which, however, reads the entire file into memory before operating on it.
printf '%s\n' H 'g/$/s//<br>/g' ',p' | ed -s test.file # print to stdout
printf '%s\n' H 'g/$/s//<br>/g' wq | ed -s test.file # in-place file edit
For more information on ed(1) see:
"Editing files with the ed text editor from scripts",
http://wiki.bash-hackers.org/doku.php?id=howto:edit-ed
If you have an up-to-date sed, just use sed -i or sed --in-place, which will modify the actual file itself.
If you want a backup, you need to supply a suffix for it. So sed -i.bak or sed --in-place=.bak will create a backup file with the .bak suffix.
Use this! Seriously! You'll appreciate it a lot the first time you damage your file due to a mistyped sed command or wrong assumption about the data in the file.
Use the redirection symbol i.e.
sed 's/$/<br>/' mytext.txt > mytext2.txt && mv mytext2.txt mytext.txt