$ cat test.txt
foo
$ echo " bar" | tee -a test.txt
foo
bar
Expected result for cat test.txt is foo bar.
You can use sed:
sed -i~ '$ s/$/ bar/' test.txt
$ is an address that means "the last line". It applies to the following command.
s/$/ bar/ replaces $, i.e. the end of line, by bar.
-i (if supported) will change the file in place, leaving the original as a backup (renamed to test.txt~). If your sed doesn't support it, redirect the output to a new file and move it over the old one.
I prefer to use ed over sed -i to edit files in place, as, unlike sed -i, it's standardized and works the same everywhere:
printf "%s\n" '$s/$/ bar/' w | ed -s test.txt
The commands are very similar; sed descended from ed as a way to work on text in the middle of a pipeline. The big change is w to write the changed file back to disk.
Related
I want to replace a line in a file with multiple lines. I know I can use \n in the sed replace, but that is rather ugly. I was hoping to HEARDOCs.
So I can do this to replace the line with multiple lines:
$ cat sedtest
DINGO=bingo
$ sed -i -e "s/^DINGO.*$/# added by $(whoami) on $(date)\nDINGO=howdy/" sedtest
$ cat sedtest
# added by user on Sun Feb 3 08:55:44 EST 2019
DINGO=howdy
In the command I want to put the replacement in new lines so it's easier to read/understand. So far I have been using HEREDOCs when I want to add new lines to a file:
CAT << EOF | sudo tee -a file1 file2 file3
line one
line two
line three
EOF
And this has worked well for appending/adding. Is it possible to do something similar but instead use the output as the replacement in sed or is there some other way to do what I'm looking for?
Is this what you're trying to do?
$ awk 'NR==FNR{new=(NR>1?new ORS:"") $0;next} $0=="DINGO=bingo"{$0=new} 1' - file <<!
# added by $(whoami) on $(date)
DINGO=howdy
!
# added by user on Sun, Feb 3, 2019 8:50:41 AM
DINGO=howdy
Note that the above is using literal string operations so it'll work for any characters in the old or new strings unlike your sed script which would fail given /s or any ERE regexp character or capture groups or backreferences or... in the input (see Is it possible to escape regex metacharacters reliably with sed for details).
This might work for you (GNU sed):
cat <<! | sed -i -e '/^DINGO/r /dev/stdin' -e '//d' file
# added by $(whoami) on $(date)
DINGO=howdy
!
This replaces lines starting DINGO with the here-document which is piped to stdin as file within the sed command.
An alternative:
cat <<! | sed -i -e '/^DINGO/e cat /dev/stdin' -e 's/bingo/howdy' file
# added by $(whoami) on $(date)
!
N.B. In the alternative solution the here-doc will only be read once!
This question already has answers here:
Sed Insert Multiple Lines
(3 answers)
Closed 1 year ago.
I have a multiline variable that I captured from STDOUT.
I want to insert an echo command using this multiline variable to line 15 in another script (target).
#!/bin/bash
TEST=`cat foo`
echo "$TEST"
sed -i "15i echo \"$TEST\" > someotherfile" target
Contents of foo :
apples
oranges
bananas
carrots
I thought the sed command read in line feeds, which I confirmed my foo has:
user#test$ cat foo | tr -cd '\n' | wc -c
4
When I run my test.sh script, I see what's in $TEST, but am getting an error for the sed command:
user#test$ ./test.sh
apples
oranges
bananas
carrots
sed: -e expression #1, char 18: unknown command: `o'
What am I doing wrong?
Thanks in advance.
GNU sed is assumed, as implied by the syntax used in the question.
#!/bin/bash
# Read contents of file 'foo' into shell variable $test.
test=$(<foo)
# \-escape the newlines in $test for use in Sed.
testEscapedForSed=${test//$'\n'/\\$'\n'}
sed -i "15i echo \"$testEscapedForSed\" > someotherfile" target
Your problem was that passing multi-line strings to sed functions such as i (insert) requires the newlines embedded in those strings to be \-escaped, so that sed knows where the string ends and additional commands, if any, start.
A (nonstandard) parameter expansion is used to replace all newlines in $test with themselves prefixed by \, using ANSI C-quoted string $'\n' to generate actual newline chars.
Also note:
I've renamed TEST to test, because all-uppercase shell-variable names should be avoided.
I've used modern command-substitution syntax $(..) in lieu of legacy syntax `...`.
$(<foo) is a slightly more efficient - although nonstandard - way of reading the content of a file at once.
Try:
Solution1:
awk 'NR==15{print;system("cat foo");next} 1' Input_file
No need to get the complete file into a variable, we could simply print it whichever line of Input_file you want to print it.
Solution2:
line=15; sed -e "${line}r foo" target
Or (in script mode)
cat script.ksh
line=15;
sed -e "${line}r foo" target
Where you could change the number of line where you want to insert the lines from another file.
The i command in sed inserts the lines of text that end with a newline, up until a line that doesn't end with a backslash. The a and c commands are similar. Classic sed doesn't like the first line to appear on the same line as the i command; GNU sed isn't as fussy.
If you were writing the command manually, you'd need to write:
15i\
echo "apples\
oranges\
bananas\
carrots" > someotherfile
At issue now is "how do you want to create this given the file foo contains the list of names?". Sometimes, using sed to generate the sed script is useful. However, it can also be intricate if you need to get backslashes at the ends of lines which are subject to an i (or a or c) command, and it is simpler to circumvent the problem.
{
echo "15i\\"
sed -e '1s/^/echo "/' -e 's/$/\\/' -e '$s/\\$/" > someotherfile/' foo
} | sed -f /dev/stdin target
GNU sed can read its script from standard input using -f -; BSD (macOS) sed doesn't like that, but you can use -f /dev/stdin instead (which also works with GNU sed), at least on systems where there is a /dev/stdin.
Interesting issue.
As already mentioned the whole story for sed to be able to insert multiline text in another file is that this new multiline text must have actually literral \n for line breaks.
So we can use sed to convert real new line chars to literal \n:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
#Alternative: a=$(sed "s#[\]#\0n#g" <(sed 's#[\]$##' <(tr '\n' '\\' <file3)))
$ echo "$a"
apples\noranges\nbananas\ncarrots
How this translation works:
* First we replace all new lines with a single backslash using tr
* Then we remove the backslash from the end of the string
* Then we replace all other backslashes with backaslash and n char.
Since now variable $a contains literal \n between lines, sed will translate them back to actuall new lines:
$ cat file4
Line1
line2
line3
$ sed "2i $a" file4
Line1
apples
oranges
bananas
carrots
line2
line3
Result:
Mutliline replacement can be done with two commands:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
$ sed "2i $a" file4
sed 2i means insert a text before line2. 2a can be used in order to insert something after line2.
Remark:
According to this post which seems to be a duplicate, translation of new lines to literal \n seems that can be done with just :
a=$(echo ${a} | tr '\n' "\\n")
But this method never worked in my system.
Remark2:
The sed operation sed "2i $a" = insert variable $a before line 2 , can be also expressed as sed "1 s/.*/\0\n$a/" = replace all chars of first line with the same chars \0 plus a new line \n plus the contents of variable $a => insert $a after line1 = insert $a before line2.
So I have a file containing millions of lines.
and now within the file I have occurrences such as
=Continent
=Country
=State
=City
=Street
Now I have an excel file in which I have the text that should replace these occurrences - as an example :
=Continent should be replaced with =Asia
Similarly for other text
Now I was thinking of writing a java program to read my input file , read the mapping file and for each occurrence search and replace.
I am being lazy here - was wondering if I could do the same using editors like VIM ?
would that be possible ?
NOTE - I dont want to do a single text replace - I have multiple text that need to be found and replaced and I dont want to do the search and replace manually for each.
EDIT1:
Contents of my file that I want to replace: "1.txt"
continent=cont_text
country=country_text
The file that contains the values I want to replace with : "to_replace.txt"
=cont_text~Asia
=country_text~India
and finally using 'sed' here is my .sh file - but I am doing something wrong - it does not replace the contents of "1.txt"
while IFS="~" read foo bar;
do
echo $foo
echo $bar
for filename in 1.txt; do
sed -i.backup 's/$foo/$bar/g;' $filename
done
done < to_replace.txt
You can't put $foo and $bar in single quotes because the shell won't expand them. You don't need the for $filename in 1.txt loop because sed will loop through the lines of 1.txt. And you can't use -i.backup inside the loop because it will change the backup file each time and not preserve the original. So your script should be:
#!/bin/bash
cp 1.txt 1.txt.backup
while IFS="~" read foo bar;
do
echo $foo
echo $bar
sed -i "s/$foo/=$bar/g;" 1.txt
done < to_replace.txt
Output:
$ cat 1.txt
continent=Asia
country=India
sed is for simple substitutions on individual lines and shell is an environment from which to call tools not a tool to manipulate text so any time you write a shell loop to manipulate text you are doing it wrong.
Just use the tool that the same guys who invented sed and shell also invented to do general text processing jobs like this, awk:
$ awk -F'[=~]' -v OFS="=" 'NR==FNR{map[$2]=$3;next} {$2=map[$2]} 1' to_replace.txt 1.txt
continent=Asia
country=India
This sed command will do it without any loop:
sed -n 's#\(^=[^~]*\)~\(.*\)#s/\1/=\2/g#p' to_replace.txt |sed -i -f- 1.txt
Or sed with extended regex:
sed -nr 's#(^=[^~]*)~(.*)#s/\1/=\2/g#p' to_replace.txt | sed -i -f- 1.txt
Explanation:
The sed command:
sed -n 's#\(^=[^~]*\)~\(.*\)#s/\1/=\2/g#p' to_replace.txt
generates an output:
s/=cont_text/=Asia/g
s/=country_text/=India/g
which is then used as a sed script for the next sed after the pipe.
$ cat 1.txt
continent=Asia
country=India
Could you please let me know how to replace a particular string present in a text file or ksh file in the server with another string ?
For example :-
I have 10 files present in the path /file_sys/file in which i have to replace the word "BILL" to "BILLING" in all the 10 files.
Works for me:
I created a file 'test' with this content: "This is a simple test". Now I execute this call to the sed command:
sed -i 's/ is / is not /' test
Afterwards the file 'test' contains this content: "This is not a simple test"
If your sed utility does not support the -i flag, then there is a somewhat awkward workaround:
sed 's/ is / is not /' test > tmp_test && mv tmp_test test
This should work. Please find the testing as well.
$ cat > file1
I am a BILL boy
sed 's/[[:alnum:] [:cntrl:] [:lower:] [:space:] [:alpha:] [:digit:] [:print:] [:upper:] [:blank:] [:graph:] [:punct:] [:xdigit:]]BILL[[:alnum:] [:cntrl:] [:lower:] [:space:] [:alpha:] [:digit:] [:print:] [:upper:] [:blank:] [:graph:] [:punct:] [:xdigit:]]/BILLING/g' file1>file2
$ cat file2
I am a BILLING boy
Using sed:
sed 's/\bBILL\b/BILLING/g' file
For inplace:
sed --in-place 's/\bBILL\b/BILLING/g' file
A little for loop might assist for dealing with multiple files, and here I'm assuming -i option is not available:
for file in $(grep -wl BILL /file_sys/file/*); do
echo $file
sed -e 's/\bBILL\b/BILLING/g' $file > tmp
mv tmp $file
done
Here's what's happening:
grep -w Search for all (and only) files with the word BILL
grep -l Listing the file names (rather than content)
$(....) Execute whats inside the brackets (command substitution)
for file in Loop over each item in the list (each file with BILL in it)
echo $file Print each file name we loop over
sed command Replace the word BILL (here, specifically delimited with word boundaries "\b") with BILLING, into a tmp file
mv command Move the tmp file back to the original name (replace original)
You can easily test this without actually changing anything - e.g. just print the file name, or just print the contents (to make sure you've got what you expect before replacing the original files).
How can I join multiple lines into one line, with a separator where the new-line characters were, and avoiding a trailing separator and, optionally, ignoring empty lines?
Example. Consider a text file, foo.txt, with three lines:
foo
bar
baz
The desired output is:
foo,bar,baz
The command I'm using now:
tr '\n' ',' <foo.txt |sed 's/,$//g'
Ideally it would be something like this:
cat foo.txt |join ,
What's:
the most portable, concise, readable way.
the most concise way using non-standard unix tools.
Of course I could write something, or just use an alias. But I'm interested to know the options.
Perhaps a little surprisingly, paste is a good way to do this:
paste -s -d","
This won't deal with the empty lines you mentioned. For that, pipe your text through grep, first:
grep -v '^$' | paste -s -d"," -
This sed one-line should work -
sed -e :a -e 'N;s/\n/,/;ba' file
Test:
[jaypal:~/Temp] cat file
foo
bar
baz
[jaypal:~/Temp] sed -e :a -e 'N;s/\n/,/;ba' file
foo,bar,baz
To handle empty lines, you can remove the empty lines and pipe it to the above one-liner.
sed -e '/^$/d' file | sed -e :a -e 'N;s/\n/,/;ba'
How about to use xargs?
for your case
$ cat foo.txt | sed 's/$/, /' | xargs
Be careful about the limit length of input of xargs command. (This means very long input file cannot be handled by this.)
Perl:
cat data.txt | perl -pe 'if(!eof){chomp;$_.=","}'
or yet shorter and faster, surprisingly:
cat data.txt | perl -pe 'if(!eof){s/\n/,/}'
or, if you want:
cat data.txt | perl -pe 's/\n/,/ unless eof'
Just for fun, here's an all-builtins solution
IFS=$'\n' read -r -d '' -a data < foo.txt ; ( IFS=, ; echo "${data[*]}" ; )
You can use printf instead of echo if the trailing newline is a problem.
This works by setting IFS, the delimiters that read will split on, to just newline and not other whitespace, then telling read to not stop reading until it reaches a nul, instead of the newline it usually uses, and to add each item read into the array (-a) data. Then, in a subshell so as not to clobber the IFS of the interactive shell, we set IFS to , and expand the array with *, which delimits each item in the array with the first character in IFS
I needed to accomplish something similar, printing a comma-separated list of fields from a file, and was happy with piping STDOUT to xargs and ruby, like so:
cat data.txt | cut -f 16 -d ' ' | grep -o "\d\+" | xargs ruby -e "puts ARGV.join(', ')"
I had a log file where some data was broken into multiple lines. When this occurred, the last character of the first line was the semi-colon (;). I joined these lines by using the following commands:
for LINE in 'cat $FILE | tr -s " " "|"'
do
if [ $(echo $LINE | egrep ";$") ]
then
echo "$LINE\c" | tr -s "|" " " >> $MYFILE
else
echo "$LINE" | tr -s "|" " " >> $MYFILE
fi
done
The result is a file where lines that were split in the log file were one line in my new file.
Simple way to join the lines with space in-place using ex (also ignoring blank lines), use:
ex +%j -cwq foo.txt
If you want to print the results to the standard output, try:
ex +%j +%p -scq! foo.txt
To join lines without spaces, use +%j! instead of +%j.
To use different delimiter, it's a bit more tricky:
ex +"g/^$/d" +"%s/\n/_/e" +%p -scq! foo.txt
where g/^$/d (or v/\S/d) removes blank lines and s/\n/_/ is substitution which basically works the same as using sed, but for all lines (%). When parsing is done, print the buffer (%p). And finally -cq! executing vi q! command, which basically quits without saving (-s is to silence the output).
Please note that ex is equivalent to vi -e.
This method is quite portable as most of the Linux/Unix are shipped with ex/vi by default. And it's more compatible than using sed where in-place parameter (-i) is not standard extension and utility it-self is more stream oriented, therefore it's not so portable.
POSIX shell:
( set -- $(cat foo.txt) ; IFS=+ ; printf '%s\n' "$*" )
My answer is:
awk '{printf "%s", ","$0}' foo.txt
printf is enough. We don't need -F"\n" to change field separator.