I have an TARGET.md file, I'm looking for a string and I want to replace it with the content of other md file, I have tried many combinations but it seems like the newline in the files are the ones sed is not liking, I just need to do this using pure bash(it doesn't have to be sed) because this is how the whole script is running:
This works:
local search="##### Header"
local replace="##### Header\\
\\
Line 1\\
Line 2\\
Line 3\\
Line 4"
sed -i '' -e "s/${search}/${replace}/" TARGET.md
But this won't:
file1.md content:
##### Header
Line 1
Line 2
Line 3
Line 4
Script:
local search="##### Header"
local replace=$(curl "path/to/file/in/other/place/file1.md")
sed -i '' -e "s/${search}/${replace}/" TARGET.md
NOTE: I don't have the file1.md in the same place, I'm doing a curl to get the raw content from it, this is why the replace is in a variable.
I'm assuming the concept is possible but my sed syntax is wrong knowing sed can handle newlines out of the box, but not sure what is the proper way to do this.
I've been searching for some days now, any help, tip or guide is appreciated!
You are using the wrong tool. sed is a line editor at heart. While you can repeatedly append to pattern space in some instances, awk with getline provides a more flexible solution. For example with your file1.md:
##### Header
Line 1
Line 2
Line 3
Line 4
and your TARGET.md as:
##### Unreleased
my dog
has fleas
The to replace "##### Unreleased" with the content of file1.md, you can do:
awk -v replace="file1.md" -v search="##### Unreleased" '
$0 == search {while (getline line < replace ) { print line }; next }
{ print }
' TARGET.md
Above you have your replace and search as with sed, but instead of using the line-editor, you use awk to locate the line containing search and the read all lines from replace using getline`. The second rule just prints all other lines as is.
Example Use/Output
In the directory containing each file, you can simply select-copy the above and middle-mouse paste into the terminal to test:
$ awk -v replace="file1.md" -v search="##### Unreleased" '
> $0 == search {while (getline line < replace ) { print line }; next }
> { print }
> ' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
Look things over and let me know if you have further questions.
Taking TARGET.md file from David's answer:
cat TARGET.md
##### Unreleased
my dog
has fleas
You can run sed with r command like this:
search="##### Unreleased"
sed -e "/$search/{r file1.md" -e ';d;}' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
Related
Suppose I have text as:
This is a sample text.
I have 2 sentences.
text is present there.
I need to replace whole text between two 'text' words. The required solution should be
This is a sample text.
I have new sentences.
text is present there.
I tried using the below command but its not working:
sed -i 's/text.*?text/text\
\nI have new sentence/g' file.txt
With your shown samples please try following. sed doesn't support lazy matching in regex. With awk's RS you could do the substitution with your shown samples only. You need to create variable val which has new value in it. Then in awk performing simple substitution operation will so the rest to get your expected output.
awk -v val="your_new_line_Value" -v RS="" '
{
sub(/text\.\n*[^\n]*\n*text/,"text.\n"val"\ntext")
}
1
' Input_file
Above code will print output on terminal, once you are Happy with results of above and want to save output into Input_file itself then try following code.
awk -v val="your_new_line_Value" -v RS="" '
{
sub(/text\.\n*[^\n]*\n*text/,"text.\n"val"\ntext")
}
1
' Input_file > temp && mv temp Input_file
You have already solved your problem using awk, but in case anyone else will be looking for a sed solution in the future, here's a sed script that does what you needed. Granted, the script is using some advanced sed features, but that's the fun part of it :)
replace.sed
#!/usr/bin/env sed -nEf
# This pattern determines the start marker for the range of lines where we
# want to perform the substitution. In our case the pattern is any line that
# ends with "text." — the `$` symbol meaning end-of-line.
/text\.$/ {
# [p]rint the start-marker line.
p
# Next, we'll read lines (using `n`) in a loop, so mark this point in
# the script as the beginning of the loop using a label called `loop`.
:loop
# Read the next line.
n
# If the last read line doesn't match the pattern for the end marker,
# just continue looping by [b]ranching to the `:loop` label.
/^text/! {
b loop
}
# If the last read line matches the end marker pattern, then just insert
# the text we want and print the last read line. The net effect is that
# all the previous read lines will be replaced by the inserted text.
/^text/ {
# Insert the replacement text
i\
I have a new sentence.
# [print] the end-marker line
p
}
# Exit the script, so that we don't hit the [p]rint command below.
b
}
# Print all other lines.
p
Usage
$ cat lines.txt
foo
This is a sample text.
I have many sentences.
I have many sentences.
I have many sentences.
I have many sentences.
text is present there.
bar
$
$ ./replace.sed lines.txt
foo
This is a sample text.
I have a new sentence.
text is present there.
bar
Substitue
sed -i 's/I have 2 sentences./I have new sentences./g'
sed -i 's/[A-Z]\s[a-z].*/I have new sentences./g'
Insert
sed -i -e '2iI have new sentences.' -e '2d'
I need to replace whole text between two 'text' words.
If I understand, first text. (with a dot) is at the end of first line and second text at the beginning of third line. With awk you can get the required solution adding values to var s:
awk -v s='\nI have new sentences.\n' '/text.?$/ {s=$0 s;next} /^text/ {s=s $0;print s;s=""}' file
This is a sample text.
I have new sentences.
text is present there.
UPDATED:
Using sed, how can I insert (NOT SUBSTITUTE) a new line on only the first match of keyword for each file.
Currently I have the following but this inserts for every line containing Matched Keyword and I want it to only insert the New Inserted Line for only the first match found in the file:
sed -ie '/Matched Keyword/ i\New Inserted Line' *.*
For example:
Myfile.txt:
Line 1
Line 2
Line 3
This line contains the Matched Keyword and other stuff
Line 4
This line contains the Matched Keyword and other stuff
Line 6
changed to:
Line 1
Line 2
Line 3
New Inserted Line
This line contains the Matched Keyword and other stuff
Line 4
This line contains the Matched Keyword and other stuff
Line 6
You can sort of do this in GNU sed:
sed '0,/Matched Keyword/s//New Inserted Line\n&/'
But it's not portable. Since portability is good, here it is in awk:
awk '/Matched Keyword/ && !x {print "Text line to insert"; x=1} 1' inputFile
Or, if you want to pass a variable to print:
awk -v "var=$var" '/Matched Keyword/ && !x {print var; x=1} 1' inputFile
These both insert the text line before the first occurrence of the keyword, on a line by itself, per your example.
Remember that with both sed and awk, the matched keyword is a regular expression, not just a keyword.
UPDATE:
Since this question is also tagged bash, here's a simple solution that is pure bash and doesn't required sed:
#!/bin/bash
n=0
while read line; do
if [[ "$line" =~ 'Matched Keyword' && $n = 0 ]]; then
echo "New Inserted Line"
n=1
fi
echo "$line"
done
As it stands, this as a pipe. You can easily wrap it in something that acts on files instead.
If you want one with sed*:
sed '0,/Matched Keyword/s//Matched Keyword\nNew Inserted Line/' myfile.txt
*only works with GNU sed
This might work for you:
sed -i -e '/Matched Keyword/{i\New Inserted Line' -e ':a;n;ba}' file
You're nearly there! Just create a loop to read from the Matched Keyword to the end of the file.
After inserting a line, the remainder of the file can be printed out by:
Introducing a loop place holder :a (here a is an arbitrary name).
Print the current line and fetch the next into the pattern space with the ncommand.
Redirect control back using the ba command which is essentially a goto to the a place holder. The end-of-file condition is naturally taken care of by the n command which terminates any further sed commands if it tries to read passed the end-of-file.
With a little help from bash, a true one liner can be achieved:
sed $'/Matched Keyword/{iNew Inserted Line\n:a;n;ba}' file
Alternative:
sed 'x;/./{x;b};x;/Matched Keyword/h;//iNew Inserted Line' file
This uses the Matched Keyword as a flag in the hold space and once it has been set any processing is curtailed by bailing out immediately.
If you want to append a line after first match only, use AWK instead of SED as below
awk '{print} /Matched Keyword/ && !n {print "New Inserted Line"; n++}' myfile.txt
Output:
Line 1
Line 2
Line 3
This line contains the Matched Keyword and other stuff
New Inserted Line
Line 4
This line contains the Matched Keyword and other stuff
Line 6
I have a big file that might have some comments in the header
# Comment line 1
# Comment line 2
# ...
# Comment line ...
# ...
# Comment line N
123
234
345
...
I need a one line solution to get the number "N+1", what's the most elegant way to achieve this using shell script? Thanks
try this sed one-liner:
sed -n '/^\s*#/!{p;q}' file
ok, if you need only the line-number:
sed -n '/^\s*#/!{=;q}' file
add short explanation:
/^\s*#/! : regex, if the line does NOT start with 0 or more empty chars (tab/space)then a '#', the line is chosen for further step.
= : print line no
q : quit processing
awk version:
awk '$1~/^[^#]/{print NR; exit}' file
So I grep for something in some file:
grep "import" test.txt | tail -1
In test.txt there is
import-one
import-two
import-three
some other stuff in the file
This will return the last search result:
import-three
Now how do I add some text -after-- import-three but before "some other stuff in the file". Basically I want to append a line but not at the end of a file but after a search result.
I understand that you want some text after each search result, which would mean after every matching line. So try
grep "import" test.txt | sed '/$/ a\Line to be added'
You can try something like this with sed
sed '/import-three/ a\
> Line to be added' t
Test:
$ sed '/import-three/ a\
> Line to be added' t
import-one
import-two
import-three
Line to be added
some other stuff in the file
One way assuming that you cannot distingish between different "import" sentences. It reverses the file with tac, then find the first match (import-three) with sed, insert a line just before it (i\) and reverse again the file.
The :a ; n ; ba is a loop to avoid processing again the /import/ match.
The command is written throught several lines because the sed insert command is very special with the syntax:
$ tac infile | sed '/import/ { i\
"some text"
:a
n
ba }
' | tac -
It yields:
import-one
import-two
import-three
"some text"
some other stuff in the file
Using ed:
ed test.txt <<END
$
?^import
a
inserted text
.
w
q
END
Meaning: go to the end of the file, search backwards for the first line beginning with import, add the new lines below (insertion ends with a "." line), save and quit
I would like to insert lines into a file in bash starting from a specific line.
Each line is a string which is an element of an array
line[0]="foo"
line[1]="bar"
...
and the specific line is 'fields'
file="$(cat $myfile)"
for p in $file; do
if [ "$p" = 'fields' ]
then insertlines() #<- here
fi
done
This can be done with sed: sed 's/fields/fields\nNew Inserted Line/'
$ cat file.txt
line 1
line 2
fields
line 3
another line
fields
dkhs
$ sed 's/fields/fields\nNew Inserted Line/' file.txt
line 1
line 2
fields
New Inserted Line
line 3
another line
fields
New Inserted Line
dkhs
Use -i to save in-place instead of printing to stdout
sed -i 's/fields/fields\nNew Inserted Line/'
As a bash script:
#!/bin/bash
match='fields'
insert='New Inserted Line'
file='file.txt'
sed -i "s/$match/$match\n$insert/" $file
Or anoter one example with the sed:
Prepare a test.txt file:
echo -e "line 1\nline 2\nline 3\nline 4" > /tmp/test.txt
cat /tmp/test.txt
line 1
line 2
line 3
line 4
Add a new line into the test.txt file:
sed -i '2 a line 2.5' /tmp/test.txt
# sed for in-place editing (-i) of the file: 'LINE_NUMBER a-ppend TEXT_TO_ADD'
cat /tmp/test.txt
line 1
line 2
line 2.5
line 3
line 4
This is definitely a case where you want to use something like sed (or awk or perl) rather than readling one line at a time in a shell loop. This is not the sort of thing the shell does well or efficiently.
You might find it handy to write a reusable function. Here's a simple one, though it won't work on fully-arbitrary text (slashes or regular expression metacharacters will confuse things):
function insertAfter # file line newText
{
local file="$1" line="$2" newText="$3"
sed -i -e "/^$line$/a"$'\\\n'"$newText"$'\n' "$file"
}
Example:
$ cat foo.txt
Now is the time for all good men to come to the aid of their party.
The quick brown fox jumps over a lazy dog.
$ insertAfter foo.txt \
"Now is the time for all good men to come to the aid of their party." \
"The previous line is missing 'bjkquvxz.'"
$ cat foo.txt
Now is the time for all good men to come to the aid of their party.
The previous line is missing 'bjkquvxz.'
The quick brown fox jumps over a lazy dog.
$
sed is your friend:
:~$ cat text.txt
foo
bar
baz
~$
~$ sed '/^bar/\na this is the new line/' text.txt > new_text.txt
~$ cat new_text.txt
foo
bar
this is the new line
baz
~$