Insert a variable in a text file [duplicate] - bash

This question already has answers here:
Replace a string in shell script using a variable
(12 answers)
Closed 4 years ago.
I am trying to using sed -i command to insert a string variable in the 1st line of a text file.
This command work : sed -i '1st header' file.txt
But when i pass a variable this doesn't work.
example :
var=$(cat <<-END
This is line one.
This is line two.
This is line three.
END
)
sed -i '1i $var' file.txt # doesn't work
sed -i ’1i $var’ file.txt # doesn't work
Any help with this problem
Thank you

First, let's define your variable a simpler way:
$ var="This is line one.
This is line two.
This is line three."
Since sed is not good at working with variables, let's use awk. This will place your variable at the beginning of a file:
awk -v x="$var" 'NR==1{print x} 1' file.txt
How it works
-v x="$var"
This defines an awk variable x to have the value of shell variable $var.
NR==1{print x}
At the first line, this tells awk to insert the value of variable x.
1
This is awk's shorthand for print-the-line.
Example
Let's define your variable:
$ var="This is line one.
> This is line two.
> This is line three."
Let's work on this test file:
$ cat File
1
2
This is what the awk command produces:
$ awk -v x="$var" 'NR==1{print x} 1' File
This is line one.
This is line two.
This is line three.
1
2
Changing a file in-place
To change file.txt in place using a recent GNU awk:
awk -i inplace -v x="$var" 'NR==1{print x} 1' file.txt
On macOS, BSD or older GNU/Linux, use:
awk -v x="$var" 'NR==1{print x} 1' file.txt >tmp && mv tmp file.txt

Using printf...
$ var="This is line one.
This is line two.
This is line three.
"
Use cat - to read from stdin and then print into a new file. Move it to the original file if you want to modify it.
$ printf "$var" | cat - file > newfile && mv newfile file;

Not the best job for sed. What about a simple cat ?
cat - file.txt <<EOF > newfile.txt
This is line one.
This is line two.
This is line three.
EOF
# you can add mv, if you really want the original file gone
mv newfile.txt file.txt
And for the original problem - sed does not like newlines and spaces in it's 'program', you need to quote and escape the line breaks:
# this works
sed $'1i "abc\\\ncde"' file.txt
# this does not, executes the `c` command from the second line
sed $'1i "abc\ncde"' file.txt

Related

bash / sed : editing of the file

I use sed to remove all lines starting from "HETATM" from the input file and cat to combine another file with the output recieved from SED
sed -i '/^HETATM/ d' file1.pdb
cat fil2.pdb file1.pdb > file3.pdb
is this way to do it in one line e.g. using only sed?
If you want to consider awk then it can be done in a single command:
awk 'FNR == NR {print; next} !/^HETATM/' file2.pdb file1.pdb > file3.pdb
With cat + grep combination please try following code. Simple explanation would be, using cat command's capability to concatenate file's output when multiple files are passed to it and using grep -v to remove all words starting from HETATM in file1.pdb before sending is as an input to cat command and creating new file named file3.pdb from cat command's output.
cat file2.pdb <(grep -v '^HETATM' file1.pdb) > file3.pdb
I'm not sure what you mean by "remove all lines starting from 'HETATM'", but if you mean that any line that appears in the file after a line that starts with "HETATM" will not be outputted, then your sed expression won't do it - it will just remove all lines starting with the pattern while leaving all following lines that do not start with the pattern.
There are ways to get the effect I believe you wanted, possibly even with sed - but I don't know sed all that well. In perl I'd use the range operator with a guaranteed non-matching end expression (not sure what will be guaranteed for your input, I used "XXX" in this example):
perl -ne 'unless (/^HETATM/../XXX/) { print; }' file1.pdb
mawk '(FNR == NR) < NF' FS='^HETATM' f1 f2

How to remove consecutive repeating characters from every line?

I have the below lines in a file
Acanthocephala;Palaeacanthocephala;Polymorphida;Polymorphidae;;Profilicollis;Profilicollis_altmani;
Acanthocephala;Eoacanthocephala;Neoechinorhynchida;Neoechinorhynchidae;;;;
Acanthocephala;;;;;;;
Acanthocephala;Palaeacanthocephala;Polymorphida;Polymorphidae;;Polymorphus;;
and I want to remove the repeating semi-colon characters from all lines to look like below (note- there are repeating semi-colons in the middle of some of the above lines too)
Acanthocephala;Palaeacanthocephala;Polymorphida;Polymorphidae;Profilicollis;Profilicollis_altmani;
Acanthocephala;Eoacanthocephala;Neoechinorhynchida;Neoechinorhynchidae;
Acanthocephala;
Acanthocephala;Palaeacanthocephala;Polymorphida;Polymorphidae;Polymorphus;
I would appreciate if someone could kindly share a bash one-liner to accomplish this.
You can use tr with "squeeze":
tr -s ';' < infile
perl -p -e 's/;+/;/g' myfile # writes output to stdout
or
perl -p -i -e 's/;+/;/g' myfile # does an in-place edit
If you want to edit the file itself:
printf "%s\n" 'g/;;/s/;\{2,\}/;/g' w | ed -s foo.txt
If you want to pipe a modified copy of the file to something else and leave the original unchanged:
sed 's/;\{2,\}/;/g' foo.txt | whatever
These replace runs of 2 or more semicolons with single ones.
could be solved easily by substitutions.
I add an awk solution by playing with the FS/OFS variable:
awk -F';+' -v OFS=';' '$1=$1' file
or
awk -F';+' -v OFS=';' '($1=$1)||1' file
Here's a sed version of alaniwi's answer:
sed 's/;\+/;/g' myfile # Write output to stdout
or
sed -i 's/;\+/;/g' myfile # Edit the file in-place

pass sed a long list of line numbers to remove from a file

I am trying to remove 500+ non-consecutive lines from a very large file with sed.
I have the lines stored in a list.txt file but I cant't use it in a for loop
for i in `cat list`; do echo 'sed -i -e ' \'"$i"d\'' huge_file.txt' ; done
because line numbers in the original file would change every time sed removes one and exits.
I should do:
sed -i -e '1d;2d;93572277d;93572278d; ......;nth ' huge_file.txt
Is there a way to pass that list to sed in a file?
you can try with awk:
awk -v s="2,3,..,n" 'BEGIN{n=split(s,t,",");for(i=1;i<=n;i++)d[t[i]]=1}
!d[NR]' huge.txt
You pass the comma-separated line numbers to awk by -v, in awk split it in array, and check each line, if the line number in the array, skip.
Test it with small file, if it worked as you expected, you can do:
awk -v '....' '....' huge.txt > tmp.txt && mv tmp.txt huge.txt
to write the change back to your original input file.
update
If you have 500 line numbers in another file, say, each number in a line, you can:
awk 'NR==FNR{a[$0]=1;next}!a[FNR]' ln.txt huge.txt
If it's just for a single particular task (not frequent) you may use the following GNU sed approach (assuming that numbers in list.txt are separated with newline \n):
sed -i "$(sed -z 's/\n/d;/g' list.txt)" huge_file.txt

remove lines based on file input pattern using sed

I have been trying to solve a simple sed line deletion problem.
Looked here and there. It didn't solve my problem.
My problem could simply be achieved by using sed -i'{/^1\|^2\|^3/d;}' infile.txt which deletes lines beginning with 1,2 and 3 from the infile.txt.
But what I want instead is to take the starting matching patterns from a file than manually feeding into the stream editor.
E.g: deletePattern
1
3
2
infile.txt
1 Line here
2 Line here
3 Line here
4 Line here
Desired output
4 Line here
Thank you in advance,
This grep should work:
grep -Fvf deletePattern infile.txt
4 Line here
But this will skip a line if patterns in deletePattern are found anywhere in the 2nd file.
More accurate results can be achieved by using this awk command:
awk 'FILENAME == ARGV[1] && FNR==NR{a[$1];next} !($1 in a)' deletePattern infile.txt
4 Line here
Putting together a quick command substitution combined with a character class will allow a relatively short oneliner:
$ sed -e "/^[$( while read -r ch; do a+=$ch; done <pattern.txt; echo "$a" )]/d" infile.txt
4 Line here
Of course, change the -e to -i for actual in-place substitution.
With GNU sed (for -f -):
sed 's!^[0-9][0-9]*$!/^&[^0-9]/d!' deletePattern | sed -f - infile.txt
The first sed transforms deletePattern into a sed script, then the second sed applies this script.
Try this:
sed '/^[123]/ d' infile.txt

sed extract substring between two characters from a file and save to variable

I am automatically building a package. The automated script needs to get the version of the package to build.
I need to get the string of the python script main.py. It says in line 15
VERSION="0.2.0.4" #DO NOT MOVE THIS LINE
I need the 0.2.0.4, in future it can easily become 0.10.3.15 or so, so the sed command must not have a fixed length.
I found this on stackoverflow:
sed -n '15s/.*\#\([0-9]*\)\/.*/\1/p'
"This suppresses everything but the second line, then echos the digits between # and /"
This does not work (adjusted). Which is the last "/"? How can I save the output into a variable called "version"?
version = sed -n ...
throws an error
command -n not found
If you just need version number.
awk -F\" 'NR==15 {print $2}' main.py
This prints everything between " on line 15. Like 0.2.0.4
With awk:
$ awk -F= 'NR==15 {gsub("\"","",$2); print $2}' main.py
0.2.0.4
Explanation
NR==15 performs actions on line number 15.
-F= defines the field separator as =.
{gsub("\"","",$2); print $2} removes the " character on the 2nd field and prints it.
Update
to be more specific the line is version="0.2.0.4" #DO NOT MOVE THIS
LINE
$ awk -F[=#] 'NR==15 {gsub("\"","",$2); print $2}' main.py
0.2.0.4
Using multiple field separator -F[=#] which means it can be either # or =.
To save it into your version variable, use the expression var=$(command) like:
version=$(awk -F[=#] 'NR==15 {gsub("\"","",$2); print $2}' main.py)
Try:
sed -n '15s/[^"]*"\(.*\)".*/\1/p' inputfile
In order to assign it to a variable, say:
VARIABLE=$(sed -n '15s/[^"]*"\(.*\)".*/\1/p' inputfile)
In order to remove the dependency that the VERSION would occur only on line 15, you could say:
sed -n '/^VERSION=/ s/[^"]*"\(.*\)".*/\1/p' inputfile
there should not be space in assigning variables
version=$(your code)
version=$(sed -r -i '15s/.*\"\([0-9]*\)\/.*/"/p' main.py)
OR
version=`sed -r -i '15s/.*\"\([0-9]*\)\/.*/"/p' main.py`

Resources