I'm new to bashes but learning quickly. One thing that stumps me is how to store an awk output as a variable if it contains multiple backslashs. I have awk code to tell me everything between the two terms "1\1\" and "\#" (quotation marks not included). I put an extra backslash behind each backslash. The echo command works fine:
#!/bin/
FILENAME=$1
$echo awk '/1\\1\\/,/\\\\#/' $FILENAME
I try to store it as a variable via the regular way:
VAR=$($echo awk '/1\\1\\/,/\\\\#/' $FILENAME)
echo $VAR
I get a totally different section of the text file I'm analyzing. I figure that the parentheses alter the meaning of a backslash, but I can't seem to find a way to store the string I am getting as a variable without using parentheses. Any suggestions? Thanks!
Edit: A major portion of the input file:
Atom 3 needs variable 1= 1.4272969666 but is 0.9695781041
Input z-matrix variables are not compatible with final structure.
1\1\CRUNTCH-C1950-1-9\FOpt\RB3LYP\CC-pVTZ\H2O3\DK0178\21-Nov-2013\0\\#
p B3LYP/CC-PVTZ opt\\Peroxide Molecule\\0,1\H,-0.2455786307,0.24093160
48,0.0173467521\O,-0.1623823861,-0.3287313,0.7975033131\O,1.2337882601
,-0.2361247615,1.0791204051\O,1.4541296394,0.9715988133,1.8071520489\H
,1.2170466675,0.7056779744,2.7089056027\\Version=EM64L-G09RevB.01\Stat
e=1-A\HF=-226.7854316\RMSD=3.950e-09\RMSF=2.016e-05\Dipole=-0.2755823,
0.261353,0.1046274\Quadrupole=-1.3992237,-2.043705,3.4429287,-1.030514
,1.5582408,-0.1409701\PG=C02 [C2(O1),X(H2O2)]\\#
ALL SCIENCE IS EITHER PHYSICS, OR STAMP COLLECTING.
The part I get the first time:
1\1\CRUNTCH-C1950-1-9\FOpt\RB3LYP\CC-pVTZ\H2O3\DK0178\21-Nov-2013\0\\#
p B3LYP/CC-PVTZ opt\\Peroxide Molecule\\0,1\H,-0.2455786307,0.24093160
48,0.0173467521\O,-0.1623823861,-0.3287313,0.7975033131\O,1.2337882601
,-0.2361247615,1.0791204051\O,1.4541296394,0.9715988133,1.8071520489\H
,1.2170466675,0.7056779744,2.7089056027\\Version=EM64L-G09RevB.01\Stat
e=1-A\HF=-226.7854316\RMSD=3.950e-09\RMSF=2.016e-05\Dipole=-0.2755823,
0.261353,0.1046274\Quadrupole=-1.3992237,-2.043705,3.4429287,-1.030514
,1.5582408,-0.1409701\PG=C02 [C2(O1),X(H2O2)]\\#
The part I get the second time:
,1.5582408,-0.1409701\PG=C02 [C2(O1),X(H2O2)]\#05,3.4429287,-1.030514
Thanks!
Edit 2 (Sorry this is getting rather long) The code and error message I got for answer 2:
Input code:
#!/bin/
file=$1
$ cat file
abc
1\1\
def
\#
ghi
$ awk '/1\\1\\/{f=1} f; /\\#/{f=0}' file
1\1\
def
\#
$ var=$(awk '/1\\1\\/{f=1} f; /\\#/{f=0}' file)
$ printf "%s\n" "$var"
1\1\
def
\#
Error:
[dave#server ~]$ bash sshell.sh peroxide.out
sshell.sh: line 5: $: command not found
sshell.sh: line 6: abc: command not found
sshell.sh: line 8: 11def: command not found
sshell.sh: line 9: #: command not found
sshell.sh: line 10: ghi: command not found
sshell.sh: line 12: $: command not found
sshell.sh: line 14: 11def: command not found
sshell.sh: line 15: #: command not found
awk: cmd. line:1: fatal: cannot open file `file' for reading (No such file or directory)
sshell.sh: line 17: $: command not found
sshell.sh: line 18: $: command not found
sshell.sh: line 20: 11def: command not found
sshell.sh: line 21: #: command not found
Let's start with this:
$ cat file
abc
1\1\
def
\#
ghi
$ awk '/1\\1\\/{f=1} f; /\\#/{f=0}' file
1\1\
def
\#
$ var=$(awk '/1\\1\\/{f=1} f; /\\#/{f=0}' file)
$ printf "%s\n" "$var"
1\1\
def
\#
Now - if that's not exactly what you want, update your question to include some sample input and expected output.
Don't ever use awk '/start/,/end/' as it makes trivial things like this slightly briefer, but requires a complete re-write for anything even marginally more interesting.
The following works for me:
#! /bin/bash
var=$(awk 'BEGIN { RS="" } {
match($0,/1\\1\\(.*)\\#/,a)
print a[1] }' input.txt)
echo "$var"
where input.txt is your input file..
Related
I have an TARGET.md file, I'm looking for a string and I want to replace it with the content of other md file, I have tried many combinations but it seems like the newline in the files are the ones sed is not liking, I just need to do this using pure bash(it doesn't have to be sed) because this is how the whole script is running:
This works:
local search="##### Header"
local replace="##### Header\\
\\
Line 1\\
Line 2\\
Line 3\\
Line 4"
sed -i '' -e "s/${search}/${replace}/" TARGET.md
But this won't:
file1.md content:
##### Header
Line 1
Line 2
Line 3
Line 4
Script:
local search="##### Header"
local replace=$(curl "path/to/file/in/other/place/file1.md")
sed -i '' -e "s/${search}/${replace}/" TARGET.md
NOTE: I don't have the file1.md in the same place, I'm doing a curl to get the raw content from it, this is why the replace is in a variable.
I'm assuming the concept is possible but my sed syntax is wrong knowing sed can handle newlines out of the box, but not sure what is the proper way to do this.
I've been searching for some days now, any help, tip or guide is appreciated!
You are using the wrong tool. sed is a line editor at heart. While you can repeatedly append to pattern space in some instances, awk with getline provides a more flexible solution. For example with your file1.md:
##### Header
Line 1
Line 2
Line 3
Line 4
and your TARGET.md as:
##### Unreleased
my dog
has fleas
The to replace "##### Unreleased" with the content of file1.md, you can do:
awk -v replace="file1.md" -v search="##### Unreleased" '
$0 == search {while (getline line < replace ) { print line }; next }
{ print }
' TARGET.md
Above you have your replace and search as with sed, but instead of using the line-editor, you use awk to locate the line containing search and the read all lines from replace using getline`. The second rule just prints all other lines as is.
Example Use/Output
In the directory containing each file, you can simply select-copy the above and middle-mouse paste into the terminal to test:
$ awk -v replace="file1.md" -v search="##### Unreleased" '
> $0 == search {while (getline line < replace ) { print line }; next }
> { print }
> ' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
Look things over and let me know if you have further questions.
Taking TARGET.md file from David's answer:
cat TARGET.md
##### Unreleased
my dog
has fleas
You can run sed with r command like this:
search="##### Unreleased"
sed -e "/$search/{r file1.md" -e ';d;}' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
I have a file that holds output from a test.
test 1
42
test 2
69
test 3
420
test 4
55378008
I would like to make the test output appear on the same line as the test name. like so:
test 1: 42
test 2: 69
test 3: 420
test 4: 55378008
I am sure there is some fancy sed, awk or perl way to do this but I am stuck.
And here is another one in sed flavor to complete the offer :
sed 'N ; s/\n/: /' input_file
For each (odd) line starting from the first, append the next (even) one in pattern space separated by a LF, then just replace this LF by :.
awk 'FNR%2{printf "%s: ", $0; next}1' file
This prints odd lines with suffix : and without newline and even lines with a newline.
pr has this built-in, but if you need whitespace adjustment as well, then sed/awk/perl solutions suggested in other answers will suit you better
$ pr -2ats': ' ip.txt
test 1: 42
test 2: 69
test 3: 420
test 4: 55378008
This combines 2 lines at a time with : as the separator.
Just replace the line feed of odd lines with :␠.
perl -pe's/\n/: / if $. % 2'
You have mentioned that you want to removing leading and trailing whitespace as well. For that, you can use the following:
perl -pe's/^\h+|\h+$/g; s/\n/: / if $. % 2'
Specifying file to process to Perl one-liner
A shell solution, which is very slow on large set of data/files.
while IFS= read -r odd_line; do
IFS= read -r even_line
printf '%s: %s\n' "$odd_line" "$even_line"
done < file.txt
On the other hand if the colon is not a requirement paste can do the job.
paste - - < file.txt
Bash solution
skips empty lines
process both UNIX/DOS format 'end of line'
accepts filename as argument or otherwise reads data from STDIN
#!/bin/bash
while read p1
do
[[ -z $p1 ]] && continue
# p1=`echo -n $p1 | tr -d "\r"` # replaced with following line
p1=${p1//$'\r'/}
read p2
echo -n "$p1: $p2"
done < ${1:-/dev/stdin}
Output
test 1: 42
test 2: 69
test 3: 420
test 4: 55378008
NOTE: no empty lines allowed between lines for join
I have two files and would like to insert the contents of one file into the other, replacing a specified line.
File 1:
abc
def
ghi
jkl
File 2:
123
The following code is what I have.
file1=numbers.txt
file2=letters.txt
linenumber=3s
echo $file1
echo $file2
sed "$linenumber/.*/r $file1/" $file2
Which results in the output:
abc
def
r numbers.txt
jkl
The output I am hoping for is:
abc
def
123
jkl
I thought it could be an issue with bash variables but I still get the same output when I manually enter the information.
How am I misunderstanding sed and/or the read command?
Your script replaces the line with the string "r $file1". The part in sed in s command is not re-interpreted as a command, but taken literally.
You can:
linenumber=3
sed "$linenumber"' {
r '"$file1"'
d
}' "$file2"
Read line number 3, print file1 and then delete the line.
See here for a good explanation and reference.
Surely we can make that a oneliner:
sed -e "$linenumber"' { r '"$file2"$'\n''d; }' "$file1"
Life example at tutorialpoints.
I would use the c command as follows:
linenumber=3
sed "${linenumber}c $(< $file1)" "$file2"
This replaces the current line with the text that comes after c.
Your command didn't work because it expands to this:
sed "3s/.*/r numbers.txt/" letters.txt
and you can't use r like that. r has to be the command that is being run.
I'm trying to use an array to define the lines to replace using sed; I can delete the lines using a variable for the line number but I can't get sed to use the variable to define the line number to write to. The problem seems to reside in the insert line. How do you pass the value of an array as a line number to sed?
#!/bin/bash
lineNum=$(sed -n '/max_allowed_packet/=' /etc/mysql/my.cnf)
IFS= #There's a space as the delimiter#
ary=($lineNum)
#for key in "${!ary[#]}";
# do
# sed -i '$ary[$key]'d /etc/mysql/my.cnf;
# #The folllowing line errors#
# sed -i "'$ary[$key]'imax_allowed_packet = 32M" /etc/mysql/my.cnf;
# #The above line errors#
#done
#for val in "${ary[#]}";
# do
# sed -i "${val}d" /etc/mysql/my.cnf;
# sed -i "${val}imax_allowed_packet = 32M" /etc/mysql/my.cnf;
# done
for val in "${ary[#]}";
do
sed -i "${val}s/.*/imax_allowed_packet = 32M" /etc/mysql/my.cnf";
done
For the first stanza of script I get the following output:
Error: sed: -e expression #1, char 1: unknown command: `''
For the second Stanza I get the following output:
sed: -e expression #1, char 3: unknown command:
'
sed: -e expression #1, char 3: unknown command:
'
For the third Stanza I get the following output:
./test.sh: line 22: unexpected EOF while looking for matching `"'
./test.sh: line 24: syntax error: unexpected end of file
Edit, rewriting the sed commands as sed -i "${ary[$key]}" generates the following error output: sed: -e expression #1, char 3: unknown command: `
I think you're over-complicating the issue. Your script can be reduced to this:
sed 's/\(max_allowed_packet\).*/\1 = 32M/' /etc/mysql.cnf
This performs a substitution on every occurrence of max_allowed_packet, setting the rest of the line to = 32M. Add the -i switch to overwrite the file when you're happy with the result.
Problems with your attempt
Shell parameters are not expanded within single quotes, so you would need to use double quotes, e.g. sed -i "${ary[$key]}d". You can use set -x to see what is happening here - at the moment, you will see the literal string $ary[$key], rather than the array value.
If I understand your intention correctly (you want to substitute the entire line), there's no need to call sed twice:
for val in "${ary[#]}"; do
sed -i.bak "${val}s/.*/imax_allowed_packet = 32M" /etc/mysql/my.cnf
done
I have chosen to loop through the values of the array, instead of the keys, in order to simplify things a little. When using the -i option, it is always a good idea to specify a backup file, as I have done.
I have a big file that might have some comments in the header
# Comment line 1
# Comment line 2
# ...
# Comment line ...
# ...
# Comment line N
123
234
345
...
I need a one line solution to get the number "N+1", what's the most elegant way to achieve this using shell script? Thanks
try this sed one-liner:
sed -n '/^\s*#/!{p;q}' file
ok, if you need only the line-number:
sed -n '/^\s*#/!{=;q}' file
add short explanation:
/^\s*#/! : regex, if the line does NOT start with 0 or more empty chars (tab/space)then a '#', the line is chosen for further step.
= : print line no
q : quit processing
awk version:
awk '$1~/^[^#]/{print NR; exit}' file