I have a big file that might have some comments in the header
# Comment line 1
# Comment line 2
# ...
# Comment line ...
# ...
# Comment line N
123
234
345
...
I need a one line solution to get the number "N+1", what's the most elegant way to achieve this using shell script? Thanks
try this sed one-liner:
sed -n '/^\s*#/!{p;q}' file
ok, if you need only the line-number:
sed -n '/^\s*#/!{=;q}' file
add short explanation:
/^\s*#/! : regex, if the line does NOT start with 0 or more empty chars (tab/space)then a '#', the line is chosen for further step.
= : print line no
q : quit processing
awk version:
awk '$1~/^[^#]/{print NR; exit}' file
Related
I have an TARGET.md file, I'm looking for a string and I want to replace it with the content of other md file, I have tried many combinations but it seems like the newline in the files are the ones sed is not liking, I just need to do this using pure bash(it doesn't have to be sed) because this is how the whole script is running:
This works:
local search="##### Header"
local replace="##### Header\\
\\
Line 1\\
Line 2\\
Line 3\\
Line 4"
sed -i '' -e "s/${search}/${replace}/" TARGET.md
But this won't:
file1.md content:
##### Header
Line 1
Line 2
Line 3
Line 4
Script:
local search="##### Header"
local replace=$(curl "path/to/file/in/other/place/file1.md")
sed -i '' -e "s/${search}/${replace}/" TARGET.md
NOTE: I don't have the file1.md in the same place, I'm doing a curl to get the raw content from it, this is why the replace is in a variable.
I'm assuming the concept is possible but my sed syntax is wrong knowing sed can handle newlines out of the box, but not sure what is the proper way to do this.
I've been searching for some days now, any help, tip or guide is appreciated!
You are using the wrong tool. sed is a line editor at heart. While you can repeatedly append to pattern space in some instances, awk with getline provides a more flexible solution. For example with your file1.md:
##### Header
Line 1
Line 2
Line 3
Line 4
and your TARGET.md as:
##### Unreleased
my dog
has fleas
The to replace "##### Unreleased" with the content of file1.md, you can do:
awk -v replace="file1.md" -v search="##### Unreleased" '
$0 == search {while (getline line < replace ) { print line }; next }
{ print }
' TARGET.md
Above you have your replace and search as with sed, but instead of using the line-editor, you use awk to locate the line containing search and the read all lines from replace using getline`. The second rule just prints all other lines as is.
Example Use/Output
In the directory containing each file, you can simply select-copy the above and middle-mouse paste into the terminal to test:
$ awk -v replace="file1.md" -v search="##### Unreleased" '
> $0 == search {while (getline line < replace ) { print line }; next }
> { print }
> ' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
Look things over and let me know if you have further questions.
Taking TARGET.md file from David's answer:
cat TARGET.md
##### Unreleased
my dog
has fleas
You can run sed with r command like this:
search="##### Unreleased"
sed -e "/$search/{r file1.md" -e ';d;}' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
UPDATED:
Using sed, how can I insert (NOT SUBSTITUTE) a new line on only the first match of keyword for each file.
Currently I have the following but this inserts for every line containing Matched Keyword and I want it to only insert the New Inserted Line for only the first match found in the file:
sed -ie '/Matched Keyword/ i\New Inserted Line' *.*
For example:
Myfile.txt:
Line 1
Line 2
Line 3
This line contains the Matched Keyword and other stuff
Line 4
This line contains the Matched Keyword and other stuff
Line 6
changed to:
Line 1
Line 2
Line 3
New Inserted Line
This line contains the Matched Keyword and other stuff
Line 4
This line contains the Matched Keyword and other stuff
Line 6
You can sort of do this in GNU sed:
sed '0,/Matched Keyword/s//New Inserted Line\n&/'
But it's not portable. Since portability is good, here it is in awk:
awk '/Matched Keyword/ && !x {print "Text line to insert"; x=1} 1' inputFile
Or, if you want to pass a variable to print:
awk -v "var=$var" '/Matched Keyword/ && !x {print var; x=1} 1' inputFile
These both insert the text line before the first occurrence of the keyword, on a line by itself, per your example.
Remember that with both sed and awk, the matched keyword is a regular expression, not just a keyword.
UPDATE:
Since this question is also tagged bash, here's a simple solution that is pure bash and doesn't required sed:
#!/bin/bash
n=0
while read line; do
if [[ "$line" =~ 'Matched Keyword' && $n = 0 ]]; then
echo "New Inserted Line"
n=1
fi
echo "$line"
done
As it stands, this as a pipe. You can easily wrap it in something that acts on files instead.
If you want one with sed*:
sed '0,/Matched Keyword/s//Matched Keyword\nNew Inserted Line/' myfile.txt
*only works with GNU sed
This might work for you:
sed -i -e '/Matched Keyword/{i\New Inserted Line' -e ':a;n;ba}' file
You're nearly there! Just create a loop to read from the Matched Keyword to the end of the file.
After inserting a line, the remainder of the file can be printed out by:
Introducing a loop place holder :a (here a is an arbitrary name).
Print the current line and fetch the next into the pattern space with the ncommand.
Redirect control back using the ba command which is essentially a goto to the a place holder. The end-of-file condition is naturally taken care of by the n command which terminates any further sed commands if it tries to read passed the end-of-file.
With a little help from bash, a true one liner can be achieved:
sed $'/Matched Keyword/{iNew Inserted Line\n:a;n;ba}' file
Alternative:
sed 'x;/./{x;b};x;/Matched Keyword/h;//iNew Inserted Line' file
This uses the Matched Keyword as a flag in the hold space and once it has been set any processing is curtailed by bailing out immediately.
If you want to append a line after first match only, use AWK instead of SED as below
awk '{print} /Matched Keyword/ && !n {print "New Inserted Line"; n++}' myfile.txt
Output:
Line 1
Line 2
Line 3
This line contains the Matched Keyword and other stuff
New Inserted Line
Line 4
This line contains the Matched Keyword and other stuff
Line 6
I have a file that holds output from a test.
test 1
42
test 2
69
test 3
420
test 4
55378008
I would like to make the test output appear on the same line as the test name. like so:
test 1: 42
test 2: 69
test 3: 420
test 4: 55378008
I am sure there is some fancy sed, awk or perl way to do this but I am stuck.
And here is another one in sed flavor to complete the offer :
sed 'N ; s/\n/: /' input_file
For each (odd) line starting from the first, append the next (even) one in pattern space separated by a LF, then just replace this LF by :.
awk 'FNR%2{printf "%s: ", $0; next}1' file
This prints odd lines with suffix : and without newline and even lines with a newline.
pr has this built-in, but if you need whitespace adjustment as well, then sed/awk/perl solutions suggested in other answers will suit you better
$ pr -2ats': ' ip.txt
test 1: 42
test 2: 69
test 3: 420
test 4: 55378008
This combines 2 lines at a time with : as the separator.
Just replace the line feed of odd lines with :␠.
perl -pe's/\n/: / if $. % 2'
You have mentioned that you want to removing leading and trailing whitespace as well. For that, you can use the following:
perl -pe's/^\h+|\h+$/g; s/\n/: / if $. % 2'
Specifying file to process to Perl one-liner
A shell solution, which is very slow on large set of data/files.
while IFS= read -r odd_line; do
IFS= read -r even_line
printf '%s: %s\n' "$odd_line" "$even_line"
done < file.txt
On the other hand if the colon is not a requirement paste can do the job.
paste - - < file.txt
Bash solution
skips empty lines
process both UNIX/DOS format 'end of line'
accepts filename as argument or otherwise reads data from STDIN
#!/bin/bash
while read p1
do
[[ -z $p1 ]] && continue
# p1=`echo -n $p1 | tr -d "\r"` # replaced with following line
p1=${p1//$'\r'/}
read p2
echo -n "$p1: $p2"
done < ${1:-/dev/stdin}
Output
test 1: 42
test 2: 69
test 3: 420
test 4: 55378008
NOTE: no empty lines allowed between lines for join
Eg. Input Text file
line 1 - Abchsffmskdv
line 2 - bsdvnld
line 3 - fsdgdhdh
If line 1, position 9 is s, then change line 3, position 4 to k
Output:
line 1 - Abchsffmskdv
line 2 - bsdvnld
line 3 - fsdkdhdh
Please try the following:
sed '
/^.\{8\}s/ { # if the 9th char == "s" then
$!n # flush current line and read next line (unless eof)
$!n # same as above to proceed to the line after next
$!s/^\(.\{3\}\)./\1k/ # replace the 4th char with "k"
}' input.txt
or as a one-liner:
sed '/^.\{8\}s/{$!n; $!n; $!s/^\(.\{3\}\)./\1k/}' input.txt
input.txt looks like:
Abchsffmskdv
bsdvnld
fsdgdhdh
iiiiiiiisiii
iiiiiii
iiiiiiii
output:
Abchsffmskdv
bsdvnld
fsdkdhdh
iiiiiiiisiii
iiiiiii
iiikiiii
You'll see the script above replaces the 4th character of the next next
line if the pattern is found regardless of the line number.
If you want to fix the line number to e.g. the 1st line, try instead:
sed '
1 {
/^.\{8\}s/ {
$!n
$!n
$!s/^\(.\{3\}\)./\1k/
}
}' input.txt
output:
Abchsffmskdv
bsdvnld
fsdkdhdh
iiiiiiiisiii
iiiiiii
iiiiiiii
which modifies the 3rd line only.
This might work for you (GNU sed):
sed -E '1h;3{G;s/(...).(.*)\n.{8}s/\1k\2/}' file
Store the first line in the hold space and append it to line 3 and pattern match for the desired criteria.
An alternative:
sed -E '1h;3{x;/^.{8}s/{x;s/./k/4;x};x}' file
i have a text file that has text.txt
aaa/bbb/ccc/ddd/eee
119
fff/ggg/hhh/iii/jjj
20
now how do i convert this output into 2 columns and store this in another text file
file count
aaa/bbb/ccc/ddd/eee 119
fff/ggg/hhh/iii/jjj 20
i want to do this using shell script
This should work
sed 'N;s/\n/ /' fileName
The above N command is an example sed's multiline capability. N commands takes first_line and second_line and separates them by \n. The pattern is then applied to
first_line\nsecond_line.
In the above example N command is followed by replace of \n with space. As a result the output becomes
first_line second_line
In pure bash:
( echo "file,count"
while read line #Read line by line
do
echo -n "$line," #Print one line, with a comma and without a newline
read line #Get the next line
echo "$line" #Print that line as second column
done < "inputFilename" ) > "outputFilename" #Redirect to output file
should do the trick (assuming you want actual comma separated values).