I need to do while loop when sed is not null. Example:
File 1
line1
line2
line3
File 2
i=1
while sed "${int}p" # here I need expression which checks if line is not null
# here echo this line and i++
I tried to write just while sed -n "${int}p" but it does not work as I expected.
You can use the = command in sed to get the number of lines:
sed -n '/./!q;=;p' input | sed 'N;s/\n/ /'
For an input:
a
b
c
d
This gives:
1 a
2 b
3 c
If you only want to get line number of the line before the first non-empty line:
sed -n '/./!q;=' input | tail -1
A while loop that prints all lines:
while read line; do
echo "$line"
done < input
If you want to count the lines until the first empty line, you could do this.
$ cat in.txt
line1
line2
line3
line4
line5
$ echo $(($(sed '/^\s*$/q' < in.txt | wc -l) - 1))
3
Related
I want to use nested process subtitution with tee in a while loop.
while read line; do
#process line
echo "--$line"
done < <(cat sample.file | tee >(grep "SPECLINE") | grep "LINESTOPROCESS")
Therefore, I need:
all lines in sample.file that contain "LINETOPROCESS" expression should be passed into the loop, and they will be printed with "--" prefix.
all lines contain "SPECLINE" needs to be printed in tee's first process substitution (in the grep).
I want to avoid cat-ting the sample.file more than once as it is too large and heavy.
With a simple sample.test file:
line1 SPECLINE
line2 LINETOPROCESS
line3 LINETOPROCESS
line4 SPECLINE
line5 I don't need it
line6 also not
line7 also not
line8 SPECLINE
line9 LINETOPROCESS
My result:
# ./test.sh
#
My desired result:
# ./test.sh
line1 SPECLINE
--line2 LINETOPROCESS
--line3 LINETOPROCESS
line4 SPECLINE
line8 SPECLINE
--line9 LINETOPROCESS
Or I can also accept this as output:
# ./test.sh
--line2 LINETOPROCESS
--line3 LINETOPROCESS
--line9 LINETOPROCESS
line1 SPECLINE
line4 SPECLINE
line8 SPECLINE
UPDATE1
greps are for demo only.
I really need those 2 substitutions.
sample.file is a http file.
grep "SPECLINE" would be "hxselect -i -s ';' -c 'div.hour'
grep "LINESTOPROCESS" would be "hxselect -i -s ';' -c 'div.otherclass' | hxpipe
hx programs are not line-oriented. They are reading from stdin and outputting to stdout.
Therefore the tee's first command will select divs with 'hour' class and separate them with ';'. Afterwards, the pipe after tee will select all divs with class 'otherclass' and hxpipe will flatten it for the loop for further processing.
I would use no process substitution at all.
while IFS= read -r line; do
if [[ $line = *SPECLINE* ]]; then
printf '%s\n' "$line"
elif [[ $line = *LINETOPROCESS* ]]; then
printf '--%s\n' "$line"
fi
done < sample.txt
You are already paying the cost of reading an input stream line-by-line in bash; no reason to add the overhead of two separate grep processes to it.
A single awk process would be even better, as it is more efficient than bash's read-one-character-at-a-time approach to reading lines of text.
awk '/SPECLINE/ {print} /LINETOPROCESS/ {print "--"$0}' sample.txt
(which is too simple if a single line could match both SPECLINE and LINETOPROCESS, but I leave that as an exercise to the reader to fix.)
The following just loops through the entire file and just prints the matching lines. All other lines are ignored.
while read line; do
case "$line" in
*SPECLINE*) echo "$line" ;;
*LINETOPROCESS*) echo "--$line" ;;
esac
done < sample.file
When you want the tee, you can make 2 changes.
Your testcode greps LINESTOPROCESS, the input is LINETO..
The output process substition gives problems like https://stackoverflow.com/a/42766913/3220113 explained. You can do this differently.
while IFS= read -r line; do
#process line
echo "--$line"
done < x2 |
tee >(grep "SPECLINE") >(grep "LINETOPROCESS") >/dev/null
I don't know hxselect, but it seems to operate on a complete well-formed XML document, so avoid the grep.
Please help me:)
How to move string from 4-line to under 1-line in bash (awk, sed)?
Example, i have a file:
line1
line2
line3
moveline4
result use SED, AWK and other utils:
line1
moveline4
line2
line3
Another approach with sed:
sed '2{N;h;d};4G;' file
Explanations:
2N: merges second and third line
h: stores both lines into the hold space
d: deletes both lines
4G: adds the pattern space content after fourth line
input file:
line1
line2
line3
moveline4
line5
line6
sed solution:
sed -n -e '1p' -e '4p' lines.txt; sed -n -e '2,3p' -e '5,$p' lines.txt
output:
line1
moveline4
line2
line3
line5
line6
You can redirect your ouput to a specific file (> and >>)
sed -n -e '1p' -e '4p' lines.txt > new_file.txt; sed -n -e '2,3p' -e '5,$p' lines.txt >> new_file.txt
explanation:
-n to avoid default printing of lines
-e to pass several commands to sed
1p to print the first line
4p to print the 4th line
2,3p to print the 2nd and 3rd line
5,$p to print the 5th line to the last line
awk solution:
awk 'NR==1||NR==4' lines.txt; awk 'NR>=2 && NR!=4' lines.txt
You can redirect your ouput to a specific file (> and >>)
awk 'NR==1||NR==4' lines.txt > new_file.txt; awk 'NR>=2 && NR!=4' lines.txt >> new_file.txt
explanation:
NR==1||NR==4 will make awk do its default action for number record 1 or 4 which is printing (lines 1 and 4)
NR>=2 && NR!=4 for lines >=2 and !=4 you print them
head/tail solution:
head -1 lines.txt; head -4 lines.txt| tail -1; head -3 lines.txt | tail -2; tail -n +5 lines.txt
for information about the behavior of head and tail please do man head and man tail
perl solution:
perl -ne 'print if 1..1' lines.txt; perl -ne 'print if 4..4' lines.txt; perl -ne 'print if 2..3' lines.txt; perl -ne 'print if 5..6' lines.txt
full bash solution:
#!/usr/bin/env bash
#print the 1st line
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
break
done <"$1"
#print the 4th line
x=1
while IFS='' read -r line || [[ -n "$line" ]]; do
if [ $x -eq 4 ]; then
echo "$line"
break
fi
x=$((x+1))
done <"$1"
#print all lines except the 1st and the 4th
x=1
while IFS='' read -r line || [[ -n "$line" ]]; do
if [ $x -ne 1 -a $x -ne 4 ]; then
echo "$line"
fi
x=$((x+1))
done <"$1"
you can call the script ./myscript.sh lines.txt and redirect the output if necessary: ./myscript.sh lines.txt > new_file.txt
I have a file that looks like this:
===SEPARATOR===
line2
line3
===SEPARATOR===
line5
line6
===SEPARATOR===
line8
...
lineX
===SEPARATOR===
How can I do a while loop and go through the file, dump anything between two ===SEPARATOR=== occurrences into another file for further processing?
I want to add only line2, line3 to the second file on the first iteration. I will parse the file; and on the next iteration I want line5 line6 in second file to do the same parsing again but on different data.
It sounds like you want to save each block of lines to a separate file.
The following solutions create output files f1, f2, containing the (non-empty) blocks of lines betwen the ===SEPARATOR=== lines.
With GNU Awk or Mawk:
awk -v fnamePrefix='f' -v RS='(^|\n)===SEPARATOR===(\n|$)' \
'NF { fname = fnamePrefix (++n); print > fname; close(fname) }' file
Pure bash - which will be slow:
#!/usr/bin/env bash
fnamePrefix='f'; i=0
while IFS= read -r line; do
[[ $line == '===SEPARATOR===' ]] && { (( ++i )); > "${fnamePrefix}${i}"; continue; }
printf '%s\n' "$line" >> "${fnamePrefix}${i}"
done < file
You can exclude all lines matching ===SEPARATOR=== with grep -v and redirect the rest to a file:
grep -vx '===SEPARATOR===' file > file_processed
-x makes sure that only lines completely matching ===SEPARATOR=== are excluded.
This uses sed to find lines between separators, and then grep -v to delete the separators.
$ sed -n '/===SEPARATOR===/,/===SEPARATOR===/ p' file | grep -v '===SEPARATOR==='
line2
line3
line8
...
lineX
There's got to be a more elegant answer that doesn't repeat the separator three times, but I'm drawing a blank.
I am assuming that you do not need the line5 and line6 . You can do it with awk like this:.
awk '$0 == "===SEPARATOR===" {interested = ! interested; next} interested {print}'
Credit goes to https://www.gnu.org/software/gawk/manual/html_node/Boolean-Ops.html#Boolean-Ops
Output:
[root#hostname ~]# cat /tmp/1 | awk '$0 == "===SEPARATOR===" {interested = ! interested; next} interested {print}' /tmp/1
line2
line3
line8
...
lineX
awk to the rescue!
with multi-char support (e.g. gawk)
$ awk -v RS='\n?===SEPARATOR===\n' '!(NR%2)' file
line2
line3
line8
...
lineX
or without that
$ awk '/===SEPARATOR===/{p=!p;next} p' file
line2
line3
line8
...
lineX
which is practically the same with #Jay Rajput's answer.
I have a file A with 400,000 lines. I have another file B that has a bunch of line numbers.
File B:
-------
98
101
25012
10098
23489
I have to extract those line numbers specified in file B from file A. That is I want to extract lines 98,101,25012,10098,23489 from file A. How to extract these lines in the following cases.
File B is a explicit file.
File B is arriving out of a pipe. For eg., grep -n pattern somefile.txt is giving me the file B.
I wanted to use see -n 'x'p fileA. However, I don't know how to give the 'x' from a file. Also, I don't to how to pipe the value of 'x' from a command.
sed can print the line numbers you want:
$ printf $'foo\nbar\nbaz\n' | sed -ne '2p'
bar
If you want multiple lines:
$ printf $'foo\nbar\nbaz\n' | sed -ne '2p;3p'
bar
baz
To transform a set of lines to a sed command like this, use sed for beautiful sedception:
$ printf $'98\n101' | sed -e 's/$/;/'
98;
101;
Putting it all together:
sed -ne "$(sed -e 's/$/p;/' B)" A
Testing:
$ cat A
1
22
333
4444
$ cat B
1
3
$ sed -ne "$(sed -e 's/$/p;/' B)" A
1
333
QED.
awk fits this task better:
fileA in file case:
awk 'NR==FNR{a[$0]=1;next}a[FNR]' fileB fileA
fileA content from pipe:
cat fileA|awk 'NR==FNR{a[$0]=1;next}a[FNR]' fileB -
oh, you want FileB in file or from pipe, then same awk cmd:
awk '...' fileB fileA
and
cat fileB|awk '...' - fileA
need advice about the following
with the following ksh script I actually copy file1 to file2
my problem is that lines in file2 are not with the same location as file1
#!/bin/ksh
while read -r line ; do
echo $line >> file2
done < file1
for example
more file1
line1
line2
line3
more file2
line1
line2
line3
the question what I need to change in my script in order to get lines location as described in file1? after I run my ksh script?
lidia
You can try:
while read -r line ; do
echo $line | sed -re 's/^\s+//' >> file2
done < file1
This uses sed to get rid of the leading whitespaces present in lines from file1.
you can set IFS=
while IFS= read -r line ; do echo "$line"; done<file