how to move string from position to position in bash (sed, awk) - bash

Please help me:)
How to move string from 4-line to under 1-line in bash (awk, sed)?
Example, i have a file:
line1
line2
line3
moveline4
result use SED, AWK and other utils:
line1
moveline4
line2
line3

Another approach with sed:
sed '2{N;h;d};4G;' file
Explanations:
2N: merges second and third line
h: stores both lines into the hold space
d: deletes both lines
4G: adds the pattern space content after fourth line

input file:
line1
line2
line3
moveline4
line5
line6
sed solution:
sed -n -e '1p' -e '4p' lines.txt; sed -n -e '2,3p' -e '5,$p' lines.txt
output:
line1
moveline4
line2
line3
line5
line6
You can redirect your ouput to a specific file (> and >>)
sed -n -e '1p' -e '4p' lines.txt > new_file.txt; sed -n -e '2,3p' -e '5,$p' lines.txt >> new_file.txt
explanation:
-n to avoid default printing of lines
-e to pass several commands to sed
1p to print the first line
4p to print the 4th line
2,3p to print the 2nd and 3rd line
5,$p to print the 5th line to the last line
awk solution:
awk 'NR==1||NR==4' lines.txt; awk 'NR>=2 && NR!=4' lines.txt
You can redirect your ouput to a specific file (> and >>)
awk 'NR==1||NR==4' lines.txt > new_file.txt; awk 'NR>=2 && NR!=4' lines.txt >> new_file.txt
explanation:
NR==1||NR==4 will make awk do its default action for number record 1 or 4 which is printing (lines 1 and 4)
NR>=2 && NR!=4 for lines >=2 and !=4 you print them
head/tail solution:
head -1 lines.txt; head -4 lines.txt| tail -1; head -3 lines.txt | tail -2; tail -n +5 lines.txt
for information about the behavior of head and tail please do man head and man tail
perl solution:
perl -ne 'print if 1..1' lines.txt; perl -ne 'print if 4..4' lines.txt; perl -ne 'print if 2..3' lines.txt; perl -ne 'print if 5..6' lines.txt
full bash solution:
#!/usr/bin/env bash
#print the 1st line
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
break
done <"$1"
#print the 4th line
x=1
while IFS='' read -r line || [[ -n "$line" ]]; do
if [ $x -eq 4 ]; then
echo "$line"
break
fi
x=$((x+1))
done <"$1"
#print all lines except the 1st and the 4th
x=1
while IFS='' read -r line || [[ -n "$line" ]]; do
if [ $x -ne 1 -a $x -ne 4 ]; then
echo "$line"
fi
x=$((x+1))
done <"$1"
you can call the script ./myscript.sh lines.txt and redirect the output if necessary: ./myscript.sh lines.txt > new_file.txt

Related

sed command within a while loop doesn't write output

I have this input file
gb|KY798440.1|
gb|KY842329.1|
MG082893.1
MG173246.1
and I want to get all the characters that are between the "|" or the full line if there is no "|". That is a desired output that looks like
KY798440.1
KY842329.1
MG082893.1
MG173246.1
I wrote:
while IFS= read -r line; do
if [[ $line == *\|* ]] ; then
sed 's/.*\|\(.*\)\|.*/\1/' <<< $line >> output_file
else echo $line >> output_file
fi
done < input_file
Which gives me
empty line
empty line
MG082893.1
MG173246.1
(note: empty line means an actual empty line - it doesn't actually writes "empty line")
The sed command works on a single example (i.e. sed 's/.*\|\(.*\)\|.*/\1/' <<< "gb|KY842329.1|" outputs KY842329.1) but within the loop it just does a line return. The else echo $line >> output_file seems to work.
Bare sed:
$ sed 's/^[^|]*|\||[^|]*$//g' file
Output:
KY798440.1
KY842329.1
MG082893.1
MG173246.1
You could do
sed '/|/s/[^|]*|\([^|]*\)|.*/\1/' input
or
awk 'NF>1 {print $2} NF < 2 { print $1}' FS=\| input
or
sed -e 's/[^|]*|//' -e 's/|.*//' input

Bash Process Substitution usage with tee and while loop

I want to use nested process subtitution with tee in a while loop.
while read line; do
#process line
echo "--$line"
done < <(cat sample.file | tee >(grep "SPECLINE") | grep "LINESTOPROCESS")
Therefore, I need:
all lines in sample.file that contain "LINETOPROCESS" expression should be passed into the loop, and they will be printed with "--" prefix.
all lines contain "SPECLINE" needs to be printed in tee's first process substitution (in the grep).
I want to avoid cat-ting the sample.file more than once as it is too large and heavy.
With a simple sample.test file:
line1 SPECLINE
line2 LINETOPROCESS
line3 LINETOPROCESS
line4 SPECLINE
line5 I don't need it
line6 also not
line7 also not
line8 SPECLINE
line9 LINETOPROCESS
My result:
# ./test.sh
#
My desired result:
# ./test.sh
line1 SPECLINE
--line2 LINETOPROCESS
--line3 LINETOPROCESS
line4 SPECLINE
line8 SPECLINE
--line9 LINETOPROCESS
Or I can also accept this as output:
# ./test.sh
--line2 LINETOPROCESS
--line3 LINETOPROCESS
--line9 LINETOPROCESS
line1 SPECLINE
line4 SPECLINE
line8 SPECLINE
UPDATE1
greps are for demo only.
I really need those 2 substitutions.
sample.file is a http file.
grep "SPECLINE" would be "hxselect -i -s ';' -c 'div.hour'
grep "LINESTOPROCESS" would be "hxselect -i -s ';' -c 'div.otherclass' | hxpipe
hx programs are not line-oriented. They are reading from stdin and outputting to stdout.
Therefore the tee's first command will select divs with 'hour' class and separate them with ';'. Afterwards, the pipe after tee will select all divs with class 'otherclass' and hxpipe will flatten it for the loop for further processing.
I would use no process substitution at all.
while IFS= read -r line; do
if [[ $line = *SPECLINE* ]]; then
printf '%s\n' "$line"
elif [[ $line = *LINETOPROCESS* ]]; then
printf '--%s\n' "$line"
fi
done < sample.txt
You are already paying the cost of reading an input stream line-by-line in bash; no reason to add the overhead of two separate grep processes to it.
A single awk process would be even better, as it is more efficient than bash's read-one-character-at-a-time approach to reading lines of text.
awk '/SPECLINE/ {print} /LINETOPROCESS/ {print "--"$0}' sample.txt
(which is too simple if a single line could match both SPECLINE and LINETOPROCESS, but I leave that as an exercise to the reader to fix.)
The following just loops through the entire file and just prints the matching lines. All other lines are ignored.
while read line; do
case "$line" in
*SPECLINE*) echo "$line" ;;
*LINETOPROCESS*) echo "--$line" ;;
esac
done < sample.file
When you want the tee, you can make 2 changes.
Your testcode greps LINESTOPROCESS, the input is LINETO..
The output process substition gives problems like https://stackoverflow.com/a/42766913/3220113 explained. You can do this differently.
while IFS= read -r line; do
#process line
echo "--$line"
done < x2 |
tee >(grep "SPECLINE") >(grep "LINETOPROCESS") >/dev/null
I don't know hxselect, but it seems to operate on a complete well-formed XML document, so avoid the grep.

print line without the first word into a variable

This is my code
title=""
line=""
fname=$1
numoflines=$(wc -l < $fname)
for ((i=2 ; i<=$numoflines ; i++))
do
...
done
In the for loop i want to print the first word of every line into $title
and the rest of the line without the first word into $line
(using bash)
tnx
I am assuming that by print to a variable you mean add the contents of each line to the variable. To do this, you can use the bash built-in function read:
while read -r t l; do title+="$t"; line+="$l"; done < "$fname"
This will add the first word of every line to $title and the rest of the line to $line.
You can do some like this:
echo "$fname"
This is my line.
My cat is green.
title=$(awk '{print $1}' <<< "$fname")
line=$(awk '{$1="";sub(/^ /,"")}1' <<< "$fname")
echo "$title"
This
My
echo "$line"
is my line.
cat is green.
Alternative approach using the cut command:
file="./myfile.txt"
title=$(cut -f1 -d ' ' "$file")
line=$(cut -f2- -d ' ' "$file")
#check print
pr -tm <(echo -e "TITLES\n$title") <(echo -e "LINES\n$line")
for the next myfile.txt
My cat is green.
Green cats are strange.
prints
TITLES LINES
My cat is green.
Green cats are strange.
do
Tempo="$( sed -n "${i} {s/^[[:blank:]]*\([^[:blank:]]*\)[[:blank:]]*\(.*\)/title='\1';line='\2'/p;q;}" ${fname} )"
eval "${Tempo}"
done
# or
do
sed -n "${i} {p;q;}" | read Line Title
# but this does not keep content available on each OS/shell
done

Bash while sed is not null

I need to do while loop when sed is not null. Example:
File 1
line1
line2
line3
File 2
i=1
while sed "${int}p" # here I need expression which checks if line is not null
# here echo this line and i++
I tried to write just while sed -n "${int}p" but it does not work as I expected.
You can use the = command in sed to get the number of lines:
sed -n '/./!q;=;p' input | sed 'N;s/\n/ /'
For an input:
a
b
c
d
This gives:
1 a
2 b
3 c
If you only want to get line number of the line before the first non-empty line:
sed -n '/./!q;=' input | tail -1
A while loop that prints all lines:
while read line; do
echo "$line"
done < input
If you want to count the lines until the first empty line, you could do this.
$ cat in.txt
line1
line2
line3
line4
line5
$ echo $(($(sed '/^\s*$/q' < in.txt | wc -l) - 1))
3

bash: only process line if not in second file

I have this block of code:
while IFS=$'\n' read -r line || [[ -n "$line" ]]; do
if [ "$line" != "" ]; then
echo -e "$lanIP\t$line" >> /tmp/ipList;
fi
done < "/tmp/includeList"
I know this must be really simple. But I have another list (/tmp/excludeList). I only want to echo the line within my while loop if the line ins't found in my excludeList. How do I do that. Is there some awk statement or something?
You can do this with grep alone:
$ cat file
blue
green
red
yellow
pink
$ cat exclude
green
pink
$ grep -vx -f exclude file
blue
red
yellow
The -v flag tells grep to only output the lines in file that are not found in exclude and the -x flags forces whole line matching.
use grep
while IFS=$'\n' read -r line || [[ -n "$line" ]]; do
if [[ -n ${line} ]] \
&& ! grep -xF "$line" excludefile &>/dev/null; then
echo -e "$lanIP\t$line" >> /tmp/ipList;
fi
done < "/tmp/includeList"
the -n $line means if $line is not empty
the grep returns true if $line is found in exclude file which is inverted by the ! so returns true if the line is not found.
-x means line matched so nothing else can appear on the line
-F means fixed string so if any metacharacters end up in $line they'll be matched literally.
Hope this helps
With awk:
awk -v ip=$lanIP -v OFS="\t" '
NR==FNR {exclude[$0]=1; next}
/[^[:space:]]/ && !($0 in exclude) {print ip, $0}
' /tmp/excludeList /tmp/includeList > /tmpipList
This reads the exclude list info an array (as the array keys) -- the NR==FNR condition is true while awk is reading the first file from the arguments. Then, while reading the include file, if the current line contains a non-space character and it does not exist in the exclude array, print it.
The equivalent with grep:
grep -vxF -f /tmp/excludeList /tmp/includeList | while IFS= read -r line; do
[[ -n "$line" ]] && printf "%s\t%s\n" "$ipList" "$line"
done > /tmp/ipList

Resources