How do I use Head and Tail to print specific lines of a file - shell

I want to say output lines 5 - 10 of a file, as arguments passed in.
How could I use head and tail to do this?
where firstline = $2 and lastline = $3 and filename = $1.
Running it should look like this:
./lines.sh filename firstline lastline

head -n XX # <-- print first XX lines
tail -n YY # <-- print last YY lines
If you want lines from 20 to 30 that means you want 11 lines starting from 20 and finishing at 30:
head -n 30 file | tail -n 11
#
# first 30 lines
# last 11 lines from those previous 30
That is, you firstly get first 30 lines and then you select the last 11 (that is, 30-20+1).
So in your code it would be:
head -n $3 $1 | tail -n $(( $3-$2 + 1 ))
Based on firstline = $2, lastline = $3, filename = $1
head -n $lastline $filename | tail -n $(( $lastline -$firstline + 1 ))

Aside from the answers given by fedorqui and Kent, you can also use a single sed command:
#!/bin/sh
filename=$1
firstline=$2
lastline=$3
# Basics of sed:
# 1. sed commands have a matching part and a command part.
# 2. The matching part matches lines, generally by number or regular expression.
# 3. The command part executes a command on that line, possibly changing its text.
#
# By default, sed will print everything in its buffer to standard output.
# The -n option turns this off, so it only prints what you tell it to.
#
# The -e option gives sed a command or set of commands (separated by semicolons).
# Below, we use two commands:
#
# ${firstline},${lastline}p
# This matches lines firstline to lastline, inclusive
# The command 'p' tells sed to print the line to standard output
#
# ${lastline}q
# This matches line ${lastline}. It tells sed to quit. This command
# is run after the print command, so sed quits after printing the last line.
#
sed -ne "${firstline},${lastline}p;${lastline}q" < ${filename}
Or, to avoid any external utilites, if you're using a recent version of bash (or zsh):
#!/bin/sh
filename=$1
firstline=$2
lastline=$3
i=0
exec <${filename} # redirect file into our stdin
while read ; do # read each line into REPLY variable
i=$(( $i + 1 )) # maintain line count
if [ "$i" -ge "${firstline}" ] ; then
if [ "$i" -gt "${lastline}" ] ; then
break
else
echo "${REPLY}"
fi
fi
done

try this one-liner:
awk -vs="$begin" -ve="$end" 'NR>=s&&NR<=e' "$f"
in above line:
$begin is your $2
$end is your $3
$f is your $1

Save this as "script.sh":
#!/bin/sh
filename="$1"
firstline=$2
lastline=$3
linestoprint=$(($lastline-$firstline+1))
tail -n +$firstline "$filename" | head -n $linestoprint
There is NO ERROR HANDLING (for simplicity) so you have to call your script as following:
./script.sh yourfile.txt firstline lastline
$ ./script.sh yourfile.txt 5 10
If you need only line "10" from yourfile.txt:
$ ./script.sh yourfile.txt 10 10
Please make sure that:
(firstline > 0) AND (lastline > 0) AND (firstline <= lastline)

Related

bash - print a line every X seconds (like sed every X lines)

I know with sed you can pipe the output of a command so that you can print every X lines.
make all | sed -n '2~5'
Is there an equivalent command to print a line every X seconds?
make all | print_line_every_sec '5'
In 5 seconds timeout read one line and discard anything else:
while
# timeout 5 seconds
! timeout 5 sh -c '
# read one line
if IFS= read -r line; then
# output the line
printf "%s\n" "$line"
# discard the input for the rest of 5 seconds
cat >/dev/null
fi
# will get here only, if there is nothing to read
'
# that means that `timeout` will always return 124 if stdin is still open
# and it will return 0 exit status only if there is nothing to read
# so we loop on nonzero exit status of timeout.
do :; done
and a oneliner:
while ! timeout 0.5 sh -c 'IFS= read -r line && printf "%s\n" "$line" && cat >/dev/null'; do :; done
But maybe something simpler - just discard 5 seconds of data each one line:
while IFS= read -r line; do
printf "%s\n" "$line"
timeout 5 cat >/dev/null
done
or
while IFS= read -r line &&
printf "%s\n" "$line" &&
! timeout 5 cat >/dev/null
do :; done
If you want the most recent message every 5 seconds, this is a try :
make all | {
display(){
if (( $SECONDS >= 5)); then
if test -n "${last_line+x}"; then
# print only if there is a message in the last 5 seconds
echo $last_line; unset last_line
fi
SECONDS=0
fi
}
SECONDS=0
while true; do
while IFS= read -t 0.001 line; do
last_line=$line
display
done
display
done
}
Even if the proposed solutions are interesting and beautiful, the most elegant solution IMHO is a awk solution. If you want to issue
make all | print_line_every_sec 5
then you have to create the script print_line_every_sec as follows, including a test to avoid an infinite loop:
#!/bin/bash
if [ $1 -le 0 ] ; then echo $(basename $0): invalid argument \'$1\'; exit 1; fi
awk -v delay=$1 'BEGIN {t = systime ()}
{if (systime() >= t) {print $0 ; t += delay}}'
This might work for you (GNU sed):
sed 'e sleep 1' file
Print a line every n (in the above example 1 ) second.
To print 5 lines every 2 seconds, use:
sed '1~5e sleep 2' file
You can do it by watch command.
If You need only print your output every X second, you could use something like this:
watch -n X "Your CMD"
if you need to designate any change on your output, it would be useful to use -d switch :
watch -n X -d "Your CMD"

How to browse a line from a file?

I have a file that contains 10 lines with this sort of content:
aaaa,bbb,132,a.g.n.
I wanna walk throw every line, char by char and put the data before the " , " is met in an output file.
if [ $# -eq 2 ] && [ -f $1 ]
then
echo "Read nr of fields to be saved or nr of commas."
read n
nrLines=$(wc -l < $1)
while $nrLines!="1" read -r line || [[ -n "$line" ]]; do
do
for (( i=1; i<=$n; ++i ))
do
while [ read -r -n1 temp ]
do
if [ temp != "," ]
then
echo $temp > $(result$i)
else
fi
done
paste -d"\n" $2 $(result$i)
done
nrLines=$($nrLines-1)
done
else
echo "File not found!"
fi
}
In parameter $2 I have an empty file in which I will store the data from file $1 after I extract it without the " , " and add a couple of comments.
Example:
My input_file contains:
a.b.c.d,aabb,comp,dddd
My output_file is empty.
I call my script: ./script.sh input_file output_file
After execution the output_file contains:
First line info: a.b.c.d
Second line info: aabb
Third line info: comp
(yes, without the 4th line info)
You can do what you want very simply with parameter-expansion and substring-removal using bash alone. For example, take an example file:
$ cat dat/10lines.txt
aaaa,bbb,132,a.g.n.
aaaa,bbb,133,a.g.n.
aaaa,bbb,134,a.g.n.
aaaa,bbb,135,a.g.n.
aaaa,bbb,136,a.g.n.
aaaa,bbb,137,a.g.n.
aaaa,bbb,138,a.g.n.
aaaa,bbb,139,a.g.n.
aaaa,bbb,140,a.g.n.
aaaa,bbb,141,a.g.n.
A simple one-liner using native bash string handling could simply be the following and give the following results:
$ while read -r line; do echo ${line%,*}; done <dat/10lines.txt
aaaa,bbb,132
aaaa,bbb,133
aaaa,bbb,134
aaaa,bbb,135
aaaa,bbb,136
aaaa,bbb,137
aaaa,bbb,138
aaaa,bbb,139
aaaa,bbb,140
aaaa,bbb,141
Paremeter expansion w/substring removal works as follows:
var=aaaa,bbb,132,a.g.n.
Beginning at the left and removing up to, and including, the first ',' is:
${var#*,} # bbb,132,a.g.n.
Beginning at the left and removing up to, and including, the last ',' is:
${var##*,} # a.g.n.
Beginning at the right and removing up to, and including, the first ',' is:
${var%,*} # aaaa,bbb,132
Beginning at the left and removing up to, and including, the last ',' is:
${var%%,*} # aaaa
Note: the text to remove above is represented with a wildcard '*', but wildcard use is not required. It can be any allowable text. For example, to only remove ,a.g.n where the preceding number is 136, you can do the following:
${var%,136*},136 # aaaa,bbb,136 (all others unchanged)
To print 2016 th line from a file named file.txt u have to run a command like this-
sed -n '2016p' < file.txt
More-
sed -n '2p' < file.txt
will print 2nd line
sed -n '2011p' < file.txt
2011th line
sed -n '10,33p' < file.txt
line 10 up to line 33
sed -n '1p;3p' < file.txt
1st and 3th line
and so on...
For more detail, please have a look in this tutorial and this answer.
In native bash the following should do what you want, assuming you replace the contents of your script.sh with the below:
#!/bin/bash
IN_FILE=${1}
OUT_FILE=${2}
IFS=\,
while read line; do
set -- ${line}
for ((i=1; i<=${#}; i++)); do
((${i}==4)) && continue
((n+=1))
printf '%s\n' "Line ${n} info: ${!i}"
done
done < ${IN_FILE} > ${OUT_FILE}
This will not print the 4th field of each line within the input file, on a new line in the output file (I assume this is your requirement as per your comment?).
[wspace#wspace sandbox]$ awk -F"," 'BEGIN{OFS="\n"}{for(i=1; i<=NF-1; i++){print "line Info: "$i}}' data.txt
line Info: a.b.c.d
line Info: aabb
line Info: comp
This little snippet can ignore the last field.
updated:
#!/usr/bin/env bash
if [ ! -f "$1" -o $# -ne 2 ];then
echo "Usage: $(basename $0) input_file out_file"
exit 127
fi
input_file=$1
output_file=$2
: > $output_file
if [ "$(wc -l < $1)" -ne 0 ];then
while true
do
read -r -n1 char
if [ "$char" == "" ];then
break
elif [ $char != "," ];then
temp=$temp$char
else
echo "line info: $temp" >> $output_file
temp=""
fi
done < $input_file
else
echo "file $1 is empty"
fi
Maybe this is what you want
Did you try
sed "s|,|\n|g" $1 | head -n -1 > $2
I assume that only the last word would not have a comma on its right.
Try this (tested with you sample line) :
#!/bin/bash
# script.sh
echo "Number of fields to save ?"
read nf
while IFS=$',' read -r -a arr; do
newarr=${arr[#]:0:${nf}}
done < "$1"
for i in ${newarr[#]};do
printf "%s\n" $i
done > "$2"
Execute script with :
$ ./script.sh inputfile outputfile
Number of fields ?
3
$ cat outputfile
a.b.c.d
aabb
comp
All words separated with commas are stored into an array $arr
A tmp array $newarr removes last $n element ($n get the read command).
It loops over new array and prints result in $2, the outputfile.

Bash script to remove redundant lines

Good afternoon,
I'm trying to make a bash script that cleans out some data output files. The files look like this:
/path/
/path/to
/path/to/keep
/another/
/another/path/
/another/path/to
/another/path/to/keep
I'd like to end up with this:
/path/to/keep
/another/path/to/keep
I want to cycle through lines of the file, checking the next line to see if it contains the current line, and if so, delete the current line from the file. Here's my code:
for LINE in $(cat bbutters_data2.txt)
do
grep -A1 ${LINE} bbutters_data2.txt
if [ $? -eq 0 ]
then
sed -i '/${LINE}/d' ./bbutters_data2.txt
fi
done
Assuming that your input file is sorted in the way that you have shown:
$ awk 'NR>1 && substr($0,1,length(last))!=last {print last;} {last=$0;} END{print last}' file
/path/to/keep
/another/path/to/keep
How it works
awk reads through the input file line by line. Every time we read a new line, we compare it to the last. If the new line does not contain the last line, then we print the last line. In more detail:
NR>1 && substr($0,1,length(last))!=last {print last;}
If this is not the first line and if the last line, called last, is not contained in the current line, $0, then print the last line.
last=$0
Update the variable last to the current line.
END{print last}
After we finish reading the file, print the last line.
I like the awk solution, but bash itself can handle the task. Note: the solution (both awk and bash), require that the lesser included paths be listed in increasing order. Here is an alternative bash solution (bash only due to the glob match operation):
#!/bin/bash
fn="${1:-/dev/stdin}" ## accept filename or stdin
[ -r "$fn" ] || { ## validate file is readable
printf "error: file not found: '%s'\n" "$fn"
exit 1
}
declare -i cnt=0 ## flag for 1st iteration
while read -r line; do ## for each line in file
## if 1st iteration, fill 'last', increment 'cnt', continue
[ $cnt -eq 0 ] && { last="$line"; ((cnt++)); continue; }
## while 'line' is a child of 'last', continue, else print
[[ $line = "${last%/}"/* ]] || printf "%s\n" "$last"
last="$line" ## update last=$line
done <"$fn"
[ ${#line} -eq 0 ] && ## print last line (updated for non POSIX line end)
printf "%s\n" "$last" ||
printf "%s\n" "$line"
exit 0
Output
$ bash path_uniql.sh < dat/incpaths.txt
/path/to/keep
/another/path/to/keep

Correctly count number of lines a bash variable

I need to count the number of lines of a given variable. For example I need to find how many lines VAR has, where VAR=$(git log -n 10 --format="%s").
I tried with echo "$VAR" | wc -l), which indeed works, but if VAR is empty, is prints 1, which is wrong. Is there a workaround for this? Something better than using an if clause to check whether the variable is empty...(maybe add a line and subtract 1 from the returned value?).
The wc counts the number of newline chars. You can use grep -c '^' for counting lines.
You can see the difference with:
#!/bin/bash
count_it() {
echo "Variablie contains $2: ==>$1<=="
echo -n 'grep:'; echo -n "$1" | grep -c '^'
echo -n 'wc :'; echo -n "$1" | wc -l
echo
}
VAR=''
count_it "$VAR" "empty variable"
VAR='one line'
count_it "$VAR" "one line without \n at the end"
VAR='line1
'
count_it "$VAR" "one line with \n at the end"
VAR='line1
line2'
count_it "$VAR" "two lines without \n at the end"
VAR='line1
line2
'
count_it "$VAR" "two lines with \n at the end"
what produces:
Variablie contains empty variable: ==><==
grep:0
wc : 0
Variablie contains one line without \n at the end: ==>one line<==
grep:1
wc : 0
Variablie contains one line with \n at the end: ==>line1
<==
grep:1
wc : 1
Variablie contains two lines without \n at the end: ==>line1
line2<==
grep:2
wc : 1
Variablie contains two lines with \n at the end: ==>line1
line2
<==
grep:2
wc : 2
You can always write it conditionally:
[ -n "$VAR" ] && echo "$VAR" | wc -l || echo 0
This will check whether $VAR has contents and act accordingly.
For a pure bash solution: instead of putting the output of the git command into a variable (which, arguably, is ugly), put it in an array, one line per field:
mapfile -t ary < <(git log -n 10 --format="%s")
Then you only need to count the number of fields in the array ary:
echo "${#ary[#]}"
This design will also make your life simpler if, e.g., you need to retrieve the 5th commit message:
echo "${ary[4]}"
try:
echo "$VAR" | grep ^ | wc -l

I need to parse a log file into multiple files based on delimiters

I have a log file which i need to parse it into multiple files.
############################################################################################
6610
############################################################################################
GTI02152 I gtirreqi 20130906 000034 TC SJ014825 GTT_E_REQ_INF テーブル挿入件数 16件
############################################################################################
Z5000
############################################################################################
GTP10000 I NIPS gtgZ5000 20130906 000054 TC SJ014825 シェル開始
############################################################################################
I need to create files like 6610.txt which will have all values under 6610 like(GTI02152..) and for z5000(GTP10000) respectively. Any help will be greatly appreciated!
Below script would help you to get the information. You can modify them to create the data you require.
#!/bin/sh
cmd=`cat data.dat | paste -d, - - - - - | cut -d ',' -f 2,4 > file.out`
$cmd
while read p; do
fileName=`echo $p | cut -d ',' -f 1`
echo $fileName
dataInfo=`echo $p | cut -d ',' -f 2`
echo $dataInfo
done< file.out
Here's an awk styled answer:
I put the following into a file named awko and chmod +x it to use it:
#!/usr/bin/awk -f
BEGIN { p = 0 } # look for filename flag - start at zero
/^\#/ { p = !p } # turn it on to find the filename
# either make a filename or write to the last filename based on the flag
$0 !~ /^\#/ {
if( p == 1 ) filename = $1 ".txt"
else print $0 > filename
}
Running awko data.txt produced two files, 6610.txt and Z5000.txt from your example data. It's capable of sending more data lines to the output files as well.
You can do it with Ruby as well:
ruby -e 'File.read(ARGV.shift).scan(/^[^#].*?(?=^[#])/m).each{|e| name = e.split[0]; File.write("#{name}.txt", e)}' file
Example output:
> for A in *.txt; do echo "---- $A ----"; cat "$A"; done
---- 6610.txt ----
6610
---- GTI02152.txt ----
GTI02152 I gtirreqi 20130906 000034 TC SJ014825 GTT_E_REQ_INF テーブル挿入件数 16件
---- GTP10000.txt ----
GTP10000 I NIPS gtgZ5000 20130906 000054 TC SJ014825 シェル開始
---- Z5000.txt ----
Z5000
This script makes the following assumptions:
Each record is separated by an empty line
#### lines are purely comment/space filler and can be ignored during parsing
The first line of each record (ignoring ####) contains the basename for the filename
The name of the logfile is passed as the first argument to this script.
#!/bin/bash
# write records to this temporary file, rename later
tempfile=$(mktemp)
while read line; do
if [[ $line == "" ]] ; then
# line is empty - separator - save existing record and start a new one
mv $tempfile $filename
filename=""
tempfile=$(mktemp)
else
# output non-empty line to record file
echo $line >> $tempfile
if [[ $filename == "" ]] ; then
# we haven't yet figured out the filename for this record
if [[ $line =~ ^#+$ ]] ; then
# ignore #### comment lines
:
else
# 1st non-comment line in record is filename
filename=${line}.txt
fi
fi
fi
done < $1
# end of input file might not have explicit empty line separator -
# make sure last record file is moved correctly
if [[ -e $tempfile ]] ; then
mv $tempfile $filename
fi

Resources