How to retrieve a single line from the file?
file.txt
"aaaaaaa"
"bbbbbbb"
"ccccccc"
"ddddddd"
I need to retrieve the line 3 ("ccccccc")
Thank you.
sed is your friend. sed -n 3p prints the third line (-n: no automatic print, 3p: print when line number is 3). You can also have much more complex patterns, for example sed -n 3,10p to print lines 3 to 10.
If the file is very big, you may consider to not cycle through the whole file, but quit after the print. sed -n '3{p;q}'
If you know you need line 3, one approach is to use head to get the first three lines, and tail to get only the last of these:
varname="$(head -n 3 file.txt | tail -n 1)"
Another approach, using only Bash builtins, is to call read three times:
{ read ; read ; IFS= read -r varname } < file.txt
Here's a way to do it with awk:
awk 'FNR==3 {print; exit}' file.txt
Explanation:
awk '...' : Invoke awk, a tool for manipulating files line-by-line. Instructions enclosed by single quotes are executed by awk.
FNR==3 {print; exit}: FNR stands for "File Number Records"; just think of it as "number of lines read so far for this file". Here we are saying, if we are on the 3rd line of the file, print the entire line and then exit awk immediately so we don't waste time reading the rest of a large file.
file.txt: specify the input file as an argument to awk to save a cat.
There are many possibilities: Try so:
sed '3!d' test
Here is a very fast version:
sed "1d; 2d; 3q"
Are other tools than bash allowed? On systems that include bash, you'll usually find sed and awk or other basic tools:
$ line="$(sed -ne 3p input.txt)"
$ echo "$line"
or
$ read line < <(awk 'NR==3' input.txt)
$ echo "$line"
or if you want to optimize this by quitting after the 3rd line is read:
$ read line < <(awk 'NR==3{print;nextfile}' input.txt)
$ echo "$line"
or how about even simpler tools (though less optimized):
$ line="`head -n 3 input.txt | tail -n 1`"
$ echo "$line"
Of course, if you really want to do this all within bash, you can still make it a one-liner, without using any external tools.
$ for (( i=3 ; i-- ; )); do read line; done < input.txt
$ echo "$line"
There are many ways to achieve the same thing. Pick one that makes sense for your task. Next time, perhaps explain your overall needs a bit better, so we can give you answers more applicable to your situation.
Since, as usual, all the other answers involve trivial and usual stuff (pipe through grep then awk then sed then cut or you-name-it), here's a very unusual and (sadly) not very well-known one (so, I hereby claim that I have the most original answer):
mapfile -s2 -n3 -t < input.txt
echo "$MAPFILE"
I would say this is fairly efficient (mapfile is quite efficient and it's a bash builtin).
Done!
Fast bash version;
while (( ${i:-1} <= 3 )); do
(( $i == 3 )) && read -r line; (( i++ ))
done < file.txt
Output
echo "$line" # Third line
"ccccccc"
Explanation
while (( ${i:-1} <= 3 )) - Count until $i equals 3 then exit loop.
(( $i == 3 )) - If $i is equal to 3 execute read line.
read -r line - Read the file line into variable $line.
(( i++ )) - Increment $i by 1 at each loop.
done < file.txt - Pipe file into while loop.
Related
This question already has an answer here:
POSIX shell equivalent to <()
(1 answer)
Closed 3 years ago.
I want to create a script to read a .txt file. This is my code:
while IFS= read -r lines
do
echo "$lines"
done < <(tail -n +2 filename.txt)
I tried a lot of things like:
<<(tail -n +2 in.txt)
< < (tail -n +2 in.txt)
< (tail -n +2 in.txt)
<(tail -n +2 in.txt)
(tail -n +2 in.txt)
I expected to print me from the second line but instead I get an error:
Syntax error: redirection unexpected
If you just want to ignore the first line, there's no good reason to use tail at all!
{
read -r first_line
while IFS= read -r line; do
printf '%s\n' "$line"
done
} <filename.txt
Using read to consume the first line leaves the original file pointer intact, so following code can read directly from the file, instead of reading from a FIFO attached to the output of the tail program; it's thus much lower-overhead.
If you did want to use tail, for the specific case raised, you don't need to use a process substitution (<(...)), but can simply pipe into your while loop. Note that this has a serious side effect, insofar as any variables you set in the loop will no longer be available after it exits; this is documented (in a cross-shell manner) in BashFAQ #24.
tail -n +2 filename.txt | while IFS= read -r line
do
printf '%s\n' "$line"
done
As it says in this answer
POSIX shell equivalent to <()
you could use named pipes to simulate process substitution in
POSIX. Your script would look like that:
#!/usr/bin/env sh
mkfifo foo.fifo
tail -n +2 filename.txt >foo.fifo &
while IFS= read -r lines
do
echo "$lines"
done < foo.fifo
rm foo.fifo
I have a log file with a lot of lines with the following format:
IP - - [Timestamp Zone] 'Command Weblink Format' - size
I want to write a script.sh that gives me the number of times each website has been clicked.
The command:
awk '{print $7}' server.log | sort -u
should give me a list which puts each unique weblink in a separate line. The command
grep 'Weblink1' server.log | wc -l
should give me the number of times the Weblink1 has been clicked. I want a command that converts each line created by the Awk command above to a variable and then create a loop that runs the grep command on the extracted weblink. I could use
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
done
(source: Read a file line by line assigning the value to a variable) but I don't want to save the output of the Awk script in a .txt file.
My guess would be:
while IFS='' read -r line || [[ -n "$line" ]]; do
grep '$line' server.log | wc -l | ='$variabel' |
echo " $line was clicked $variable times "
done
But I'm not really familiar with connecting commands in a loop, as this is my first time. Would this loop work and how do I connect my loop and the Awk script?
Shell commands in a loop connect the same way they do without a loop, and you aren't very close. But yes, this can be done in a loop if you want the horribly inefficient way for some reason such as a learning experience:
awk '{print $7}' server.log |
sort -u |
while IFS= read -r line; do
n=$(grep -c "$line" server.log)
echo "$line" clicked $n times
done
# you only need the read || [ -n ] idiom if the input can end with an
# unterminated partial line (is illformed); awk print output can't.
# you don't really need the IFS= and -r because the data here is URLs
# which cannot contain whitespace and shouldn't contain backslash,
# but I left them in as good-habit-forming.
# in general variable expansions should be doublequoted
# to prevent wordsplitting and/or globbing, although in this case
# $line is a URL which cannot contain whitespace and practically
# cannot be a glob. $n is a number and definitely safe.
# grep -c does the count so you don't need wc -l
or more simply
awk '{print $7}' server.log |
sort -u |
while IFS= read -r line; do
echo "$line" clicked $(grep -c "$line" server.log) times
done
However if you just want the correct results, it is much more efficient and somewhat simpler to do it in one pass in awk:
awk '{n[$7]++}
END{for(i in n){
print i,"clicked",n[i],"times"}}' |
sort
# or GNU awk 4+ can do the sort itself, see the doc:
awk '{n[$7]++}
END{PROCINFO["sorted_in"]="#ind_str_asc";
for(i in n){
print i,"clicked",n[i],"times"}}'
The associative array n collects the values from the seventh field as keys, and on each line, the value for the extracted key is incremented. Thus, at the end, the keys in n are all the URLs in the file, and the value for each is the number of times it occurred.
How do i iterate through a file and print the first word only. The line is colon separated. example
root:01:02:toor
the file contains several lines. And this is what i've done so far but it does'nt work.
FILE=$1
k=1
while read line; do
echo $1 | awk -F ':'
((k++))
done < $FILE
I'm not good with bash-scripting at all. So this is probably very trivial for one of you..
edit: variable k is to count the lines.
Use cut:
cut -d: -f1 filename
-d specifies the delimiter
-f specifies the field(s) to keep
If you need to count the lines, just
count=$( wc -l < filename )
-l tells wc to count lines
awk -F: '{print $1}' FILENAME
That will print the first word when separated by colon. Is this what you are looking for?
To use a loop, you can do something like this:
$ cat test.txt
root:hello:1
user:bye:2
test.sh
#!/bin/bash
while IFS=':' read -r line || [[ -n $line ]]; do
echo $line | awk -F: '{print $1}'
done < test.txt
Example of reading line by line in bash: Read a file line by line assigning the value to a variable
Result:
$ ./test.sh
root
user
A solution using perl
%> perl -F: -ane 'print "$F[0]\n";' [file(s)]
change the "\n" to " " if you don't want a new line printed.
You can get the first word without any external commands in bash like so:
printf '%s' "${line%%:*}"
which will access the variable named line and delete everything that matches the glob :* and do so greedily, so as close to the front (that's the %% instead of a single %).
Though with this solution you do need to do the loop yourself. If this is the only thing you want to do with the variable the cut solution is better so you don't have to do the file iteration yourself.
I have a simple bash script I have written to count the number of lines in a collection of text files, and I store each number of lines as a variable using a for loop. I would like to print each variable to the same text file, so that I may access all the line counts at once, from the same file.
My code is:
for f in *Daily.txt; do
lines=$(cat $f | wc -l);
lines=$(($num_lines -1));
echo $lines > /destdrive/linesTally2014.txt;
done
When I run this, the only output I receive is of the final file, not all the other files.
If anyone could help me with this I would really appreciate it. I am new to bash scripting, so please excuse this novice question.
You create the file on each iteration. Move the I/O redirection after the done. Use:
for f in *Daily.txt
do
echo $(( $(wc -l < $f) - 1))
done > /destdrive/linesTally2014.txt
This avoids the variable; if you have a need for it, you can use a fixed version of the original code (use $lines throughout, instead of using $num_lines once). Note that the code in the question has a UUoC (Useless Use of cat) that this version avoids.
You can avoid the loop with
wc -l *Daily.txt | awk '{ print $1 }' > /destdrive/linesTally2014.txt
or (when you want 1 less)
wc -l *Daily.txt | awk '{ print $1 -1 }' > /destdrive/linesTally2014.txt
The above suggestions are probably better, but the problem you're having with your script is your use of the > for redirection, which overwrites the file. Use >> and it will append to the file.
echo $lines >> /destdrive/linesTally2014.txt
I have a list of files (~1000) and there is 1 file per line in my text file named: 'files.txt'
I have a macro that looks something like the following:
#!/bin/sh
b=$(sed '${1}q;d' files.txt)
cat > MyMacro_${1}.C << +EOF
myFile = new TFile("/MYPATHNAME/$b");
+EOF
and I use this input script by doing
./MakeMacro.sh 1
and later I want to do
./MakeMacro.sh 2
./MakeMacro.sh 3
...etc
So that it reads the n'th line of my files.txt and feeds that string to my created .C macro.
So that it reads the n'th line of my files.txt and feeds that string to my created .C macro.
Given this statement and your tags, I'm going to answer using shell tools and not really address the issue of the .c macro.
The first line of your script contains a sed script. There are numerous ways to get the Nth line from a text file. The simplest might be to use head and tail.
$ head -n "${i}" files.txt | tail -n 1
This takes the first $i lines of files.txt, and shows you the last 1 lines of that set.
$ sed -ne "${i}p" files.txt
This use of sed uses -n to avoid printing by default, then prints the $ith line. For better performance, try:
$ sed -ne "${i}{p;q;}" files.txt
This does the same, but quits after printing the line, so that sed doesn't bother traversing the rest of the file.
$ awk -v i="$i" 'NR==i' files.txt
This passes the shell variable $i into awk, then evaluates an expression that tests whether the number of records processed is the same as that variable. If the expression evaluates true, awk prints the line. For better performance, try:
$ awk -v i="$i" 'NR==i{print;exit}' files.txt
Like the second sed script above, this will quit after printing the line, so as to avoid traversing the rest of the file.
Plenty of ways you could do this by loading the file into an array as well, but those ways would take more memory and perform less well. I'd use one-liners if you can. :)
To take any of these one-liners and put it into your script, you already have the notation:
if expr "$i" : '[0-9][0-9]*$' >/dev/null; then
b=$(sed -ne "${i}{p;q;}" files.txt)
else
echo "ERROR: invalid line number" >&2; exit 1
fi
If I am understanding you correctly, you can do a for loop in bash to call the script multiple times with different arguments.
for i in `seq 1 n`; do ./MakeMacro.sh $i; done
Based on the OP's comment, it seems that he wants to submit the generated files to Condor. You can modify the loop above to include the condor submission.
for i in `seq 1 n`; do ./MakeMacro.sh $i; condor_submit <OutputFile> ; done
i=0
while read file
do
((i++))
cat > MyMacro_${i}.C <<-'EOF'
myFile = new TFile("$file");
EOF
done < files.txt
Beware: you need tab indents on the EOF line.
I'm puzzled about why this is the way you want to do the job. You could have your C++ code read files.txt at runtime and it would likely be more efficient in most ways.
If you want to get the Nth line of files.txt into MyMacro_N.C, then:
{
echo
sed -n -e "${1}{s/.*/myFile = new TFILE(\"&\");/p;q;}" files.txt
echo
} > MyMacro_${1}.C
Good grief. The entire script should just be (untested):
awk -v nr="$1" 'NR==nr{printf "\nmyFile = new TFile(\"/MYPATHNAME/%s\");\n\n",$0 > ("MyMacro_"nr".C")}' files.txt
You can throw in a ;exit before the } if performance is an issue but I doubt if it will be.