I want to count the number of lines output from a command in a bash script. i.e.
COUNT=ls | wc -l
But I also want the script to output the original output from ls. How to get this done? (My actual command is not ls and it has side effects. So I can't run it twice.)
The tee(1) utility may be helpful:
$ ls | tee /dev/tty | wc -l
CHANGES
qpi.doc
qpi.lib
qpi.s
4
info coreutils "tee invocation" includes this following example, which might be more instructive of tee(1)'s power:
wget -O - http://example.com/dvd.iso \
| tee >(sha1sum > dvd.sha1) \
>(md5sum > dvd.md5) \
> dvd.iso
That downloads the file once, sends output through two child processes (as started via bash(1) process substitution) and also tee(1)'s stdout, which is redirected to a file.
ls | tee tmpfile | first command
cat tmpfile | second command
Tee is a good way to do that, but you can make something simpler:
ls > __tmpfile
cat __tmpfile | wc -l
cat __tmpfile
rm __tmpfile
Related
I try to constantly display the size of the current folder using watch , the below command does not work however, what do I do wrong ? I use zsh shell
$ watch ls -a | wc -l
what do I do wrong ?
Shell parses | as a pipe. So when shell sees:
watch ls -a | wc -l
It parses it as two command with one command output redirected to the other:
( watch ls -a ) | ( wc -l )
It runs the command watch with two arguments ls and -a and a command wc with single argument -l. Because watch ls -a never ends and wc -l only outputs when the input ends, you don't see anything printed out. wc -l waits until all input lines are printed, which never happens.
Because watch internally calls the shell, you can:
watch 'ls -a | wc -l'
This runs a single command watch with one argument ls -a | wc -l. watch internally spawns a shell and passes the string ls -a | wc -l to it. Then this internal shell spawns two new processes ls -a and wc -l with input/output connected.
Till today I was always able to find answer for my all bash questions. But now I stuck. I am testing 20TB RAID6 configuration working on LSI 9265.
I wrote script to create files from /dev/urandom and I am creating second to calculate md5 from all files with two addons.
One is to use time command to calculate md5sum execution time
Second is use pv command to show progress of each md5sum command
My command looks like this:
filename="2017-03-13_12-38-08"
/usr/bin/time -f "real read %E" pv $filename | md5sum | sed "s/-/$filename /"
This is example terminal printout:
/usr/bin/time -f "real read %E" pv $i | md5sum | sed "s/-/$i/"
1GiB 0:00:01 [ 551MiB/s] [==================================================================================================>] 100%
real read 0:01.85
f561af8cc0927967c440fe2b39db894a 2017-03-13_12-38-08
And I want to log it to file. I failed all tries using 2>&1, using tee, using brackets. I know pv uses stdErr but this doesnt help in finding solution. I can only catch "f561af8cc0927967c440fe2b39db894a 2017-03-13_12-38-08_done"
which is not enough.
This is the solution:
(time pv -f $filename | md5sum | sed "s/-/$filename/") 2>&1 | tee output.log
or equivalent but without printing into terminal only file to output.log
(time pv -f $filename | md5sum | sed "s/-/$filename/") > output.log 2>&1
So I've made a script which is collecting data from many different files:
#!/bin/bash
mkdir DATAPOOL"$1"
grep achi *out>runner
grep treat *out>>runner
cat runner | grep Primitive *gout | grep '= '|awk '{print $1,$6}' > CellVolume"$1".txt
cat runner | grep ' c ' *gout | grep 'Angstrom '|awk '{print $1,$3}' > Cellc"$1".txt
cat runner | grep 'Final energy ' *gout |awk '{print $1,$5}' > CellEnergy"$1".txt
etc etc
cat runner |awk '{print "~/xtlanal",$1," > ",$1}' >runner2
vi runner2
:1,$s/gout:/xtl/
:1,$s/gout:/dat/
:wq
source runner2
grep Summary *dat | grep 'CAT-O ' |awk '{print $1,$6}' > AVE_NaO_"$1".txt
mv *txt DATAPOOL"$1"
So I end up with all the required text files when run without the vi part and so I know it all works. Furthermore when I run it with the vi commands, it just stops running at the vi command and then i can manually enter the 3 commands and I end up with the correct results. What I'm struggling with is I cant get vi to run the commands on its own so I can just execute the file multiple times within different directories and not have to manually enter commands time and time again.
Any help would be greatly appreciated.
Cheers
something like this as a bash script:
#!/bin/bash
vi filename.txt -c ':g/^/m0' -c ':wq'
where -c execute a command. Here the command is to reverse the lines in a textfile. After done, :wq to save and exit. (man vi to get more about -c)
If you don't want to type -c twice, you can do it this way:
vi -c "g/^/m0 | wq" filename.txt
For scripted editing tasks, you can use ed instead of vi:
ed runner2 <<'END'
1,$s/gout:/xtl/
1,$s/gout:/dat/
w
q
END
For global line-oriented search and replace, sed is a good choice:
sed -i 's/gout:/xtl/; s/gout:/dat/' runner2
Tested on VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Apr 10 2018 21:31:58)
The vi -c "g/^/m0 | wq" filename.txt may appear to work, but it does not actually!
Typing vi -c "g/^/m0 | wq" filename.txt will result in vi writing and quitting before any major changes are made to the file. (using the pipe in this situation will attempt to execute the wq line by line forcing it to quit before the intended operation)
In order to see a demonstration try typing it without the q and see how slow it works writing line by line:
vi -c "g/^/m0 | w" filename.txt
The more efficient way is using -c as B. Kocis states, or use +.
As B. Kocis stated:
#!/bin/bash
vi filename.txt -c ':g/^/m0' -c ':wq'
or
vi filename.txt +g/^/m0 +wq
How can I both write to a file and display to screen using pipe with tee?
This command actually do it, the problem is that it writes to a new file and tail -f give me an error "truncate file".
ls -al | tee file.txt
-a option of tee is what you are looking for
-a, --append
append to the given FILEs, do not overwrite
so your line would be:
ls -al | tee -a file.txt
couldn't find this on SO. I ran the following command in the terminal:
>> grep -Rl "curl" ./
and this displays the list of files where the keyword curl occurs. I want to count the number of files. First way I can think of, is to count the number of lines in the output that came in the terminal. How can I do that?
Pipe the result to wc using the -l (line count) switch:
grep -Rl "curl" ./ | wc -l
Putting the comment of EaterOfCode here as an answer.
grep itself also has the -c flag which just returns the count
So the command and output could look like this.
$ grep -Rl "curl" ./ -c
24
EDIT:
Although this answer might be shorter and thus might seem better than the accepted answer (that is using wc). I do not agree with this anymore. I feel like remembering that you can count lines by piping to wc -l is much more useful as you can use it with other programs than grep as well.
Piping to 'wc' could be better IF the last line ends with a newline (I know that in this case, it will)
However, if the last line does not end with a newline 'wc -l' gives back a false result.
For example:
$ echo "asd" | wc -l
Will return 1 and
$ echo -n "asd" | wc -l
Will return 0
So what I often use is grep <anything> -c
$ echo "asd" | grep "^.*$" -c
1
$ echo -n "asd" | grep "^.*$" -c
1
This is closer to reality than what wc -l will return.
"abcd4yyyy" | grep 4 -c gives the count as 1