I am programming a Bash-like shell. I am having trouble understanding how this interaction works.
This command
ls > out | cat < out
only outputs the ls the first time I run it, and then nothing. In zsh it outputs everytime but not in Bash.
You're trying to give the parser conflicting directives.
This is like telling someone to "Turn to the left on your right."
<, >, and | all instruct the interpreter to redirect I/O according to rules.
Look at this bash example:
$: echo two>two # creates a file named two with the word two in it
$: echo one | cat < two <<< "three" << END
four
END
four
$: echo one | cat < two <<< three
three
$: echo one | cat < two
two
$: echo one | cat
one
Understand that putting a pipe character (|) between commands links the output of the first one to the input of the second one, so also giving each an input/output redirection that conflicts with that is nonsensical.
ls | cat # works - output of ls is input for cat
ls > out; cat < out # works - ls outputs to out, then cat reads out
ls > >(cat) # works
cat < <(ls) # works
but ls >out | cat sends the output from ls to out, and then attaches the output of that operation (of which there is none, because it's already been captured) to cat, which exits with no input or output.
If what you wanted was to have the output both go to a file and to the console, then either use ls > out; cat < out which makes them separate operations, or try
ls | tee out
which explicitly splits the stream to both the file and stdout.
Related
i have a program that outputs to stdout (actually it outputs to stderr, but i can easily redirect that to stdout with 2>&1 or the like.
i would like to run grep on the output of the program, and redirect all matches to stderr while leaving the unmatched lines on stdout (alternatively, i'd be happy with getting all lines - not just the unmatched ones - on stdout)
e.g.
$ myprogram() {
cat <<EOF
one line
a line with an error
another line
EOF
}
$ myprogram | greptostderr error >/dev/null
a line with an error
$ myprogram | greptostderr error 2>/dev/null
one line
another line
$
a trivial solution would be:
myprogram | tee logfile
grep error logfile 1>&2
rm logfile
however, i would rather get the matching lines on stderr when they occur, not when the program exits...
eventually, I found this, which gave me a hint to for a a POSIX solution like so:
greptostderr() {
while read LINE; do
echo $LINE
echo $LINE | grep -- "$#" 1>&2
done
}
for whatever reasons, this does not output anything (probably a buffering problem).
a somewhat ugly solution that seems to work goes like this:
greptostderr() {
while read LINE; do
echo $LINE
echo $LINE | grep -- "$#" | tee /dev/stderr >/dev/null
done
}
are there any better ways to implement this?
ideally i'm looking for a POSIX shell solution, but bash is fine as well...
I would use awk instead of grep, which gives you more flexibility in handling both matched and unmatched lines.
myprogram | awk -v p=error '{ print > ($0 ~ p ? "/dev/stderr" : "/dev/stdout")}'
Every line will be printed; the result of $0 ~ p determines whether the line is printed to standard error or standard output. (You may need to adjust the output file names based on your file system.)
I'm working on recreating my own shell environment copied from bash, and I found a pretty weird behavior with the real bash: when I enter
cat | tr -d 'o' < file1
(file1 contains only the text "Bonjour")
It outputs Bnjur, so until here no problem but it stays in a 'waiting for input' state until I press enter. At first I thought it was cat reading on stdin after tr execution, but it doesn't behave the same, it just waits for the user to press enter and (apparently) does nothing.
I saw on some bash documentation that < redirection redirects the input to the first SimpleCommand (before the first pipe), so it should redirect file1 on cat then redirect cat's output to tr, so it should only output Bnjur and nothing else, so why do we have to press enter to exit the command ?
Thanks for your help.
The < file1 redirection only applies to the tr command, not the entire pipeline.
So cat is reading from the original standard input, which is connected to the terminal. It's hanging because it's waiting for you to type something.
Meanwhile, tr is reading from the file. It exits when it finishes processing the file.
Once you type something, cat will write it to the pipe. Since tr has exited, there's no reader on the pipe, so cat will get a SIGPIPE signal and terminate.
If you want the redirection to apply to cat, use
cat < file1 | tr -d 'o'
If you want it to apply to the entire pipeline, you can group it in a subshell:
( cat | tr -d '0' ) < file1
You are redirecting input from file into tr, cat itself has no input and is thus taking input from stdin. Try this instead.
cat file1 | tr -d 'o'
In bash I have
grep -vf <(cat myfile <(grep -f myfile otherfile)) otherfile
Given the repetition of myfile, I thought I could pipe it via stdin like so
cat myfile | grep -vf <(cat - <(grep -f - otherfile)) otherfile
However this gives me different results.
So my question is where does the innermost substituted process i.e the grep -f - otherfile, get it's stdin from
A secondary question would be whether there is any advantage to trying to substitute the repeated file name with the same thing passed from stdin
Bash will fork a subshell for process substitution and it will inherit stdin from the current shell.
For your case, the whole right side of | is also running in a subshell so <()'s stdin is the same as this subshell. So <()'s stdin is also from cat myfile.
See the following simpler example:
[STEP 100] # echo $BASH_VERSION
5.0.7(3)-release
[STEP 101] # echo hello | cat <( tr a-z A-Z )
HELLO
[STEP 102] #
Construct the pipe to execute the following job.
"Output of ls should be displayed on the screen and from this output the lines
containing the word ‘poem’ should be counted and the count should be
stored in a file.”
If bash is allowed, use a process substitution as the receiver for tee
ls | tee >( grep -c poem > number.of.poetry.files)
Your attempt was close:
ls | tee /dev/tty | grep poem | wc -l >number_of_poems
The tee /dev/tty copies all ls output to the terminal. This satisfies the requirement that "Output of ls should be displayed on the screen." while also sending ls's output to grep's stdin.
This can be further simplified:
ls | tee /dev/tty | grep -c poem >number_of_poems
Note that neither of these solutions require bash. Both will work with lesser shells and, in particular, with dash which is the default /bin/sh under debian-like systems.
This sounds like a homework assignment :)
#!/bin/bash
ls
ls -l | grep -c poem >> file.txt
The first ls will display the output on the screen
The next line uses a series of pipes to output the number of files/directories containing "poem"
If there were 5 files with poem in them, file.txt would read 5. If file.txt already exists, the new count will be appended to the end. If you want overwrite file each time, change the line to read ls -l | grep -c poem > file.txt
I want to put certain lines in to two different files in a shell script. How should I put the syntax for this.
Example:
A for loop prints 6 lines, and I want that the first two lines should be appended to 1st file, and the last 4 lines should be appended to the other file.
A for loop prints 6 lines, and I want that the first two lines should
be appended to 1st file, and the last 4 lines should be appended to
the other file.
There is no way. One option would be to redirect everything to a file and then copy the desired sections of the log to other files.
for i in {1..6}; do
echo $i > log
done
head -4 log >> logfile1 # Appends the first four lines to logfile1
tail -2 log >> logfile2 # Appends the last two lines to logfile2
Answer
If you're using BASH you can use tee to send the same input to both head -n2 and tail -n4 at the same time using a combination of process substitution and a pipe:
$ for i in {1..6}; do echo $i; done | tee >(head -n2 >first2.txt) | tail -n4 >last4.txt
$ cat first2.txt
1
2
$ cat last4.txt
3
4
5
6
Explanation
By default tee takes its STDIN and copies it to file(s) specified as arguments in addition to its STDOUT. Since process substitution returns a /dev/fd path to a file descriptor (echo >(true) to see an example) tee is able to write to that path like any other regular file.
Here's what the tee command looks like after substitution:
tee /dev/fd/xx | tail -n4 >last4.txt
Or more visually:
tee | tail -n4 >last4.txt
:
/dev/fd/xx
:
:..>(head -n2 >first2.txt)
So the output gets copied both to the head process (Whose output is redirected to first2.txt) and out STDIN which is piped to the tail process:
Note that process substitution is a BASH-ism, so if you're using a different shell or concerned about POSIX compliance it might not be available.