I'm using a bash script to concatenate the execution of a series of programs. Each of these programs produces an output and has two flags that can be set as -iInputFile.txt and -oOutputFile.txt, if no flag is set then standard input and output are automatically selected. Most of the time I simply concatenate my programs as
./Program1 | ./Program2 | ./Program3
but if I happen to need to save the data to a file, and then also access it from the next file I need to do
./Program1 | ./Program2 -oFile.txt && ./Program3 -iFile.txt
so my question is if there is a way to provide an input, for example 010, and only convert the symbols between script 2 and 3 from | to && while leaving everything else untouched. Hard-coding it would be impossible since I have up to 12 programs concatenated so it would have even 12! combinations. It's my first time asking so if anything is unclear from the question I'll edit to provide any information required, thank you all in advance.
If you are scripting this, you can hard-code in tee between the pipeline and use bash's default value parameter expansion to essentially turn off the 'write to file feature'
./Program1 | tee ${outFile1:- /dev/null} | ./Program2 | tee ${outFile2:- /dev/null} | \
./Program3 | tee ${outFile3:- /dev/null}
Note that the last call to tee might be superfluous
Proof of Concept
$ unset outFile; echo foo | tee ${outFile:- /dev/null} | cat - && cat ./tmp
foo
cat: ./tmp: No such file or directory
$ outFile=./tmp; echo foo | tee ${outFile:- /dev/null} | cat - && cat ./tmp
foo
foo
Related
Construct the pipe to execute the following job.
"Output of ls should be displayed on the screen and from this output the lines
containing the word ‘poem’ should be counted and the count should be
stored in a file.”
If bash is allowed, use a process substitution as the receiver for tee
ls | tee >( grep -c poem > number.of.poetry.files)
Your attempt was close:
ls | tee /dev/tty | grep poem | wc -l >number_of_poems
The tee /dev/tty copies all ls output to the terminal. This satisfies the requirement that "Output of ls should be displayed on the screen." while also sending ls's output to grep's stdin.
This can be further simplified:
ls | tee /dev/tty | grep -c poem >number_of_poems
Note that neither of these solutions require bash. Both will work with lesser shells and, in particular, with dash which is the default /bin/sh under debian-like systems.
This sounds like a homework assignment :)
#!/bin/bash
ls
ls -l | grep -c poem >> file.txt
The first ls will display the output on the screen
The next line uses a series of pipes to output the number of files/directories containing "poem"
If there were 5 files with poem in them, file.txt would read 5. If file.txt already exists, the new count will be appended to the end. If you want overwrite file each time, change the line to read ls -l | grep -c poem > file.txt
We use some package called Autosys and there are some specific commands of this package. I have a list of variables which i like to pass in one of the Autosys commands as variables one by one.
For example one such variable is var1, using this var1 i would like to launch a command something like this
autosys_showJobHistory.sh var1
Now when I launch the below written command, it gives me the desired output.
echo "var1" | while read line; do autosys_showJobHistory.sh $line | grep 1[1..6]:[0..9][0..9] | grep 24.12.2012 | tail -1 ; done
But if i put the var1 in a file say Test.txt and launch the same command using cat, it gives me nothing. I have the impression that command autosys_showJobHistory.sh does not work in that case.
cat Test.txt | while read line; do autosys_showJobHistory.sh $line | grep 1[1..6]:[0..9][0..9] | grep 24.12.2012 | tail -1 ; done
What I am doing wrong in the second command ?
Wrote all of below, and then noticed your grep statement.
Recall that ksh doesn't support .. as an indicator for 'expand this range of values'. (I assume that's your intent). It's also made ambiguous by your lack of quoting arguments to grep. If you were using syntax that the shell would convert, then you wouldn't really know what reg-exp is being sent to grep. Always better to quote argments, unless you know for sure that you need the unquoted values. Try rewriting as
grep '1[1-6]:[0-9][0-9]' | grep '24.12.2012'
Also, are you deliberately using the 'match any char' operator '.' OR do you want to only match a period char? If you want to only match a period, then you need to escape it like \..
Finally, if any of your files you're processing have been created on a windows machine and then transfered to Unix/Linux, very likely that the line endings (Ctrl-MCtrl-J) (\r\n) are causing you problems. Cleanup your PC based files (or anything that was sent via ftp) with dos2unix file [file2 ...].
If the above doesn't help, You'll have to "divide and conquer" to debug your problem.
When I did the following tests, I got the expected output
$ echo "var1" | while read line ; do print "line=${line}" ; done
line=var1
$ vi Test.txt
$ cat Test.txt
var1
$ cat Test.txt | while read line ; do print "line=${line}" ; done
line=var1
Unrelated to your question, but certain to cause comment is your use of the cat commnad in this context, which will bring you the UUOC award. That can be rewritten as
while read line ; do print "line=${line}" ; done < Test.txt
But to solve your problem, now turn on the shell debugging/trace options, either by changing the top line of the script (the shebang line) like
#!/bin/ksh -vx
Or by using a matched pair to track the status on just these lines, i.e.
set -vx
while read line; do
print -u2 -- "#dbg: Line=${line}XX"
autosys_showJobHistory.sh $line \
| grep 1[1..6]:[0..9][0..9] \
| grep 24.12.2012 \
| tail -1
done < Test.txt
set +vx
I've added an extra debug step, the print -u2 -- .... (u2=stderror, -- closes option processing for print)
Now you can make sure no extra space or tab chars are creeping in, by looking at that output.
They shouldn't matter, as you have left your $line unquoted. As part of your testing, I'd recommend quoting it like "${line}".
Then I'd comment out the tail and the grep lines. You want to see what step is causing this to break, right? So does the autosys_script by itself still produce the intermediate output you're expecting? Then does autosys + 1 grep produce out as expected, +2 greps, + tail? You should be able to easily see where you're loosing your output.
IHTH
I'm looking for a way to use the ouput of a command (say command1) as an argument for another command (say command2).
I encountered this problem when trying to grep the output of who command but using a pattern given by another set of command (actually tty piped to sed).
Context:
If tty displays:
/dev/pts/5
And who displays:
root pts/4 2012-01-15 16:01 (xxxx)
root pts/5 2012-02-25 10:02 (yyyy)
root pts/2 2012-03-09 12:03 (zzzz)
Goal:
I want only the line(s) regarding "pts/5"
So I piped tty to sed as follows:
$ tty | sed 's/\/dev\///'
pts/5
Test:
The attempted following command doesn't work:
$ who | grep $(echo $(tty) | sed 's/\/dev\///')"
Possible solution:
I've found out that the following works just fine:
$ eval "who | grep $(echo $(tty) | sed 's/\/dev\///')"
But I'm sure the use of eval could be avoided.
As a final side node: I've noticed that the "-m" argument to who gives me exactly what I want (get only the line of who that is linked to current user). But I'm still curious on how I could make this combination of pipes and command nesting to work...
One usually uses xargs to make the output of one command an option to another command. For example:
$ cat command1
#!/bin/sh
echo "one"
echo "two"
echo "three"
$ cat command2
#!/bin/sh
printf '1 = %s\n' "$1"
$ ./command1 | xargs -n 1 ./command2
1 = one
1 = two
1 = three
$
But ... while that was your question, it's not what you really want to know.
If you don't mind storing your tty in a variable, you can use bash variable mangling to do your substitution:
$ tty=`tty`; who | grep -w "${tty#/dev/}"
ghoti pts/198 Mar 8 17:01 (:0.0)
(You want the -w because if you're on pts/6 you shouldn't see pts/60's logins.)
You're limited to doing this in a variable, because if you try to put the tty command into a pipe, it thinks that it's not running associated with a terminal anymore.
$ true | echo `tty | sed 's:/dev/::'`
not a tty
$
Note that nothing in this answer so far is specific to bash. Since you're using bash, another way around this problem is to use process substitution. For example, while this does not work:
$ who | grep "$(tty | sed 's:/dev/::')"
This does:
$ grep $(tty | sed 's:/dev/::') < <(who)
You can do this without resorting to sed with the help of Bash variable mangling, although as #ruakh points out this won't work in the single line version (without the semicolon separating the commands). I'm leaving this first approach up because I think it's interesting that it doesn't work in a single line:
TTY=$(tty); who | grep "${TTY#/dev/}"
This first puts the output of tty into a variable, then erases the leading /dev/ on grep's use of it. But without the semicolon TTY is not in the environment by the moment bash does the variable expansion/mangling for grep.
Here's a version that does work because it spawns a subshell with the already modified environment (that has TTY):
TTY=$(tty) WHOLINE=$(who | grep "${TTY#/dev/}")
The result is left in $WHOLINE.
#Eduardo's answer is correct (and as I was writing this, a couple of other good answers have appeared), but I'd like to explain why the original command is failing. As usual, set -x is very useful to see what's actually happening:
$ set -x
$ who | grep $(echo $(tty) | sed 's/\/dev\///')
+ who
++ sed 's/\/dev\///'
+++ tty
++ echo not a tty
+ grep not a tty
grep: a: No such file or directory
grep: tty: No such file or directory
It's not completely explicit in the above, but what's happening is that tty is outputting "not a tty". This is because it's part of the pipeline being fed the output of who, so its stdin is indeed not a tty. This is the real reason everyone else's answers work: they get tty out of the pipeline, so it can see your actual terminal.
BTW, your proposed command is basically correct (except for the pipeline issue), but unnecessarily complex. Don't use echo $(tty), it's essentially the same as just tty.
You can do it like this:
tid=$(tty | sed 's#/dev/##') && who | grep "$tid"
I would like to pipe standard output of a program while keeping it on screen.
With a simple example (echo use here is just for illustration purpose) :
$ echo 'ee' | foo
ee <- the output I would like to see
I know tee could copy stdout to file but that's not what I want.
$ echo 'ee' | tee output.txt | foo
I tried
$ echo 'ee' | tee /dev/stdout | foo but it does not work since tee output to /dev/stdout is piped to foo
Here is a solution that works at on any Unix / Linux implementation, assuming it cares to follow the POSIX standard. It works on some non Unix environments like cygwin too.
echo 'ee' | tee /dev/tty | foo
Reference: The Open Group Base Specifications Issue 7
IEEE Std 1003.1, 2013 Edition, §10.1:
/dev/tty
Associated with the process group of that process, if any. It is
useful for programs or shell procedures that wish to be sure of
writing messages to or reading data from the terminal no matter how
output has been redirected. It can also be used for applications that
demand the name of a file for output, when typed output is desired and
it is tiresome to find out what terminal is currently in use. In each process, a synonym for the controlling terminal
Some environments like Google Colab have been reported not to implement /dev/tty while still having their tty command returning a usable device. Here is a workaround:
tty=$(tty)
echo 'ee' | tee $tty | foo
or with an ancient Bourne shell:
tty=`tty`
echo 'ee' | tee $tty | foo
Another thing to try is:
echo 'ee' | tee >(foo)
The >(foo) is a process substitution.
Edit:
To make a bit clearer, (.) here start a new child process to the current terminal, where the output is being redirected to.
echo ee | tee >(wc | grep 1)
# ^^^^^^^^^^^^^^ => child process
Except that any variable declarations/changes in child process do not reflect in the parent, there is very few of concern with regard to running commands in a child process.
Try:
$ echo 'ee' | tee /dev/stderr | foo
If using stderr is an option, of course.
Access to "/dev/stdout" is denied on some systems, but access to the user terminal is given by "/dev/tty".
Using "wc" for "foo", the above examples work OK (on linux, OSX, etc.) as:
% echo 'Hi' | tee /dev/tty | wc
Hi
1 1 3
To add a count at the bottom of a list of matching files, I use something like:
% ls [A-J]* | tee /dev/tty | wc -l
To avoid having to remember all this, I define aliases:
% alias t tee /dev/tty
% alias wcl wc -l
so that I can simply say:
% ls [A-J]* | t | wcl
POSTSCRIPT: For the younger set, who might titter at its pronunciation as "titty", I might add that "tty" was once the common
abbreviation for a "teletype" terminal, which used a roll of yellow
paper and had round keys that often stuck.
first you need to figure out the terminal associated with your screen (or whichever screen you want the output to display on):
tty
then you can tee the output to that terminal and pipe the other copy through your foo program:
echo ee | tee /dev/pty/2 | foo
As I build *nix piped commands I find that I want to see the output of one stage to verify correctness before building the next stage but I don't want to re-run each stage. Does anyone know of a program that will help with that? It would keep the output of the last stage automatically to use for any new stages. I usually do this by sending the result of each command to a temporary file (i.e. tee or run each command one at a time) but it would be nice for a program to handle this.
I envision something like a tabbed interface where each tab is labeled with each pipe command and selecting a tab shows the output (at least a hundred lines) of applying that command to to the previous result.
Use 'tee' to copy the intermediate results out to some file as well as pass them on to the next stage of the pipe, like so:
cat /var/log/syslog | tee /tmp/syslog.out | grep something | tee /tmp/grep.out | sed 's/foo/bar/g' | tee /tmp/sed.out | cat >>/var/log/syslog.cleaned
You can also use pipes if you need bidirectional communication (i.e. with netcat):
mknod backpipe p
nc -l -p 80 0<backpipe | tee -a inflow | nc localhost 81 | tee -a outflow 1>backpipe
(via)
There's also the "pv" command - available in debian / ubuntu repostitories which shows you the throughput of your pipes.
An example from the man page :
Transferring a file from another process and passing the expected size to pv:
cat file | pv -s 12345 | nc -w 1 somewhere.com 3000
tee(1) is your friend. It sends its input to both the specified file and stdout.
Stick it between your pipes. For example:
ls | tee /tmp/out1 | sort | tee /tmp/out2 | sed 's/foo/bar/g'