I'm having difficulty grasping how pipes work. Initially I thought of them as per the title but I couldn't get a simple example to work e.g.
mkdir temp
cd temp
echo "rubbish" > txtfile
ls | cat
I'm wondering why it returns the output from 'ls' rather than the output of 'cat txtfile' (i.e. "rubbish"). I've read many pipe tutorials but none of them seem to go beyond "STDOUT of LHS becomes STDIN for RHS" and I'm left wondering what is STDIN of RHS. Does it become the first argument? Where does it slot in when RHS of pipe has options or more than one argument. Is there any kind of macro substitution taking place or is my thinking wide of the mark.
Edit: I'm still none the wiser 5 comments later. I'll certainly take a look at Roadowl's pv utility but for now if I type
ls | cut -c 2-4
I get
xtf
which I'd expect. So, does cut take its input from stdin but cat doesn't?
Edit2: I stuck the question up on askubuntu (I originally put it up here by mistake). The answer there https://askubuntu.com/questions/1316848/does-output-from-lhs-of-pipe-become-an-arg-for-rhs-of-pipe throws a bit more light on it.
Edit3: While reading the answers here and ask ubuntu and the links therein it struck me (again) how woeful bash (& cohorts) are. It's almost like they're designed to trip you up. I only started using bash a couple of months back and every time I write a script I have to read endless web pages to get it to work or discover where I'm going wrong. Take a simple [[ $1=="..." ]] condition. You forget the spaces round the operator and the else condition might wipe some files you want without so much as a warning. Yes, you can do great things with it without a lot of typing but at times it's like using a tightrope to get from skyscraper A to skyscraper B to avoid using 2 lifts. What's up with gold c code like cat(ls())? That said, thanks to everyone who contributed.
I guess, you meant while performing
ls | cat
ls should return txtfile and which should go as a file input to cat command.
But, the things happening in the background are different :
First your shell creates a pipe using pipe(int pipefd[2]) system-call. This pipe has 2 ends, one is read and another is write.
When ls command is executing, it writes its output to the write end of the pipe and cat simultaneously reads from the read end of the pipe.
So, here STDOUT of ls is the write end whereas STDIN for cat is read end of the pipe.
While reading from the pipe cat will consider it as a stream of bytes and not as a name of the file.
So basically, cat is printing whatever is coming as a stream of bytes.
Read about pipe() over here : pipe(2) — Linux manual page
ls | cut -c 2-4
Here, cut reads its standard input, gets the line txtfile, takes characters 2 to 4 from it, producing xtf, and prints that on standard output. That's what the command line option tells it to do.
ls | cat
Here, cat reads its standard input, gets the line txtfile, and prints that on standard output, unchanged. That's what cat does. If there were further lines, it would do the same for those.
Both read standard input unless one or more file names are given as arguments. That standard input is connected to the terminal (the same one where you enter the command line), unless you use pipes or redirections to change that.
So, run the command cut -c 2-4, and enter the line abcdefghijkl, and it will print out bcd. Because without any arguments, it reads its standard input, which is the terminal, by default. Similarly for running just cat, you'll get back the same line you entered.
Running ls | cut -c 2-4 changes where the standard input comes from, but it doesn't create any new command line arguments (other than the -c and 2-4 you gave). Command line arguments are not the same as the standard input.
So, echo txtfile | cat is not the same as running cat txtfile, any more than running echo txtfile | cut -c 2-4 is the same as running cut -c 2-4 txtfile. For some reason, you seem to expect the pipe should work differently for cat than it does for cut.
I try to create a one-liner for opening less on the last screen of an multi-screen output coming from standard input. The reason for this is that I am working on a program that produces a long AST and I need to be able to traverse up and down through it but I would prefer to start at the bottom. I came up with this:
$ python a.py 2>&1 | tee >(lines=+$(( $(wc -l) - $LINES))) | less +$lines
First, I need to compute number of lines in output and subtract $LINES from it so I know what's the uppermost line of the last screen. I will need to reuse a.py output later so I use tee with process substitution for that purpose. As the last step I point less to open an original stdout on a particular line. Of course, it doesn't work in Bash because $lines is not set in last step as every subcommand is run in a subshell. In ZSH, even though pipe commands are not run in a subshell, process substitution still is and therefore it doesn't work neither. It's not a homework or a work task, I just wonder whether it's possible to do what I want without creating a temporary file in Bash or ZSH. Any ideas?
less supports this innately. The + syntax you're using accepts any less command you could enter while it's running, including G for go-to-end.
... | less +G
does exactly what you want.
This is actually mentioned explicitly as an example in the man page (search for "+G").
The real answer to your question should be the option +G to less, but you indicated that the problem definition is not representative for the abstract problem you want to solve. Therefore, please consideer this alternative problem:
python a.py 2>&1 | \
awk '
{a[NR]=$0}
END{
print NR
for (i=1;i<=NR;i++)print a[i]
}
' | {
read -r l
less -j-1 +$l
}
The awk command is printing the number of lines, and then all the lines in sequence. We define the first line to contain some meta information. This is piped to a group of commands delimited by { and }. The first line is consumed by read, which stores it in variable $l. The rest of the lines are taken by less, where this variable can be used. -j-1 is used, so the matched line is at the bottom of the screen.
Many people have shown how to keep spaces when reading a line in bash. But I have a character based algorithm which need to process each end every character separately - spaces included. Unfortunately I am unable to get bash read to read a single space character from input.
while read -r -n 1 c; do
printf "[%c]" "$c"
done <<< "mark spitz"
printf "[ ]\n"
yields
[m][a][r][k][][s][p][i][t][z][][ ]
I've hacked my way around this, but it would be nice to figure out how to read a single any single character.
Yep, tried setting IFS, etc.
Just set the input field separator(a) so that it doesn't treat space (or any character) as a delimiter, that works just fine:
printf 'mark spitz' | while IFS="" read -r -n 1 c; do
printf "[%c]" "$c"
done
echo
That gives you:
[m][a][r][k][ ][s][p][i][t][z]
You'll notice I've also slightly changed how you're getting the input there, <<< appears to provide a extraneous character at the end and, while it's not important to the input method itself, I though it best to change that to avoid any confusion.
(a) Yes, I'm aware that you said you've tried setting IFS but, since you didn't actually show how you'd tried this, and it appears to work fine the way I do it, I have to assume you may have just done something wrong.
I have a file which has very long rows of data. When i try to read using shell script, the data comes into multiple lines,ie, breaks at certain points.
Example row:
B_18453583||Active|917396140129|405819121107402|Active|7396140129||7396140129|||||||||18-MAY-10|||||18-MAY-10|405819121107402|Outgoing International Calls,Outgoing Calls,WAP,Call Waiting,MMS,Data Service,National Roaming-Voice,Outgoing International Calls except home country,Conference Call,STD,Call Forwarding-Barr,CLIP,Incoming Calls,INTSNS,WAPSNS,International Roaming-Voice,ISD,Incoming Calls When Roaming Internationally,INTERNET||For You Plan||||||||||||||||||
All this is the content of a single line.
I use a normal read like this :
var=`cat pranay.psv`
for i in $var; do
echo $i
done
The output comes as:
B_18453583||Active|917396140129|405819121107402|Active|7396140129||7396140129|||||||||18- MAY-10|||||18-MAY-10|405819121107402|Outgoing
International
Calls,Outgoing
Calls,WAP,Call
Waiting,MMS,Data
Service,National
Roaming-Voice,Outgoing
International
Calls
except
home
country,Conference
Call,STD,Call
Forwarding-Barr,CLIP,Incoming
Calls,INTSNS,WAPSNS,International
Roaming-Voice,ISD,Incoming
Calls
When
Roaming
Internationally,INTERNET||For
You
Plan||||||||||||||||||
How do i print all in single line??
Please help.
Thanks
This is because of word splitting. An easier way to do this (which also disbands with the useless use of cat) is this:
while IFS= read -r -d $'\n' -u 9
do
echo "$REPLY"
done 9< pranay.psv
To explain in detail:
$'...' can be used to create human readable strings with escape sequences. See man bash.
IFS= is necessary to avoid that any characters in IFS are stripped from the start and end of $REPLY.
-r avoids interpreting backslash in text specially.
-d $'\n' splits lines by the newline character.
Use file descriptor 9 for data storage instead of standard input to avoid greedy commands like cat eating all of it.
You need proper quoting. In your case, you should use the command read:
while read line ; do
echo "$line"
done < pranay.psv
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
Shell scripts are often used as glue, for automation and simple one-off tasks. What are some of your favorite "hidden" features of the Bash shell/scripting language?
One feature per answer
Give an example and short description of the feature, not just a link to documentation
Label the feature using bold title as the first line
See also:
Hidden features of C
Hidden features of C#
Hidden features of C++
Hidden features of Delphi
Hidden features of Python
Hidden features of Java
Hidden features of JavaScript
Hidden features of Ruby
Hidden features of PHP
Hidden features of Perl
Hidden features of VB.Net
insert preceding line's final parameter
alt-. the most useful key combination ever, try it and see, for some reason no one knows about this one.
press it again and again to select older last parameters.
great when you want to do something else to something you used just a moment ago.
If you want to keep a process running after you log out:
disown -h <pid>
is a useful bash built-in. Unlike nohup, you can run disown on an already-running process.
First, stop your job with control-Z, get the pid from ps (or use echo $!), use bg to send it to the background, then use disown with the -h flag.
Don't forget to background your job or it will be killed when you logout.
Almost everything listed under EXPANSION section in the manual
In particular, parameter expansion:
$ I=foobar
$ echo ${I/oo/aa} #replacement
faabar
$ echo ${I:1:2} #substring
oo
$ echo ${I%bar} #trailing substitution
foo
$ echo ${I#foo} #leading substitution
bar
My favorite:
sudo !!
Rerun the previous command with sudo.
More magic key combinations:
Ctrl + r begins a “reverse incremental search” through your command history. As you continue to type, it retrieves the most recent command that contains all the text you enter.
Tab completes the word you've typed so far if it's unambiguous.
Tab Tab lists all completions for the word you've typed so far.
Alt + * inserts all possible completions, which is particularly helpful, say, if you've just entered a potentially destructive command with wildcards:
rm -r source/d*.c Alt + *
rm -r source/delete_me.c source/do_not_delete_me.c
Ctrl + Alt + e performs alias, history, and shell expansion on the current line. In other words, the current line is redisplayed as it will be processed by the shell:
ls $HOME/tmp Ctrl Alt + e
ls -N --color=tty -T 0 /home/cramey
Get back history commands and arguments
It's possible to selectively access previous commands and arguments using the ! operator. It's very useful when you are working with long paths.
You can check your last commands with history.
You can use previous commands with !<n> being n the index of the command in history, negative numbers count backwards from the last command in history.
ls -l foo bar
touch foo bar
!-2
You can use previous arguments with !:<n>, zero is the command, >= 1 are the arguments.
ls -l foo
touch !:2
cp !:1 bar
And you can combine both with !<n>:<m>
touch foo bar
ls -l !:1 !:2
rm !-2:1 !-2:2
!-2
You can also use argument ranges !<n>:<x>-<y>
touch boo far
ls -l !:1-2
Other ! special modifiers are:
* for all the arguments
ls -l foo bar
ls !*
^ for the first argument (!:1 == !^)
$ for the last argument
ls -l foo bar
cat !$ > /dev/null
I like the -x feature, allowing to see what's going on in your script.
bash -x script.sh
SECONDS=0; sleep 5 ; echo "that took approximately $SECONDS seconds"
SECONDS
Each time this parameter is
referenced, the number of seconds
since shell invocation is returned.
If a value is assigned to SECONDS,
the value returned upon subsequent
references is the number of seconds
since the assignment plus the value
assigned. If SECONDS is unset, it
loses its special properties, even if
it is subsequently reset.
Here is one of my favorites. This sets tab completion to not be case sensitive. It's really great for quickly typing directory paths, especially on a Mac where the file system is not case sensitive by default. I put this in .inputrc in my home folder.
set completion-ignore-case on
The special variable random:
if [[ $(($RANDOM % 6)) = 0 ]]
then echo "BANG"
else
echo "Try again"
fi
Regular expression handling
Recent bash releases feature regular expression matching, so you can do:
if [[ "mystring" =~ REGEX ]] ; then
echo match
fi
where REGEX is a raw regular expression in the format described by man re_format.
Matches from any bracketed parts are stored in the BASH_REMATCH array, starting at element 1 (element 0 is the matched string in its entirety), so you can use this to do regex-powered parsing too.
Ctrlx Ctrle
This will load the current command into the editor defined in the variable VISUAL. This is really useful for long commands like some of those listed here.
To use vi as your editor:
export VISUAL=vi
Quick & Dirty correction of typos (especially useful for long commands over slow connections where using the command history and scrolling through it would be horrible):
$ cat /proc/cupinfo
cat: /proc/cupinfo: No such file or directory
$ ^cup^cpu
Also try !:s/old/new which substitutes old with new in the previous command once.
If you want to substitute many occurrences you can do a global substitution with !:gs/old/new.
You can use the gs and s commands with any history event, e.g.
!-2:s/old/new
To substitute old with new (once) in the second to last command.
Here two of my favorites:
To check the syntax w/o really executing the script use:
bash -n script.sh
Go back to the last directory (yes I know pushd and popd, but this is quicker)
cd -
Using Infix Boolean Operators
Consider the simple if:
if [ 2 -lt 3 ]
then echo "Numbers are still good!"
fi
That -lt looks kinda ugly. Not very modern. If you use double brackets around your boolean expression you can the normal boolean operators!
if [[ 2 < 3 ]]
then echo "Numbers are still good!"
fi
Arrays:
#!/bin/bash
array[0]="a string"
array[1]="a string with spaces and \"quotation\" marks in it"
array[2]="a string with spaces, \"quotation marks\" and (parenthesis) in it"
echo "There are ${#array[*]} elements in the array."
for n in "${array[#]}"; do
echo "element = >>${n}<<"
done
More details on arrays (and other advanced bash scripting stuff) can be found in the Advanced Bash-Scripting Guide.
Running a command before displaying the bash prompt
Set a command in the "PROMPT_COMMAND" env variable and it will be run automatically before each prompt.
Example:
[lsc#home]$ export PROMPT_COMMAND="date"
Fri Jun 5 15:19:18 BST 2009
[lsc#home]$ ls
file_a file_b file_c
Fri Jun 5 15:19:19 BST 2009
[lsc#home]$ ls
For the next april fools, add "export PROMPT_COMMAND=cd" to someone's .bashrc then sit back and watch the confusion unfold.
Magic key combinations from the bash man pages:
Ctrl + a and Ctrl + e move the cursor to the beginning and end of the current line, respectively.
Ctrl + t and Alt + t transpose the character and word before the cursor with the current one, then move the cursor forward.
Alt + u and Alt + l convert the current word (from the cursor to the end) to uppercase and lowercase.
Hint: Press Alt + – followed by either of these commands to convert the beginning of the current word.
Bonus man tips:
While viewing man pages, use / to search for text within the pages. Use n to jump ahead to the next match or N for the previous match.
Speed your search for a particular command or sub-section within the man pages by taking advantage of their formatting:
o Instead of typing /history expansion to find that section, try /^history, using the caret (^) to find only lines that begin with "history."
o Try / read, with a few leading spaces, to search for that builtin command. Builtins are always indented in the man pages.
export TMOUT=$((15*60))
Terminate bash after 15 minutes of idle time, set to 0 to disable. I usually put this to ~/.bashrc on my root accounts. It's handy when administrating your boxes and you may forget to logout before walking away from the terminal.
Undo
C-S-- Control Shift Minus Undo-es typing actions.
Kill / Yank
Any delete operation C-w (delete previous word), C-k (delete to end of line), C-u (delete to start of line) etc... copies it's deleted text to the kill ring, you can paste the last kill with: C-y and cycle through (and paste from) the ring of deleted items with Alt-y
You can ignore certain files while tab completing by setting th FIGNORE variable.
For example, if you have a subverion repo and you want to navigate more easily do
export FIGNORE=".svn"
now you can cd without being blocked by .svn directories.
Using arithmetic:
if [[ $((2+1)) = $((1+2)) ]]
then echo "still ok"
fi
Brace expansion
Standard expansion with {x,y,z}:
$ echo foo{bar,baz,blam}
foobar foobaz fooblam
$ cp program.py{,.bak} # very useful with cp and mv
Sequence expansion with {x..y}:
$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z
$ echo {a..f}{0..3}
a0 a1 a2 a3 b0 b1 b2 b3 c0 c1 c2 c3 d0 d1 d2 d3 e0 e1 e2 e3 f0 f1 f2 f3
I recently read Csh Programming Considered Harmful which contained this astounding gem:
Consider the pipeline:
A | B | C
You want to know the status of C, well, that's easy: it's in $?, or
$status in csh. But if you want it from A, you're out of luck -- if
you're in the csh, that is. In the Bourne shell, you can get it, although
doing so is a bit tricky.
Here's something I had to do where I ran dd's
stderr into a grep -v pipe to get rid of the records in/out noise, but had
to return the dd's exit status, not the grep's:
device=/dev/rmt8
dd_noise='^[0-9]+\+[0-9]+ records (in|out)$'
exec 3>&1
status=`((dd if=$device ibs=64k 2>&1 1>&3 3>&- 4>&-; echo $? >&4) |
egrep -v "$dd_noise" 1>&2 3>&- 4>&-) 4>&1`
exit $status;
Truncate content of a file (zeroing file)
> file
Specifically, this is very good for truncating log files, when the file is open by another process, which still may write to the file.
Not really a feature but rather a direction: I found many "hidden features", secrets and various bash usefulness at commandlinefu.com. Many of the highest rated answers to this answers, I learned them on that site :)
Another small one:
Alt+#
comments out the current line and moves it into the history buffer.
So when you're assembling a command line and you need to issue an interim command to e.g. find a file, you just hit alt+#, issue the other command, go up in the history, uncomment and proceed.
Braces in lieu of do and done in for loop
For loop body are usually in do...done (just an example):
for f in *;
do
ls "$f";
done
But we can use a C style using braces:
for f in *; {
ls "$f";
}
I think this looks better than do...doneand I prefer this one. I have not yet found this in any Bash documentation, so this is really a hidden feature.
C style numeric expressions:
let x="RANDOM%2**8"
echo -n "$x = 0b"
for ((i=8; i>=0; i--)); do
let n="2**i"
if (( (x&n) == n )); then echo -n "1"
else echo -n "0"
fi
done
echo ""
These properties are another one of my favorites.
export HISTCONTROL=erasedups
export HISTSIZE=1000
The first one makes sure bash doesn't log commands more than once, will really improves history's usefulness. The other expands the history size to 1000 from the default of 100. I actually set this to 10000 on my machines.