I'm trying to install RVM. There is a magical command line:
bash < <(curl -s https://rvm.io/install/rvm)
I know what bash and curl are. I know the first < is the I/O redirection. But what does <() syntax mean?
What's the difference between this command and
bash < `curl -s https://rvm.io/install/rvm`
?(the latter command doesn't work)
This is bash's process substitution.
The expression <(list) gets replaced by a file name, either a named FIFO or an entry under /dev/fd. So to actually redirect input from it, you have to use < <(list).
[edit; forgot to answer your second question]
The backticks are called "command substitution". Unlike process substitution, it is part of the POSIX shell specification (i.e., not a bash extension). The shell runs the command in the backticks and substitutes its output on the command line. So this would make sense:
cat < `echo /etc/termcap`
But this would not:
cat < `cat /etc/termcap`
The latter is similar to your example; it tries to use the (complex) output of a command as a file name from which to redirect standard input.
The others have already answered your question very nicely. I'll just add an example to build on them... 99% of the time when I personally use <(), it's to diff the output of two different commands in one shot. For instance,
diff <( some_command ) <( some_other_command )
The syntax for io redirection is
process < file
Hence you need whatever appears after the io redirect to be a filename.
The backtick expansion literally puts the results of the command into the command line. Thus,
`curl -s https://rvm.io/install/rvm`
expands to something like
#!/usr/bin/env bash ...
and the shell would be confused because it would see
bash < #...
instead of a filename.
the <() operator is process substitution, which spawns a new process to run the command within the (..). A new file or pipe is created that will capture the result. The fact that the arrow is pointing left <() instead of >() means that the output from the inner process will be written to the file, which can be read by the process.
In your case, bash < <(...) will be seen as something like bash < /dev/fd/100
If you actually want to see what is going on, run
echo <(curl -s https://rvm.io/install/rvm)
It is called Process Substitution.
Related
I have made a program that acts as a shell and for testing purposes, I try and use the < operator, but receive this error in my bash
the purpose is to take ls as an input and run it in my mock shell
Is there a specific reason I am receiving this error, could it be from my code?
the name of my program is rshell:
[xx#xx rshell]$ ./bin/rshell < ls
bash: ls: No such file or directory
The right-hand side of the input redirection operator < requires a filename. So the shell interprets "ls" in your example as a filename. You get an error because there is no such file.
If you want to pass the output of ls to your shell, use a pipe:
ls | ./bin/rshell
Or process substitution:
./bin/rshell < <(ls)
If you want to pass the text "ls" on standard input to your shell, use a here-string:
./bin/rshell <<< ls
The syntax you have, ./bin/rshell < ls, means "read from a file named 'ls'".
If you are trying to send the output of the ls command into /bin/rshell, you need to either pipeline it with a pipe:
ls | ./bin/rshell
or read from an anonymous filehandle:
./bin/rshell < <(ls)
Basically, > and < are for dealing with files (file i/o), and | is what you want for dealing with passing output from one command to another (command i/o); but, you can combine the two with anonymous filehandles using the <(command) syntax.
Inside of a shell script, you can do fancier things, such as storing the output of a command in a string variable, then using that string as input, parsing it up, and all kinds of fun stuff. First things first, though.
An excellent resource for learning all things bash is the Advanced Bash-Scripting Guide: http://www.tldp.org/LDP/abs/html/index.html
In general I don't understand how to make most commands in a UNIX shell script do loop work the same as they work directly from the command line (using bash).
As a simple test, a script called looping.sh to execute an SQL script (what's in filelist.txt doesn't matter in this case):
for i in $(cat filelist.txt)
do $(sqlplus DB_USER/password#abc #test.sql)
done
results in
looping.sh: line 2: SQL*Plus:: command not found
for each line in filelist.txt. Other variations on the 2nd line don't work, like putting it in quotes etc.
Or, if filelist.txt has names of other sh scripts, let's say a single line in this case called_file1.sh and I want to execute it
for i in $(cat filelist.txt)
do exec $i
done
results in
: not found line 2: exec: called_file1.sh
The files are all in the same folder. I tried variations for the second line like /bin/sh $i, putting it in quotes and so on. What's the magic way to execute a command in the do loop?
$(...) takes the contents and runs it as a command and then returns the output from the command.
So when you write:
for i in $(cat filelist.txt)
do $(sqlplus DB_USER/password#abc #test.sql)
done
what the shell does when it hits the body of the loop is run sqlplus DB_USER/password#abc #test.sql and then it takes the output from that command (whatever it may be) and replaces the $(...) bit with it. So you end up with (not exactly since it happens again every loop but for sake of illustration) a loop that looks like this:
for i in $(cat filelist.txt)
do <output of 'sqlplus DB_USER/password#abc #test.sql' command>
done
and if that output isn't a valid shell command you are going to get an error.
The solution there is to not do that. You don't want the wrapping $() there at all.
for i in $(cat filelist.txt)
do sqlplus DB_USER/password#abc #test.sql
done
In your second example:
for i in $(cat filelist.txt)
do exec $i
done
you are telling the shell that the filename in $i is something that it should try to execute like a binary or executable shell script.
In your case two things are happening here. The filename in $i can't be found and (and this is harder to notice) the filename in $i contains a carriage-return at the end (probably a DOS line-ending file). (That's why the error message is a bit more confused then normal.) (I actually wonder about that since I wouldn't have expected that from an unquoted $i but from a quoted "$i" but I might just be wrong about that.)
So, for this case, you need to both strip the carriage-returns from the file (see point 1 of the "Before asking about problematic code" section of the bash tag info wiki for more about this) and then you need to make sure that filename is an executable script and you have the correct path to it.
Oh, also, exec never returns so that loop will only execute one file ever.
If you want multiple executions then drop exec.
That all being said you Don't Read Lines With For. See Bash FAQ 001 for how to correctly (and safely) read lines from a file.
Is there any bash trick that allows giving some parameters in command line to a program that gets its inputs via input stream? Something like this:
program < 'a=1;b=a*2;'
but < needs a file input stream.
For very short here-documents, there are also here-strings:
program <<< "a=1;b=a*2"
I think
echo 'a=1;b=a*2;' | program
is what you need. This process is called "piping"
As a side note: doing the opposite (i.e. piping other programs output as arguments) could be done with xargs
echo works great. The other answer is Here-documents [1]
program <<EOF
a=1;b=a*2;
EOF
I use echo when I have one very short thing on one line, and heredocs when I have something that requires newlines.
[1] http://tldp.org/LDP/abs/html/here-docs.html
shopt -s expand_aliases
alias 'xscript:'='<<:ends'
xscript: bc | anotherprog | yetanotherprog ...
a=1;b=a*2;
:ends
Took me a year to hack this one out. Premium bash script here fellas. Give respect where due please :)
I call this little 'diddy' xscript because you can expand bash variables and substitutions inside of the here document.
alias 'script:'='<<":ends"'
The above version does not expand substitutions.
xscript: cat
The files in our path are: `ls -A`
:ends
script: cat
The files in our path are: `ls -A`
:ends
I'm not finished!
source <(xscript: cat
echo \$BASH "hello world, I'mma script genius!"
echo You can thank me now $USER
:ends
)
This command works fine:
$ bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
However, I don't understand how exactly stable is passed as a parameter to the shell script that is downloaded by curl. That's the reason why I fail to achieve the same functionality from within my own shell script - it gives me ./foo.sh: 2: Syntax error: redirection unexpected:
$ cat foo.sh
#!/bin/sh
bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
So, the questions are: how exactly this stable param gets to the script, why are there two redirects in this command, and how do I change this command to make it work inside my script?
Regarding the "redirection unexpected" error:
That's not related to stable, it's related to your script using /bin/sh, not bash. The <() syntax is unavailable in POSIX shells, which includes bash when invoked as /bin/sh (in which case it turns off nonstandard functionality for compatibility reasons).
Make your shebang line #!/bin/bash.
Understanding the < <() idiom:
To be clear about what's going on -- <() is replaced with a filename which refers to the output of the command which it runs; on Linux, this is typically a /dev/fd/## type filename. Running < <(command), then, is taking that file and directing it to your stdin... which is pretty close the behavior of a pipe.
To understand why this idiom is useful, compare this:
read foo < <(echo "bar")
echo "$foo"
to this:
echo "bar" | read foo
echo "$foo"
The former works, because the read is executed by the same shell that later echoes the result. The latter does not, because the read is run in a subshell that was created just to set up the pipeline and then destroyed, so the variable is no longer present for the subsequent echo.
Understanding bash -s stable:
bash -s indicates that the script to run will come in on stdin. All arguments, then, are fed to the script in the $# array ($1, $2, etc), so stable becomes $1 when the script fed in on stdin is run.
The standard way to capture command output in Bourne shell is to use the $() syntax:
output=$(mycommand)
For commands that have a lot of output, however, this requires the shell allocate memory for the whole thing as one long string. I'd prefer to find something that does the moral equivalent of the Unix C function popen, to get a new file descriptor I could read from:
newfd=popen(mycommand)
while read -u $newfd LINE; do
#process output
done
Is this even possible?
#!bash
ls | while read X
do
echo $X is a directory entry
done
Replace 'ls' with the command of your choice